The primary target of content based image retrieval is to return a list of images that are most similar to a query image. This is usually done by ordering the images based on a similarity score. In most state-of-the-art systems, the magnitude of this score is very different from a query to another. This prevents us from making a proper decision about the correctness of the returned images. This paper considers the applications where a confidence measurement is required, such as in copy detection or when a re-ranking stage is applied on a short-list such as geometrical verification. For this purpose, we formulate image search as an outlier detection problem, and propose a framework derived from extreme values theory. We translate the raw similarity score returned by the system into a relevance score related to the probability that a raw score deviates from the estimated model of scores of random images. The method produces a relevance score which is normalized in the sense that it is more consistent across queries. Experiments performed on several popular image retrieval benchmarks and state-of-the-art image representations show the interest of our approach.