Wikia

34,142pages on
this wiki

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

In signal detection theory, a receiver operating characteristic (ROC), also receiver operating curve, is a graphical plot of the sensitivity vs. (1 - specificity) for a binary classifier system as its discrimination threshold is varied. The ROC can also be represented equivalently by plotting the fraction of true positives (TP) vs. the fraction of true negatives (TN). The usage receiver operator characteristic is also common.

ROC curves are used to evaluate the results of a prediction and were first employed in the study of discriminator systems for the detection of radio signals in the presence of noise in the 1940s, following the attack on Pearl Harbor. The initial research was motivated by the desire to determine how the US RADAR "receiver operators" had missed the Japanese aircraft.

In the 1960s they began to be used in psychophysics, to assess human (and occasionally animal) detection of weak signals. They also proved to be useful for the evaluation of machine learning results, such as the evaluation of Internet search engines. They are also used extensively in epidemiology and medical research and are frequently mentioned in conjunction with evidence-based medicine.

The best possible prediction method would yield a graph that was a point in the upper left corner of the ROC space, i.e. 100% sensitivity (all true positives are found) and 100% specificity (no false positives are found). A completely random predictor would give a straight line at an angle of 45 degrees from the horizontal, from bottom left to top right: this is because, as the threshold is raised, equal numbers of true and false positives would be let in. Results below this no-discrimination line would suggest a detector that gave wrong results consistently, and could therefore be simply used to make a detector that gave useful results by inverting its decisions.

Sometimes, the ROC is used to generate a summary statistic. Three common versions are:

• the intercept of the ROC curve with the line at 90 degrees to the no-discrimination line
• the area between the ROC curve and the no-discrimination line
• the area under the ROC curve, often called AUC.
• $d'$ (pronounced "d-prime"), the distance between the mean of the distribution of activity in the system under noise-alone conditions and its distribution under signal plus noise conditions, divided by their standard deviation, under the assumption that both these distributions are normal with the same standard deviation. Under these assumptions, it can be proved that the shape of the ROC depends only on $d'$.

However, any attempt to summarize the ROC curve into a single number loses information about the pattern of tradeoffs of the particular discriminator algorithm.

The machine learning community most often uses the ROC AUC statistic. This measure can be interpreted as the probability that when we randomly pick one positive and one negative example, the classifier will assign a higher score to the positive example than to the negative. In engineering, the area between the ROC curve and the no-discrimination line is often preferred, because of its useful mathematical properties as a non-parametric statistic. This area is often simply known as the discrimination. In psychophysics, $d'$ is the most commonly used measure.

The illustration to the right shows the use of ROC graphs for the discrimination between the quality of different epitope predicting algorithms. If you wish to discover at least 60% of the epitopes in a virus protein, you can read out of the graph that about 1/3 of the output would be falsely marked as an epitope. The information that is not visible in this graph is that the person that uses the algorithms knows what threshold settings give a certain point in the ROC graph.