Rationale and Objectives. Rank-order experiments often provide a reasonable
method of determining whether a large-scale receiver operating characteris
tic study can be justified. The authors' purpose was to formalize a propose
d method for analyzing rank-order imaging experiments and provide methods t
hat can be used in determining sample sizes for both cases and raters.
Materials and Methods. Simulations were conducted to determine the adequacy
of the normal approximation of a statistic used to test the null hypothesi
s of random ordering. For a multireader experiment, formulas are presented
and guidelines are provided to enable investigators to determine the number
of required readers (raters) and cases for a specific study.
Results. When there are at least five ordered images per case, 10 cases are
sufficient to test a random rank order. When there are only three or four
images for a case, 20 cases are required. The authors constructed tables of
statistical power for selected numbers of ordered images, numbers of cases
, and degrees of trend, and they also provide an approximation for use in s
ituations that are not tabled.
Conclusion. The statistical methods for analyzing rank-order experiments an
d estimating sample sizes for study planning are relatively simple to imple
ment. The derived formulas for sample size estimation, when applied to typi
cal imaging experiments, indicate that modest numbers of cases and readers
are required for rank-order studies.