BACKGROUND AND PURPOSE: Accurate conformal radiotherapy treatment requires manual delineation of target volumes and organs at risk (OAR) that is both time-consuming and subject to large inter-user variability. One solution is atlas-based automatic segmentation (ABAS) where a priori information is used to delineate various organs of interest. The aim of the present study is to establish the accuracy of one such tool for the head and neck (H&N) using two different evaluation methods. MATERIALS AND METHODS: Two radiotherapy centres were provided with an ABAS tool that was used to outline the brainstem, parotids and mandible on several patients. The results were compared to manual delineations for the first centre (EM1) and reviewed/edited for the second centre (EM2), both of which were deemed as equally valid gold standards. The contours were compared in terms of their volume, sensitivity and specificity with the results being interpreted using the Dice similarity coefficient and a receiver operator characteristic (ROC) curve. RESULTS: Automatic segmentation took typically approximately 7min for each patient on a standard PC. The results indicated that the atlas contour volume was generally within +/-1SD of each gold standard apart from the parotids for EM1 and brainstem for EM2 that were over- and under-estimated, respectively (within +/-2SD). The similarity of the atlas contours with their respective gold standard was satisfactory with an average Dice coefficient for all OAR of 0.68+/-0.25 for EM1 and 0.82+/-0.13 for EM2. All data had satisfactory sensitivity and specificity resulting in a favourable position in ROC space. CONCLUSIONS: These tests have shown that the ABAS tool exhibits satisfactory sensitivity and specificity for the OAR investigated. There is, however, a systematic over-segmentation of the parotids (EM1) and under-segmentation of the brainstem (EM2) that require careful review and editing in the majority of cases. Such issues have been discussed with the software manufacturer and a revised version is due for release.