With advances in Deep Learning (DL), DL-based segmentation has become an effective and resource saving way to create segmentations for large amounts of medical imaging data. However, the reliability of DL models remains uncertain, as the models may produce errors. Uncertainty Quantification (UQ) has been frequently used to infer segmentation quality, with higher uncertainty being associated with worse quality. Various approaches have been proposed to quantify, but their clinical applicability tends to be limited. Furthermore, studies on evaluating automated liver segmentation are lacking, which is important assess due to the unique challenges liver segmentation poses. 

Therefore, for this study we test three different UQ methods – a 5-model Deep Ensemble (DE), Monte-Carlo Dropout, and Test-Time Augmentation – to assess whether uncertainty estimates (UEs) can be used to flag scans requiring review. To answer this question, we used 3D CT data from the Oxford Risk Factor and Noninvasive Imaging (ORFAN) study. We used N = 221 3D CT liver scans with ground truth segmentations to train our model, and to validate our manual binary quality control label against accuracy and calibration metrics. In addition, we used a separate test-set of N = 500 scans to validate our UEs against the binary QC label. For our UEs we included four different estimates, two structural metrics – within-sample Dice Similarity Coefficient and Average Symmetric Surface Distance – and two aggregated voxel-level metrics – predictive entropy and mutual information.