Background
Breast cancer is the number one leading cause for cancer-related death in women worldwide, with morbidity and mortality rates varying largely between different countries. Late-stage diagnosis and limited access to diagnostics have been directly linked to low survival rates in low- and middle-income countries [1].
While mammography and BUS are the standard diagnostic tools for breast cancer detection in many high-income countries, POCUS could be a cost-effective alternative. POCUS is a simplified ultrasound approach which consists of a stand-alone portable ultrasound probe where the images are read on a smartphone or tablet. It has been used for a broad range of medical applications, especially in emergency and critical care [2,3], and could potentially also be used in the assessment of breast symptoms [4]. POCUS as a cost-effective and accessible imaging method in limited-resource settings lately has attracted increasing attention [4,5]. Due to the reduced image quality of POCUS compared to BUS, the question regarding how suitable POCUS is to replace BUS for breast cancer diagnosis remains. To face the pronounced lack of radiologists in many low- and middle-income countries [6], AI has been suggested to be used to automatically analyze ultrasound images [7,8].
Methods
We conducted a multi-reader multi-case (MRMC) study involving 40 women of which 70 POCUS and 70 case-matched BUS images (11 malignant, 21 benign, and 38 normal cases each) were acquired (examples in Fig. 1). The patients were between 26-82 years old (mean 50.3, standard deviation 16.7), and 10 of the 40 had malignant lesions. The ground truth was determined by a standard assessment consisting of mammography and ultrasound examinations by a breast radiologist, followed by histopathological diagnosis in case of biopsy.
Four breast radiologists read each set of ultrasound images with a washout period of at least 4 weeks in between. The reader task was to score the images on a 5-level scale (≥ 3 was considered positive). The images were also analyzed by two in-house developed deep learning-based AI algorithms, which can classify images as malignant, benign or normal. One of the algorithms is a convolutional neural network [9], while the second algorithm is a deep ensemble that includes an uncertainty quantification method that can measure predictive uncertainties and can flag unsuitable images using a safety threshold [10]. The threshold for the AI method with safety threshold was chosen such that 15% of the data was excluded due to too high predictive uncertainties.
The performance of radiologists and AI, respectively, on POCUS and BUS were assessed using receiver operating characteristic (ROC) curve, the area under the ROC curve (AUC), sensitivity, and specificity. Statistical significance was measured based on the AUC values by estimating the confidence interval (CI) over the difference in the AUCs. This was done by performing bootstrap on the test set, i.e., resample the test set with replacement 1000 times. The significance level was picked at 0.05 and Bonferroni correction was used to compensate for multiple comparisons.
