xrAI - Improving Quality and Efficiency in Chest Radiograph Interpretation by Radiologists
Primary Purpose
Pulmonary Disease
Status
Withdrawn
Phase
Not Applicable
Locations
Study Type
Interventional
Intervention
Radiograph interpretation for pulmonary abnormalities
Sponsored by
About this trial
This is an interventional diagnostic trial for Pulmonary Disease
Eligibility Criteria
Inclusion Criteria:
- Radiologist currently practicing at a Pureform Radiology clinic in Calgary, Canada.
Exclusion Criteria:
- Radiologists not currently practicing at a Pureform Radiology clinic in Calgary, Canada.
- Physicians currently practicing at a Pureform Radiology clinic in Calgary, Canada, but that are not radiologists by training.
Sites / Locations
Arms of the Study
Arm 1
Arm 2
Arm Type
Placebo Comparator
Experimental
Arm Label
Control
Treatment
Arm Description
Participants will review the 500 chest radiographs without the assistance of xrAI
Participants will review the 500 chest radiographs with the assistance of xrAI
Outcomes
Primary Outcome Measures
Number of abnormalities identified divided by number of total of images analyzed (accuracy)
Accuracy is defined as the ratio of the images where the physician's prediction matched the labels of the dataset.
Accuracy= (TP+FP) / (TP+FP+TN+FN)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Number of true abnormalities identified divided by the total of abnormalities identified (precision)
Precision is defined as the probability of a radiograph being abnormal if a physician makes the determination that it is abnormal.
Precision= TP / (TP+FP)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Number of true abnormalities identified divided by the sum of true abnormalities identified and abnormalities missed (recall)
Recall is defined as the probability of a physician catching an abnormality in an image if one exists (based on the labels of the dataset).
Recall= TP / (TP+FN)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Secondary Outcome Measures
Mean of radiologist accuracy (as defined in outcome 1)
Investigators will calculate the mean of the accuracy of all participants in each group.
Mean of radiologist precision (as defined in outcome 2)
Investigators will calculate the mean of the precision of all participants in each group.
Mean of radiologist recall (as defined in outcome 3)
Investigators will calculate the mean of the recall of all participants in each group.
Full Information
NCT ID
NCT04221100
First Posted
January 6, 2020
Last Updated
February 21, 2021
Sponsor
1QB Information Technologies Inc.
1. Study Identification
Unique Protocol Identification Number
NCT04221100
Brief Title
xrAI - Improving Quality and Efficiency in Chest Radiograph Interpretation by Radiologists
Official Title
xrAI - Improving Quality and Efficiency in Chest Radiograph Interpretation by Radiologists
Study Type
Interventional
2. Study Status
Record Verification Date
February 2021
Overall Recruitment Status
Withdrawn
Why Stopped
Not enough participants enrolled in the study
Study Start Date
March 2021 (Anticipated)
Primary Completion Date
May 2021 (Anticipated)
Study Completion Date
May 2021 (Anticipated)
3. Sponsor/Collaborators
Responsible Party, by Official Title
Sponsor
Name of the Sponsor
1QB Information Technologies Inc.
4. Oversight
Studies a U.S. FDA-regulated Drug Product
No
Studies a U.S. FDA-regulated Device Product
No
Data Monitoring Committee
No
5. Study Description
Brief Summary
xrAI (pronounced "X-ray") serves as a clinical assistance tool for trained clinical professionals who are interpreting chest radiographs. The tool is designed as a quality control and adjunct, limited, clinical decision support tool, and does not replace the role of clinical professionals. It highlights areas on chest radiographs for review by an interpreting clinician.
The objective of this study is to utilize machine learning and artificial intelligence algorithms (xrAI) to improve the quality and efficiency in the interpretation of chest radiographs by radiologists.
The hypothesis is that the addition of xrAI's analysis will reduce inter-observer variability in the interpretation of chest radiographs and increase participants' sensitivity, recall, and accuracy in pulmonary abnormality screening.
Detailed Description
To investigate the effect of xrAI for radiologists that interpret chest radiographs as part of their daily responsibilities, the investigators have designed a randomized control trial.
The pulmonary abnormalities detected by xrAI and included in the definition of abnormal are as follows: any linear scar or fibrosis, atelectasis, consolidation, abscess or cavity, nodule, pleural effusion, severe cases of emphysema and COPD (mild cases with hyperinflation but not significant emphysema are not flagged), and pneumothorax.
To assess the causal effect of xrAI the investigators randomly assign 10 to 14 radiologists to either treatment (x-ray images processed by xrAI) or control (no xrAI processing) groups. Participants will only review images once. Each participant will perform 500 radiograph interpretations in total.
Participants in the control group will be asked to interpret the same 500 images without xrAI's analysis.
To increase the precision of the estimate and better investigate potential differences between clinical professionals, investigators block randomize the assignment of treatment or control group.
To analyse the effect of xrAI, the investigators will estimate the average treatment effect (ATE) for each group by comparing the performance of the treatment and control groups using randomization-based inference (Green and Gerber, 2012).
6. Conditions and Keywords
Primary Disease or Condition Being Studied in the Trial, or the Focus of the Study
Pulmonary Disease
7. Study Design
Primary Purpose
Diagnostic
Study Phase
Not Applicable
Interventional Study Model
Parallel Assignment
Masking
Participant
Allocation
Randomized
Enrollment
0 (Actual)
8. Arms, Groups, and Interventions
Arm Title
Control
Arm Type
Placebo Comparator
Arm Description
Participants will review the 500 chest radiographs without the assistance of xrAI
Arm Title
Treatment
Arm Type
Experimental
Arm Description
Participants will review the 500 chest radiographs with the assistance of xrAI
Intervention Type
Device
Intervention Name(s)
Radiograph interpretation for pulmonary abnormalities
Intervention Description
The pulmonary abnormalities detected by xrAI and included in the definition of abnormality are as follows: any linear scar or fibrosis, atelectasis, consolidation, abscess or cavity, nodule, pleural effusion, severe cases of emphysema and COPD (mild cases with hyperinflation but not significant emphysema are not flagged), pneumothorax.
Participants in the treatment group will interpret 500 images presented alongside the results of xrAI's processing in a dark room and asked to categorize each image into one of the following categories: lungs are clear, at least one pulmonary abnormality is present, not sure.
Participants in the control group will be asked to interpret the same 500 images as the treatment group but without xrAI's analysis.
Primary Outcome Measure Information:
Title
Number of abnormalities identified divided by number of total of images analyzed (accuracy)
Description
Accuracy is defined as the ratio of the images where the physician's prediction matched the labels of the dataset.
Accuracy= (TP+FP) / (TP+FP+TN+FN)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
Title
Number of true abnormalities identified divided by the total of abnormalities identified (precision)
Description
Precision is defined as the probability of a radiograph being abnormal if a physician makes the determination that it is abnormal.
Precision= TP / (TP+FP)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
Title
Number of true abnormalities identified divided by the sum of true abnormalities identified and abnormalities missed (recall)
Description
Recall is defined as the probability of a physician catching an abnormality in an image if one exists (based on the labels of the dataset).
Recall= TP / (TP+FN)
TP (true positives) = cases interpreted as abnormal that are abnormal; FP (false positives) = cases wrongly interpreted to be abnormal; TN (true negatives) = cases correctly interpreted to be normal; FN (false negatives) = abnormal cases wrongly interpreted as normal.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
Secondary Outcome Measure Information:
Title
Mean of radiologist accuracy (as defined in outcome 1)
Description
Investigators will calculate the mean of the accuracy of all participants in each group.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
Title
Mean of radiologist precision (as defined in outcome 2)
Description
Investigators will calculate the mean of the precision of all participants in each group.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
Title
Mean of radiologist recall (as defined in outcome 3)
Description
Investigators will calculate the mean of the recall of all participants in each group.
Time Frame
Time needed to analyze 500 images. Participants will be asked to completed the exercise within 2 weeks.
10. Eligibility
Sex
All
Accepts Healthy Volunteers
Accepts Healthy Volunteers
Eligibility Criteria
Inclusion Criteria:
Radiologist currently practicing at a Pureform Radiology clinic in Calgary, Canada.
Exclusion Criteria:
Radiologists not currently practicing at a Pureform Radiology clinic in Calgary, Canada.
Physicians currently practicing at a Pureform Radiology clinic in Calgary, Canada, but that are not radiologists by training.
12. IPD Sharing Statement
Plan to Share IPD
Undecided
IPD Sharing Plan Description
The results of this study will be shared though we have not yet decided the format (publication in a medical journal, conference, white paper) not the type of information that will be shared (individual anonymized participant data or outcomes of the study). We will update this section once the IPD sharing plan is confirmed.
Learn more about this trial
xrAI - Improving Quality and Efficiency in Chest Radiograph Interpretation by Radiologists
We'll reach out to this number within 24 hrs