Comparing Clinical Decision-making of AI Technology to a Multi-professional Care Team in eCBT for Depression

Comparing Clinical Decision-making of AI Technology to a Multi-professional Care Team in eCBT for Depression

Comparing Clinical Decision-making of AI Technology to a Multi-professional Care Team in an Electronic Cognitive Behavioural Therapy Program for Depression

Depression is a leading cause of disability worldwide, affecting up to 300 million people globally. Despite its high prevalence and debilitating effects, only one-third of patients newly diagnosed with depression initiate treatment. Electronic cognitive behavioural therapy (e-CBT) is an effective treatment for depression and is a feasible solution to make mental health care more accessible. Due to its online format, e-CBT can be combined with variable therapist engagement to address different care needs. Typically, a multi-professional care team determines which combination therapy is the most beneficial to the patient. However, this process can add to the costs of these programs. Artificial intelligence (AI) technology has been proposed to offset these costs. Therefore, this study aims to determine a cost-effective method to decrease depressive symptoms and increase treatment adherence to e-CBT. This will be done by comparing AI technology to a multi-professional care team when allocating the correct intensity of care for individuals diagnosed with depression. This study is a double-blinded randomized controlled trial recruiting individuals (n = 186) experiencing depression according to the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5). The degree of care intensity a participant will receive will be randomly decided by either: (1) a machine learning algorithm (n = 93), or (2) an assessment made by a group of healthcare professionals (n = 93). Subsequently, participants will receive depression-specific e-CBT treatment through the secure online platform, OPTT. There will be three available intensities of therapist interaction: (1) e-CBT; (2) e-CBT with a 15-20-minute phone/video call; and (3) e-CBT with pharmacotherapy. This approach aims to accurately allocate care tailored to each patient's needs, allowing for more efficient use of resources.

Participants (n = 186: n = 31 per e-CBT group * 2 arms) will be recruited at Queen's University from outpatient psychiatry clinics at both Kingston Health Sciences Centre sites (Hotel Dieu Hospital and Kingston General Hospital), as well as Providence Care Hospital in Kingston, Ontario. Additionally, self-referrals and referrals from family doctors, physicians, and clinicians across Ontario will be accepted. After obtaining informed consent from the participant, the participant will be evaluated using the Mini International Psychiatric Assessment (MINI) through a secure video appointment to confirm a diagnosis of Major Depressive Disorder using the DSM-5 by a trained professional on the research team. All eligible participants will be randomized to receive a treatment plan based on the decision of either the healthcare team (Arm 1) or the Triage Module using an AI algorithm (Arm 2). Participants will be randomly allocated to one of the two arms of the study by a research assistant on the team who will also balance the group based on demographic variables (i.e., sex, gender, age, and income). Participants and therapists in the study will be blinded to which treatment arm the participant belongs to. By the nature of this study, participants and therapists will not be blinded to which treatment intensity the participant will receive since it will be evident whether the participant is receiving a phone/video call in addition to usual e-CBT care or pharmacotherapy. Each participant will be provided with an effective form of treatment (i.e., e-CBT) regardless of which group they will be allocated to. Participants will be informed that there is no incentive for joining the program and that joining or withdrawing at any point will not affect them negatively. It will also be explained to the participants that the program is not a crisis resource and that they will not always have access to their therapists. In the case of an emergency, participants will be directed to proper resources, and this event will be reported to the study's lead psychiatrist (principal investigator). All data will be anonymized and will be analyzed by members of the research team who are not directly involved in the patient's care. Treatment Arm 1: Healthcare Team Allocation Allocation of treatment intensity by the multi-professional healthcare team will be based on the following criteria: The severity of MDD symptoms (using DSM-5 criteria). Mental health factors (prior treatments and responses, current and past psychotic/manic episodes, current and past suicidal/homicidal ideation/attempts, family mental health history, past psychiatric history, and hospital admissions). Medical factors (current medical conditions and medications, personal and family medical history). Social factors (support system and living situation, and occupational, social, and personal functional impairment). Additionally, to assess the severity of MDD symptoms and the functional impairments, participants will complete the PHQ-9 and Sheehan Disability Scale (SDS) before the assessment appointment. The assessment appointment will be conducted by the trained research assistant on the multidisciplinary team who will relay the information to the rest of the team later to deliberate on treatment intensity allocation. All assessments will occur virtually through phone and video calls. Together, the healthcare team will decide whether the participant should be assigned to the e-CBT-only treatment, e-CBT treatment with weekly phone/video calls, or e-CBT treatment with pharmacotherapy. This process mimics the current triage process in clinical settings. To track cost-effectiveness, the trained research assistant will track the total duration of the individual assessment and team deliberation meetings for analysis of the total time commitment per patient. Treatment Arm 2: AI Algorithm Allocation Allocation of treatment intensity by the proposed AI algorithm will be based on the machine learning and natural language processing (NLP) of textual data provided by participants and their PHQ-9 score collected through a pre-treatment screening module called the Triage Module. This module, developed by the research team, (1) provides psychoeducation on the effects of psychotherapy, (2) collects PHQ-9 scores, and (3) asks participants six open-ended questions regarding their mental health history, their experiences with mental health disorders, and what mental health difficulties they are currently facing. Based on the participant's answers to the open-ended questions, a variable called "Symptomatic Score" will be calculated using the NLP algorithm. If the PHQ-9 score < 19 and the Symptomatic Score > 0.75, the participant will be assigned to the e-CBT-only treatment group. However, if either the PHQ-9 score is > 19 or the Symptomatic Score is < 0.75, the participants will be assigned to the e-CBT treatment with weekly phone/video calls. If both scenarios occur and the PHQ-9 > 19 and Symptomatic Score < 0.75, then the participant will be assigned to the e-CBT treatment with pharmacotherapy. To gather the relevant data (i.e., participant compliance and change in depression severity, as evaluated by the PHQ-9), the triage module was designed. As previously explained, NLP of the participants' written accounts of their challenges with depression in the Triage Module will be used to calculate a Symptomatic Score. To verify the AI's treatment allocation logic, the completion rate and the change in PHQ-9 scores were assessed in a sample of participants (n = 190) who were previously enrolled in e-CBT-only treatment. The decision-making algorithm determined that the e-CBT-only program was suitable for 62 out of the 190 participants (33%). Within these 62 participants, 54% had completed the e-CBT-only program in its entirety and only 20% had a final PHQ-9 score > 14. Furthermore, the algorithm indicated that e-CBT with telephone calls would be suitable for 100 out of the 190 participants (53%). Of the 100 participants, 41% completed the whole round of e-CBT-only therapy and 31% had a final PHQ-9 score > 14. Lastly, the algorithm indicated that e-CBT with video call was appropriate for 28 out of 190 participants (14%). Of these 28 participants, 35% completed the whole round of e-CBT-only therapy and 40% had a final PHQ-9 score > 14. The logic of the AI's decision is therefore justified as those participants allocated to the e-CBT-only group had the highest percentage of completion and lowest percentage of final PHQ-9 scores > 14 when completing e-CBT-only. Therefore, these individuals require minimal therapist intensity, and e-CBT-only is sufficient. Conversely, participants allocated to the e-CBT with video call had the lowest completion rates and highest rates of final PHQ-9 scores > 14 when enrolled in e-CBT-only. These findings justify the AI's logic that greater therapist interaction is required. It is also important to note that demographic factors like age (below or above 40 years), sex (male or female) and income (less or more than $50K) did not have any significant effects on the number of sessions completed by participants (p = 0.92, 0.18 & 0.9 for age, sex, and income respectively). The demographic factors did not affect the change in PHQ-9 score (i.e., the difference between the beginning and end of treatment scores) either (p = 0.2, 0.46 & 0.39 for age, sex, and income respectively). e-CBT Program The e-CBT sessions used in this study include content based on cognitive restructuring and behavioural activation techniques. The purpose of the sessions is to help participants become aware of inaccurate or negative thinking patterns so that they can view challenging situations more clearly and respond to them effectively. The sessions prompt participants to understand their situation/environment and the resulting thoughts, behaviours, physical reactions, and feelings. The goal of this program is to help change participants' negative and/or ineffective thoughts to more effective ways of thinking. As expressed in CBT, changing thoughts can subsequently affect feelings, behaviours, and physical reactions to stressful situations. Therapists Each participant will be assigned a care provider that will provide feedback for their weekly sessions before the start of their next session. The assigned care provider will be independent of the multi-professional healthcare team that conducted the intake assessment. All care providers are trained in psychotherapy and have experience delivering electronic psychotherapy. They will be informed of the aim and the content of each therapeutic session. They will also continue receiving specialized training through webinars, workshops and exercises with feedback provided by the lead psychiatrist on the research team, a trained and licensed psychotherapist. All care providers will be supervised by a trained psychotherapist and the lead psychiatrist, and all feedback will be reviewed before submission to the participants. e-CBT Weekly Feedback Weekly homework is reviewed by the independent care provider assigned to the participant, who will provide text-based personalized feedback on OPTT before the next weekly session. Additionally, the participants and care providers can communicate asynchronously on OPTT to relay any questions or concerns. The care providers will be provided with sample feedback templates and scripts for the telephone and video call sessions. Templates and scripts will be adapted from previous studies conducted by the research team. Feedback templates and scripts will vary between sessions, and care providers will personalize them for each patient. The feedback templates follow a generic structure starting with, acknowledging the participant's time and effort since the last session, summarising the CBT concepts taught in the previous session, reviewing the event they explained in their homework, validating the participant's experience(s), and encouraging the participant to keep up with the sessions. The feedback is written in a letter format to increase personalization and build rapport with the participants.

If eligible for this randomized controlled trial, participants (n = 186) will be randomized to receive an e-CBT treatment recommended by a multi-professional healthcare team consisting of a psychiatrist, psychiatric medical resident, and a trained research assistant (Arm 1, control group; n = 93), or the AI machine learning algorithm (Arm 2, experimental group; n = 93).

To ensure blinding, all participants will complete the intake assessment by the healthcare team (Arm 1) and the Triage Module (Arm 2). Only the relevant data (i.e., Arm 1: intake assessment vs. Arm 2: Triage Module) will be analyzed depending on the treatment arm that the participant is randomly assigned to.

Allocation of treatment intensity by the proposed AI algorithm will be based on the machine learning and natural language processing (NLP) of textual data provided by participants and their PHQ-9 score collected through a pre-treatment screening module called the Triage Module. This module, developed by the research team, (1) provides psychoeducation on the effects of psychotherapy, (2) collects PHQ-9 scores, and (3) asks participants six open-ended questions regarding their mental health history, their experiences with mental health disorders, and what mental health difficulties they are currently facing. Based on the participant's answers to the open-ended questions, a variable called "Symptomatic Score" will be calculated using the NLP algorithm.

Allocation of treatment intensity by the multi-professional healthcare team will be based on the following criteria: The severity of MDD symptoms (using DSM-5 criteria). Mental health factors (prior treatments and responses, current and past psychotic/manic episodes, current and past suicidal/homicidal ideation/attempts, family mental health history, past psychiatric history, and hospital admissions). Medical factors (current medical conditions and medications, personal and family medical history). Social factors (support system and living situation, and occupational, social, and personal functional impairment).

The participant will submit their weekly homework and receive personalized feedback from their assigned therapist on OPTT. The feedback adds customization by acknowledging the participant's experiences in the past week and ensures the participant has understood the CBT concepts.

In addition to the e-CBT program (see 1 above), the participant will receive a weekly phone/video call from their assigned therapist. The goal is to build on the therapeutic relationship and to add personalization with direct verbal encouragement. This phone/video call is limited to a one-time, 15-20 minutes call each intervention week.44 The purpose is to check with the patient on their treatment progress. The secure call will either be a phone or video (via Microsoft Teams) call, depending on the preference of the patient.

In addition to the e-CBT program (see 1 above), the participant will receive standard pharmacotherapy following DSM-5 guidelines. A pharmacotherapy allocation system has been developed (Figure 1; Figure 2) that follows clinical guidelines. All medications will be prescribed by a psychiatrist on the research team. All medications are a part of the clinical standard of care. The medications will be provided to the participant through the normal process of receiving medication (i.e., pharmacy). Participants allocated to the e-CBT + Phone Call + Pharmacotherapy arm will begin the pharmacotherapy optimization process at the same time as they begin the e-CBT program. Oversight of medication in the e-CBT + Pharmacotherapy arm will be conducted by a psychiatrist on the team who will make a judgement regarding whether to alter the medications. This will not require any additional study visits/time commitment for the participants in this arm.

Inclusion Criteria: Diagnosed with MDD by a trained research assistant according to the criteria outlined in the DSM-5 Ability to provide informed consent Ability to speak and read English Having consistent and reliable access to the internet Exclusion Criteria: Active psychosis Acute mania Severe alcohol, or substance use disorder Active suicidal or homicidal ideation Currently receiving psychotherapy