Treating Comorbid Depression During Care Transitions Using Relational Agents (RA-CBT)

Depression is common among individuals with chronic illness such as chronic heart failure (CHF) or chronic obstructive pulmonary disease (COPD) and is a strong predictor of poor clinical outcomes and high rates of avoidable 30-day readmissions. While depression is a treatable condition, few people with depression receive effective treatment. The proposed study aims to develop a relational agent system to deliver depression treatment to patients with chronic illness and comorbid depression. While depression is a treatable condition, less than 36% of people with depression receive effective treatment. The proposed study aims to design and develop a technology-driven relational agent system to deliver a 6 module, blended approach of cognitive behavioral therapy and self-management education (RA-CBT) to support patients with CHF or COPD and comorbid depression. The relational agent is a human animation program that interacts with patients, integrates best practices from provider-patient communication theory, emulating the face-to-face conversational behavior of an empathic provider emphasizing nonverbal communicative behavior such as gaze, posture, gestures, etc. The RA-CBT program will be accessed via tablet technology by participants. The study investigators will conduct a pre-post feasibility study to understand the feasibility and acceptability of using the RA-CBT system and its effectiveness in treating comorbid depression. If successful, this new approach to depression treatment would immediately expand access and scalability for post-discharge mental health support in the care transition.

Depression is a treatable disorder with effective pharmacologic and psychotherapy options for treatment. Evidence suggests that psychotherapy interventions such as cognitive behavior therapy (CBT) and self-management education (Lorig) can improve depression, physical symptoms and quality of life in patients with chronic medical conditions.While depression is treatable, less than 36% of patients afflicted with mood disorders get treatment. In order to meet the imminent need for post-discharge mental health support for patients with chronic illness, innovative approaches to delivering mental healthcare that are accessible on short notice, are urgently needed. The study investigators now aim to translate the RED-D intervention to an automated system to expand immediate access and scalability for this support service. Cognitive behavioral therapy for depression (CBT) and self-management education have been studied for the treatment of comorbid depression in chronically ill patients. CBT is a psychosocial intervention that posits that depressive symptoms occur when individuals have a dysfunctional view of themselves, the world and the future. CBT teaches techniques to examine dysfunctional beliefs that perpetuate depression and to develop more accurate and helpful ones to alleviate that condition. It has largely been documented that CBT is among the most effective treatments for depression and shown to be more effective than other psychosocial treatments, as effective as pharmacotherapy and as effective as adding another medication for patients who do not respond to one antidepressant alone. CBT has also been successfully used to treat comorbid depression; the ENRICHD trial demonstrated that a collaborative care approach, hinged on CBT, is an effective treatment for depressive symptoms among post MI patients and superior to usual care. Several studies have also demonstrated the efficacy of self-management education. For example, Lorig et al showed that self-management education enhances self-efficacy and reduces health service utilization among patients with chronic illness. This and similar programs include key processes of identifying and addressing problems, problem solving skills training, action plan design, and applying skills to medical and psychosocial aspects of chronic illness. However, self-management education alone shows limited efficacy for chronically ill patients with significant depression. A few interventions have combined CBT for depression with self-management support. Culley et al showed that a brief CBT intervention integrating self-management strategies reduced both depressive and dyspnea symptoms in a trial with veterans diagnosed with either CHF or COPD in a primary care setting. This evidence suggests a blended and brief skills-based approach to treatment that addresses both physical symptoms and depression is more effective for depressed patients with chronically illness. However, conventional CBT has limitations. Traditionally, CBT is delivered in a face-to-face format requiring a patient to meet with a therapist in an office setting for weekly sessions for 1 weeks or more. Adherence can be a barrier for CBT due to factors related to stigma, transportation, or access to trained CBT clinicians. This has led to trials of alternative formats for delivering CBT including both telephone delivered CBT and computer-based CBT (cCBT), both of which are shown to increase adherence to treatment compared with face-to-face CBT. Many cCBT models are in use today because of their numerous technological advantages including cost effectiveness, increased access and flexibility, and also reduced stigma for those seeking treatment for mental health conditions. cCBT is also shown to be effective for people over the age of 50, suggesting that cCBT is feasible with older patients. However, there are still several limitations to cCBT that can impact efficacy. For example, current cCBT systems largely rely on textbased interactions, creating a potential literacy barrier for some users. Similar to face-to-face CBT, nonadherence is also a problem for cCBT. Sustained adherence is essential to reach a successful therapeutic outcome with cCBT. Yet, despite their promise, evidence shows that outside of clinical trial conditions, many cCBT users do not complete the intervention, thus adherence is still a problem, although cCBT systems with embedded adherence support messages such as weekly reminders have demonstrated higher rates of program completion. The study investigators propose to automate delivery of AHRQ's RED-D intervention and extend it with integrated self-management education for patients with CHF or COPD and comorbid depression following hospitalization. This intervention will be delivered using Relational Agent (RA) technology. Relational agents are animated counselors that provide patients with virtual consultations by simulating face-to-face conversation with an empathic provider. RAs provide an ideal medium for the CBT intervention because of: 1) their demonstrated ability to address non-adherence; 2) empathic communication features; 3) the ability to tailor the intervention to the individual patient; 4) the study investigators' past work demonstrating that hospitalized patients with depressive symptoms report high acceptance of the RA; and 5) the potential to address mental healthcare access barriers. Similar efforts using RA technology are being explored by the US Department of Defense to address the burden of PTSD among war veterans. Traditional CBT has been effective for medically ill patients but limited in its impact on physical symptom experiences and therefore quality of life and healthcare utilization. Cully et al have shown a brief, face-to-face CBT intervention integrating self-management education is more effective for medically ill patients in primary care for reducing depression symptoms and improving quality of life. The relational agent CBT (RA-CBT) system will include six modules tailored for chronically ill patients with comorbid depression, including key components of face-to-face CBT. The final session will include a follow up exit interview to understand the patinet participant's experience with the RA-CBT program and to evaluate the effectiveness of the RA-CBT agent. The final session will incorporate a quantitative questionnaire and a voluntary qualitative interview. The RA-CBT program will include the following topics: cognitive restructuring (identify behaviors, emotions, and thoughts associated with chronic disease and depression, evaluate them and identify more helpful ones), problem solving, behavioral activation, adherence with medical appointments and medication management, and disease self-management. The disease self-management module will address relaxation, sleep hygiene and physical symptom management. Each module will contain a homework component. The RA-CBT protocol will be empathic, reflective and interactive, enhancing the possibility to develop engagement, increase adherence, and establish a trusting relationship and therapeutic alliance. Whereas current cCBT users cannot ask questions, the study team's RA-CBT intervention will make use of psychosocial technology that will allow the RA to interact with and respond to questions that arise. Drs. Bickmore, Jack and Mitchell have conducted a series of studies using RA technology for health education and behavior change, including inpatient education (AHRQ grant 1R18HS017196-01 and AHRQ RFTO "Communication- Focused Technologies") and counseling for patients of all levels of health literacy. Dr. Bickmore and his team at Northeastern University have extensive experience designing and implementing RA psychosocial technology that simulates face-to-face conversation that is sensitive to the qualities of CBT that help establish an empathetic relationship, honing the use of social interaction with conversational technology in order to build trust and a therapeutic alliance between the agent and patient. There are many reasons why RAs provide an effective medium for health communication with patients, especially those with limited health literacy. The human-computer interface relies minimally on text (as compared to current cCBT models) and uses the universally understood format of face-to-face conversation, making it accessible to patients with limited reading ability. Most importantly, RAs enhance recall of critical information through the use of nonverbal conversational behaviors (gesture, tone) by providing redundant channels for conveying semantic content in order to enhance the likelihood of message comprehension. In addition, all cultures have nonverbal means for marking emphasis (for example, eyebrow raising and 'beat' or 'baton' hand gestures in American English) that may highlight salient parts of a message. Unlike a pre-recorded video lecture, the study team's RA-CBT intervention provides a flexible and dynamic communication medium that relies on synthetic speech tailored to the patient and the discourse context using language dynamically assembled using a dialogue engine (i.e. use of the patient's name, medical record information, and other personally relevant information) that responds to the context of the conversation (i.e. the questions asked, patient responses during earlier sessions, etc.) Patient contributions to the dialogue are made by selecting what they want to say from a multiple choice menu, dynamically updated for each turn of the conversation (no natural language understanding of unconstrained text or speech is performed). Alternative technologies do not offer this adaptive, face-to-face encounter that accommodates patients' needs. This interface has been used by over 2,500 patients including many with no prior computer experience.

Chronic Illness, Chronic Pain, Depression, Comorbid Depression, Cognitive Behavioral Therapy, Relational Agent Technology, CBT, Virtual Counselor

The pre-post feasibility testing/ intervention includes the use of tablet technology and cognitive behavioral therapy (CBT) for depression and self-management education for patients with chronic illness and/or chronic pain and depression. The pre-post feasibility testing phase will include 25 subjects. The total duration of pre-post feasibility testing will be 6 weeks upon the retrieval of the RA-CBT tablet and the inclusion of follow up questionnaires, quantitative exit interview, and/or an optional extended qualitative exit interview.

The feasibility testing phase will include 25 subjects. Total duration of pre-post feasibility testing will be 6 weeks upon retrieval of the RA-CBT tablet and follow up questionnaires, quantitative exit interview, and/or an optional extended qualitative exit interview. The extended exit interview is a qualitative semi-formal interview focusing on gathering participant experiences working with the intervention, the barriers and facilitators encountered, as well as overall feedback for the pre-post feasibility study and the relational agent.

Primary outcome is the change in depression score measured by the PHQ-8 at the exit interview after the completion of the 6 weeks with the RA-CBT Program. The study investigators hypothesize that there will be a decrease in depressive symptoms at the end of 6 weeks or at the completion of the intervention.

A secondary outcome is a positive change in health-related quality of life and functional status measured by the Medical Outcomes Study Short Form 12 (SF-12), which has been validated in patients with CHF, COPD, chronic pain due to advanced osteoarthritis, inoperable spinal stenosis, and chronic headaches. The SF-12 will measure physical functionality, role limitations due to physical health problems, bodily pain, general health, vitality (energy/fatigue), social functioning, role limitations due to emotional problems and mental health (psychological distress and psychological well-being).

A secondary outcome is an increase in how activated the patient is in regards to their health and health care treatment, measured by the Patient Activation Measure (PAM-13).

A secondary outcome is a positive change in quality of life enjoyment and satisfaction measured by the Quality of Life Enjoyment and Satisfaction Questionnaire- Short Form.

A secondary outcome is an increase in quality of life satisfaction measured by the Work and Social Adjustment Scale (WSAS).

A secondary outcome is assessment of users' attitudes towards working with the RA-CBT agent on health related goals using the Working Alliance Inventory (WAI).

A secondary outcome includes the adherence to treatment with RA-CBT technology among intervention participants. There will be assessment of the proportion of participants who complete all 6 modules within 6 weeks. There will be examination of the proportion who failed to engage in treatment (<2 sessions or modules) and proportion who failed to complete treatment (>2 but <6). The study investigators will also examine the length of time that participants are engaged with the overall program, per module, and per bonus activity.

A secondary outcome is the assessment of user's satisfaction with the intervention and relational agent measured by the Client Satisfaction Questionnaire (CSQ-8)

Inclusion Criteria: The study intervention will include both men and women admitted to Boston Medical Center in the last 24 hours. The study investigators will enroll those who are: over 18 years old, and/or admitted to the general medical or cardiology service of BMC with primary diagnosis of CHF or COPD exacerbation and/or patients of ambulatory care clinics, specialty care clinics and/or chronic pain groups of BMC Family Medicine Health Centers and/ or patients of Behavioral Health in the Department of Family Medicine and/or BMC Family Medicine Health Centers and/or participants recruited from BU/BMC ecommunications and community settings such as community centers, recreational centers, senior centers, and weight loss centers. and/or have ever been diagnosed with any symptoms of chronic illness such as Chronic Heart Failure (CHF) and/or Chronic Obstructive Pulmonary Disease speak English with health providers, have access to a telephone have permanent housing and/or are not housed within shelter programs Live in the Boston area, and do not plan on leaving the Boston area for more than 2 weeks in the next 6 months are able to independently consent. If patients meet these criteria, the PHQ-2 will be administered during feasibility screening. Those who score ≥3 on the PHQ-2 during this pre-consent screening and will be consented. The PHQ-2 score will carry over into post-consent screening of the feasibility testing enrollment form. The PHQ-8 will be administered post-consent during the enrollment process. Those who score ≥5 on the PHQ-8 and meet the other post-consent screening on the enrollment form will be enrolled in the study. Exclusion Criteria: Using clear protocols, the study investigators will identify those who do not have indications for CBT amd will exclude: Under 18 years of age Unable to speak English with health providers Those who do not have indications for CBT (e.g., active substance abuse, dementia, bipolar disease, schizophrenia, psychotic symptoms, prior history of ECT, among others based on Mini International Neuropsychiatric Inventory). Those who are suicidal or homicidal, Those who are currently in police custody and/or incarcerated, Those who do not live in the Boston (or MA) area, or those who plan on leaving the Boston area for more than 2 weeks in the next 6 months. If they do not live in permanent housing and/or live in shelter programs, Those who are already engaged in psychotherapy Those who are currently pregnant or planning to become pregnant during the course of the study Those who are receiving current treatment for active cancer diagnosis or who have been diagnosed with sickle cell anemia. The study investigators will not approach those patients who are on neurosurgery, hematology/ oncology, and/or trauma service.

Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009 Apr 2;360(14):1418-28. doi: 10.1056/NEJMsa0803563. Erratum In: N Engl J Med. 2011 Apr 21;364(16):1582.

Jack BW, Chetty VK, Anthony D, Greenwald JL, Sanchez GM, Johnson AE, Forsythe SR, O'Donnell JK, Paasche-Orlow MK, Manasseh C, Martin S, Culpepper L. A reengineered hospital discharge program to decrease rehospitalization: a randomized trial. Ann Intern Med. 2009 Feb 3;150(3):178-87. doi: 10.7326/0003-4819-150-3-200902030-00007.

Mitchell SE, Paasche-Orlow MK, Forsythe SR, Chetty VK, O'Donnell JK, Greenwald JL, Culpepper L, Jack BW. Post-discharge hospital utilization among adult medical inpatients with depressive symptoms. J Hosp Med. 2010 Sep;5(7):378-84. doi: 10.1002/jhm.673.

Owens P, Myers M, Elixhauser A, Brach C, eds. Care of Adults with Mental Health and Substance Abuse Disorders in U.S. Community Hospitals, 2004. HCUP Factbook No. 10 Publication No. 07-0008; 2007.

Silver MA. Depression and heart failure: an overview of what we know and don't know. Cleve Clin J Med. 2010 Jul;77 Suppl 3:S7-S11. doi: 10.3949/ccjm.77.s3.02.

Sullivan M, Simon G, Spertus J, Russo J. Depression-related costs in heart failure care. Arch Intern Med. 2002 Sep 9;162(16):1860-6. doi: 10.1001/archinte.162.16.1860.

Dowson CA, Town GI, Frampton C, Mulder RT. Psychopathology and illness beliefs influence COPD self-management. J Psychosom Res. 2004 Mar;56(3):333-40. doi: 10.1016/S0022-3999(03)00040-0.

Cuijpers P, van Straten A, van Oppen P, Andersson G. Are psychological and pharmacologic interventions equally effective in the treatment of adult depressive disorders? A meta-analysis of comparative studies. J Clin Psychiatry. 2008 Nov;69(11):1675-85; quiz 1839-41. doi: 10.4088/jcp.v69n1102. Epub 2008 Aug 12.

Wolf NJ, Hopko DR. Psychosocial and pharmacological interventions for depressed adults in primary care: a critical review. Clin Psychol Rev. 2008 Jan;28(1):131-161. doi: 10.1016/j.cpr.2007.04.004. Epub 2007 Apr 29.

NICE Clinical Guideline 90 - Depression - The Treatment and Management of Depression in Adults. http://www.nice.org.uk/nicemedia/live/12329/45888/45888.pdf.

Cully JA, Paukert A, Falco J, Stanley M. Cognitive-behavioral therapy: Innovations for cardiopulmonary patients with depression and anxiety. Cognitive and Behavioral Practice. 2009; 16(4):394-407.

Kunik ME, Veazey C, Cully JA, Souchek J, Graham DP, Hopko D, Carter R, Sharafkhaneh A, Goepfert EJ, Wray N, Stanley MA. COPD education and cognitive behavioral therapy group treatment for clinically significant symptoms of depression and anxiety in COPD patients: a randomized controlled trial. Psychol Med. 2008 Mar;38(3):385-96. doi: 10.1017/S0033291707001687. Epub 2007 Oct 9.

National Institute of Mental Health. Statistics Web Site. http://www.nimh.nih.gov/statistics/index.shtml. Accessed June 03, 2013.

Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. JAMA. 2002 Nov 20;288(19):2469-75. doi: 10.1001/jama.288.19.2469.

Kunik ME, Braun U, Stanley MA, Wristers K, Molinari V, Stoebner D, Orengo CA. One session cognitive behavioural therapy for elderly patients with chronic obstructive pulmonary disease. Psychol Med. 2001 May;31(4):717-23. doi: 10.1017/s0033291701003890.

Brenes GA. Anxiety and chronic obstructive pulmonary disease: prevalence, impact, and treatment. Psychosom Med. 2003 Nov-Dec;65(6):963-70. doi: 10.1097/01.psy.0000097339.75789.81.

Kostis JB, Rosen RC, Cosgrove NM, Shindler DM, Wilson AC. Nonpharmacologic therapy improves functional and emotional status in congestive heart failure. Chest. 1994 Oct;106(4):996-1001. doi: 10.1378/chest.106.4.996.

Allen NB. Cognitive therapy of depression. Aaron T Beck, A John Rush, Brian F Shaw, Gary Emery. New York: Guilford Press, 1979. Aust N Z J Psychiatry. 2002 Apr;36(2):275-8. doi: 10.1046/j.1440-1614.2002.t01-5-01015.x. No abstract available.

Lynch D, Laws KR, McKenna PJ. Cognitive behavioural therapy for major psychiatric disorder: does it really work? A meta-analytical review of well-controlled trials. Psychol Med. 2010 Jan;40(1):9-24. doi: 10.1017/S003329170900590X. Epub 2009 May 29.

Tolin DF. Is cognitive-behavioral therapy more effective than other therapies? A meta-analytic review. Clin Psychol Rev. 2010 Aug;30(6):710-20. doi: 10.1016/j.cpr.2010.05.003. Epub 2010 May 25.

Roshanaei-Moghaddam B, Pauly MC, Atkins DC, Baldwin SA, Stein MB, Roy-Byrne P. Relative effects of CBT and pharmacotherapy in depression versus anxiety: is medication somewhat better for depression, and CBT somewhat better for anxiety? Depress Anxiety. 2011 Jul;28(7):560-7. doi: 10.1002/da.20829. Epub 2011 May 23.

Thase ME, Friedman ES, Biggs MM, Wisniewski SR, Trivedi MH, Luther JF, Fava M, Nierenberg AA, McGrath PJ, Warden D, Niederehe G, Hollon SD, Rush AJ. Cognitive therapy versus medication in augmentation and switch strategies as second-step treatments: a STAR*D report. Am J Psychiatry. 2007 May;164(5):739-52. doi: 10.1176/ajp.2007.164.5.739.

Cowan MJ, Freedland KE, Burg MM, Saab PG, Youngblood ME, Cornell CE, Powell LH, Czajkowski SM; ENRICHD Investigators. Predictors of treatment response for depression and inadequate social support--the ENRICHD randomized clinical trial. Psychother Psychosom. 2008;77(1):27-37. doi: 10.1159/000110057. Epub 2007 Dec 14.

ENRICHD Investigators. Enhancing Recovery in Coronary Heart Disease (ENRICHD) study intervention: rationale and design. Psychosom Med. 2001 Sep-Oct;63(5):747-55.

Cramm JM, Nieboer AP. Self-management abilities, physical health and depressive symptoms among patients with cardiovascular diseases, chronic obstructive pulmonary disease, and diabetes. Patient Educ Couns. 2012 Jun;87(3):411-5. doi: 10.1016/j.pec.2011.12.006. Epub 2012 Jan 4.

Harrison M, Reeves D, Harkness E, Valderas J, Kennedy A, Rogers A, Hann M, Bower P. A secondary analysis of the moderating effects of depression and multimorbidity on the effectiveness of a chronic disease self-management programme. Patient Educ Couns. 2012 Apr;87(1):67-73. doi: 10.1016/j.pec.2011.06.007. Epub 2011 Jul 20.

Cully JA, Stanley MA, Deswal A, Hanania NA, Phillips LL, Kunik ME. Cognitive-behavioral therapy for chronic cardiopulmonary conditions: preliminary outcomes from an open trial. Prim Care Companion J Clin Psychiatry. 2010;12(4):PCC.09m00896. doi: 10.4088/PCC.09m00896blu.

McNaughton JL. Brief interventions for depression in primary care: a systematic review. Can Fam Physician. 2009 Aug;55(8):789-96.

Cuijpers P, Huibers M, Ebert DD, Koole SL, Andersson G. How much psychotherapy is needed to treat depression? A metaregression analysis. J Affect Disord. 2013 Jul;149(1-3):1-13. doi: 10.1016/j.jad.2013.02.030. Epub 2013 Mar 22.

Cully JA, Armento ME, Mott J, Nadorff MR, Naik AD, Stanley MA, Sorocco KH, Kunik ME, Petersen NJ, Kauth MR. Brief cognitive behavioral therapy in primary care: a hybrid type 2 patient-randomized effectiveness-implementation design. Implement Sci. 2012 Jul 11;7:64. doi: 10.1186/1748-5908-7-64.

Marks IM, Gega L. Review by Jeroen Ruwaard and Alfred Lange (Cognitive Behaviour Therapy, 2009, 38(2), p. 132) of Hands-on-Help: Computer-Aided Psychotherapy (Maudsley Monograph 49) by I. M. Marks, K. Cavanagh, and L. Gega. New York: Psychology Press 2007. Letter to the editors. Cogn Behav Ther. 2009;38(3):192. doi: 10.1080/16506070903162889. No abstract available.

Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res. 2004 Dec 22;6(4):e46. doi: 10.2196/jmir.6.4.e46.

Hoifodt RS, Strom C, Kolstrup N, Eisemann M, Waterloo K. Effectiveness of cognitive behavioural therapy in primary health care: a review. Fam Pract. 2011 Oct;28(5):489-504. doi: 10.1093/fampra/cmr017. Epub 2011 May 9.

Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an Internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry. 2006 Jan;40(1):59-62. doi: 10.1080/j.1440-1614.2006.01743.x.

Spek V, Nyklicek I, Smits N, Cuijpers P, Riper H, Keyzer J, Pop V. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med. 2007 Dec;37(12):1797-806. doi: 10.1017/S0033291707000542. Epub 2007 Apr 30.

Clarke G, Reid E, Eubanks D, O'Connor E, DeBar LL, Kelleher C, Lynch F, Nunley S. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res. 2002 Dec;4(3):E14. doi: 10.2196/jmir.4.3.e14.

McLay RN, Graap K, Spira J, Perlman K, Johnston S, Rothbaum BO, Difede J, Deal W, Oliver D, Baird A, Bordnick PS, Spitalnick J, Pyne JM, Rizzo A. Development and testing of virtual reality exposure therapy for post-traumatic stress disorder in active duty service members who served in Iraq and Afghanistan. Mil Med. 2012 Jun;177(6):635-42. doi: 10.7205/milmed-d-11-00221.

Bickmore TW, Mitchell SE, Jack BW, Paasche-Orlow MK, Pfeifer LM, Odonnell J. Response to a Relational Agent by Hospital Patients with Depressive Symptoms. Interact Comput. 2010 Jul 1;22(4):289-298. doi: 10.1016/j.intcom.2009.12.001.

Bickmore TW, Pfeifer LM, Paasche-Orlow MK. Using computer agents to explain medical documents to patients with low health literacy. Patient Educ Couns. 2009 Jun;75(3):315-20. doi: 10.1016/j.pec.2009.02.007. Epub 2009 Mar 17.

Bickmore T, Pfeifer L, Paasche-Orlow M. Health document explanation by virtual agents. Intelligent Virtual Agents. Paris: 2007:183-196.

Bickmore TW, Pfeifer LM, Byron D, Forsythe S, Henault LE, Jack BW, Silliman R, Paasche-Orlow MK. Usability of conversational agents by patients with inadequate health literacy: evidence from two clinical trials. J Health Commun. 2010;15 Suppl 2:197-210. doi: 10.1080/10810730.2010.499991.

Bickmore T, Gruber A, Picard R. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Educ Couns. 2005 Oct;59(1):21-30. doi: 10.1016/j.pec.2004.09.008.

O'Hea E, Houseman J, Bedek K, Sposato R. The use of cognitive behavioral therapy in the treatment of depression for individuals with CHF. Heart Fail Rev. 2009 Mar;14(1):13-20. doi: 10.1007/s10741-008-9081-2. Epub 2008 Jan 29.

Dekker RL. Cognitive therapy for depression in patients with heart failure: a critical review. Heart Fail Clin. 2011 Jan;7(1):127-41. doi: 10.1016/j.hfc.2010.10.001.

Humeniuk R, Ali R, Babor TF, Farrell M, Formigoni ML, Jittiwutikarn J, de Lacerda RB, Ling W, Marsden J, Monteiro M, Nhiwatiwa S, Pal H, Poznyak V, Simon S. Validation of the Alcohol, Smoking And Substance Involvement Screening Test (ASSIST). Addiction. 2008 Jun;103(6):1039-47. doi: 10.1111/j.1360-0443.2007.02114.x. Epub 2008 Mar 28.

McNair S, Checchi K, Rubin A, Marcello T, Bickmore T, Simon S. A pilot study of a computer-based relational agent to screen for substance-use problems in primary care. Society of Behavioral Medicine Annual Meeting, 2013.

Bickmore TW, Silliman RA, Nelson K, Cheng DM, Winter M, Henault L, Paasche-Orlow MK. A randomized controlled trial of an automated exercise coach for older adults. J Am Geriatr Soc. 2013 Oct;61(10):1676-83. doi: 10.1111/jgs.12449. Epub 2013 Sep 3.

King AC, Bickmore TW, Campero MI, Pruitt LA, Yin JL. Employing virtual advisors in preventive care for underserved communities: results from the COMPASS study. J Health Commun. 2013;18(12):1449-64. doi: 10.1080/10810730.2013.798374. Epub 2013 Aug 13.

Watson A, Bickmore T, Cange A, Kulshreshtha A, Kvedar J. An internet-based virtual coach to promote physical activity adherence in overweight adults: randomized controlled trial. J Med Internet Res. 2012 Jan 26;14(1):e1. doi: 10.2196/jmir.1629.

Bickmore T, Schulman D, Yin L. Maintaining Engagement in Long-term Interventions with Relational Agents. Appl Artif Intell. 2010 Jul 1;24(6):648-666. doi: 10.1080/08839514.2010.492259.

Velicer W, Reading C, Blissmer B, et al. Using relational agents in tailored interventions for multiple risk factors: Preliminary 12 month results. Society of Behavioral Medicine Annual Meeting, 2013.

Pfeifer L, Bickmore T. Longitudinal remote follow-up by intelligent conversational agents for posthospitalization care. AAAI Spring Symposium on AI and Health Communication. 2011.

Gardiner P, Hempstead MB, Ring L, Bickmore T, Yinusa-Nyahkoon L, Tran H, Paasche-Orlow M, Damus K, Jack B. Reaching women through health information technology: the Gabby preconception care system. Am J Health Promot. 2013 Jan-Feb;27(3 Suppl):eS11-20. doi: 10.4278/ajhp.1200113-QUAN-18.

Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606-13. doi: 10.1046/j.1525-1497.2001.016009606.x.

Sheehan DV, Lecrubier Y, Sheehen KH, Janavas J, Weiler E. Comparison of the Mini International Neuropsychiatric Interview (MINI) with the SCID-P and the CIDI: A validity study (abstract). Psychopharmacology Bulletin. 1995; 31:616.

Ware JE Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992 Jun;30(6):473-83.

Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004 Aug;39(4 Pt 1):1005-26. doi: 10.1111/j.1475-6773.2004.00269.x.

Beck AT, Steer RA, Brown GK. Manual for the Beck Depression Inventory. 2nd ed. San Antonio: Harcourt Brace; 1996.

Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann. 1982;5(3):233-7. doi: 10.1016/0149-7189(82)90074-x.

Horvath A, Greenberg L. Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology. 1989 Apr; 36(2):223-233.

Lowe B, Kroenke K, Herzog W, Grafe K. Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). J Affect Disord. 2004 Jul;81(1):61-6. doi: 10.1016/S0165-0327(03)00198-8.