Systematic Reviews
Copyright ©The Author(s) 2016. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Psychiatr. Jun 22, 2016; 6(2): 269-282
Published online Jun 22, 2016. doi: 10.5498/wjp.v6.i2.269
Review of key telepsychiatry outcomes
Sam Hubley, Sarah B Lynch, Christopher Schneck, Marshall Thomas, Jay Shore
Sam Hubley, Christopher Schneck, Marshall Thomas, Jay Shore, Helen and Arthur E Johnson Depression Center, Department of Family Medicine, Aurora, CO 80045, United States
Sarah B Lynch, Department of Clinical Health Psychology, University of Colorado Denver, Denver, CO 80217-3364, United States
Author contributions: All authors conceived the manuscript and collaborated on the design of the review.
Conflict-of-interest statement: The authors have no conflicts of interest to disclose.
Data sharing statement: As this was a review of published data there were no participants to be approached for informed consent for data sharing. No additional data are available.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Correspondence to: Sam Hubley, Assistant Professor, Helen and Arthur E Johnson Depression Center, Department of Family Medicine, 13199 E. Montview Blvd, Suite 330, Aurora, CO 80045, United States. samuel.hubley@ucdenver.edu
Telephone: +1-303-724882 Fax: +1-303-7244698
Received: October 28, 2015
Peer-review started: November 3, 2015
First decision: December 4, 2015
Revised: January 27, 2016
Accepted: March 14, 2016
Article in press: March 15, 2016
Published online: June 22, 2016

Abstract

AIM: To conduct a review of the telepsychiatry literature.

METHODS: We conducted a systematic search of the literature on telepsychiatry using the search terms, “telepsychiatry”, “telemental health”, “telecare”, “telemedicine”, “e-health”, and “videoconferencing”. To meet criteria for inclusion, studies had to: (1) be published in a peer-reviewed journal after the year 2000; (2) be written in English; (3) use videoconferencing technology for the provision of mental health assessment or treatment services; and (4) use an adequately-powered randomized controlled trial design in the case of treatment outcome studies. Out of 1976 studies identified by searches in PubMed (Medline database), Ovid medline, PsychInfo, Embase, and EBSCO PSYCH, 452 met inclusion criteria. Studies that met all inclusion criteria were organized into one of six categories: (1) satisfaction; (2) reliability; (3) treatment outcomes; (4) implementation outcomes; (5) cost effectiveness; and (6) and legal issues. All disagreements were resolved by reassessing study characteristics and discussion.

RESULTS: Overall, patients and providers are generally satisfied with telepsychiatry services. Providers, however, tend to express more concerns about the potentially adverse of effects of telepsychiatry on therapeutic rapport. Patients are less likely to endorse such concerns about impaired rapport with their provider. Although few studies appropriately employ non-inferiority designs, the evidence taken together suggests that telepsychiatry is comparable to face-to-face services in terms of reliability of clinical assessments and treatment outcomes. When non-inferiority designs were appropriately used, telepsychiatry performed as well as, if not better than face-to-face delivery of mental health services. Studies using both rudimentary and more sophisticated methods for evaluating cost-effectiveness indicate that telepsychiatry is not more expensive than face-to-face delivery of mental health services and that telepsychiatry is actually more cost-effective in the majority of studies reviewed. Notwithstanding legal concerns about loss of confidentiality and limited capacity to respond to psychiatric emergencies, we uncovered no published reports of these adverse events in the use of telepsychiatry.

CONCLUSION: A large evidence base supports telepsychiatry as a delivery method for mental health services. Future studies will inform optimal approaches to implementing and sustaining telepsychiatry services.

Key Words: Telepsychiatry, Telemental health, Videoconferencing, Treatment access, Implementation

Core tip: Telepsychiatry represents a highly promising approach to reducing the treatment gap by making it easier for patients, especially those in isolated contexts, to access expert mental health care. There is a robust evidence base for the use of telepsychiatry as a delivery method for mental health services. Given sufficient empirical justification for telepsychiatry in routine clinical settings, future research studies should focus on clarifying best practices for implementing and sustaining telepsychiatry services.


Citation: Hubley S, Lynch SB, Schneck C, Thomas M, Shore J. Review of key telepsychiatry outcomes. World J Psychiatr 2016; 6(2): 269-282
INTRODUCTION

Innovative approaches to delivering mental health services are urgently needed to increase access to evidence-based care. Telepsychiatry, which in its contemporary use refers to the delivery of mental health services via video-based conferencing, has great potential to address mental health disparities by extending the reach of mental health care to those living in rural areas or to those who otherwise have limited access to care. Rapid changes in technology and the medical landscape have undoubtedly accelerated the growth and reach of telepsychiatry. The enthusiasm for the delivery of mental health services via telepsychiatry is evident in its adoption in large health care organizations such as the United States Department of Veterans Affairs[1] and establishment of national practice guidelines[2].

Prior reviews have focused on different domains of the evidence based for telepsychiatry[3,4]. We sought to build on prior reviews by systematically reviewing and critically summarizing the evidence base for telepsychiatry. For the purposes of this review, telepsychiatry is defined as the provision of mental health services by a mental health professional via videoconferencing technology.

MATERIALS AND METHODS

We conducted a review of the telepsychiatry literature in PubMed (Medline database), Ovid medline, PsychInfo, Embase, and EBSCO PSYCH. We used the keywords, “telepsychiatry”, “telemental health”, “telecare”, “telemedicine”, “e-health”, and “videoconferencing”. To meet criteria for inclusion, studies had to: (1) be published in a peer-reviewed journal after the year 2000; (2) be written in English; (3) use videoconferencing technology for the provision of mental health assessment or treatment services; and (4) use an adequately-powered randomized controlled trial (RCT) design in the case of treatment outcome studies. Additionally, we searched reference lists of included studies to identify additional publications not captured by our literature search. The last search was conducted in June 2015.

The first two authors (Sam Hubley and Sarah B Lynch) reviewed all abstracts to identify eligible studies. Studies that met all inclusion criteria were organized into one of six categories: (1) satisfaction; (2) reliability; (3) treatment outcomes; (4) implementation outcomes; (5) cost effectiveness; and (6) and legal issues. All disagreements were resolved by reassessing study characteristics and discussion. When consensus was not reached between the first two authors, the last author (Jay Shore) made the final decision.

RESULTS

We identified a total of 1668 full-text articles based on our literature search and excluded 1534 based on the inclusion criteria (Figure 1). Of the remaining 134, 86 reported on satisfaction with telepsychiatry, 38 evaluated reliability of clinical assessments conducted via telepsychiatry, 32 were RCTs, 43 reported on implementation outcomes, 29 estimated cost-effectiveness, and 23 evaluated legal issues associated with telepsychiatry mental health services. Note that some studies reported on more than one outcome (n = 117) and were thus included in each relevant section.

Figure 1
Figure 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.
Satisfaction

Adequate patient and provider satisfaction is a prerequisite for wide scale implementation of telepsychiatry. This is especially true in light of the emphasis on patients’ experience of care as a key component of the Triple Aim Framework[5], by which many healthcare innovations are evaluated. There is a substantial body of literature focusing on patient and provider satisfaction with telepsychiatry. Studies employed a range of descriptive, qualitative, experimental, and mixed-methods designs to assess satisfaction outcomes.

Patient satisfaction: The majority of studies summarize patients’ responses to quantitative self-report questionnaires with descriptive statistics and report high satisfaction with telepsychiatry services. The Client Satisfaction Questionnaire[6] is a commonly used measure in these studies and consists of 8 items rated on a 5-point Likert scale. Many investigators also developed their own satisfaction measures that consist of similar items rated on a Likert-scale. Of the 31 studies reviewed, 23 concluded that patients rated their experience with telepsychiatry services as “good” to “excellent”[7-28], while the remaining 7 studies reported mixed reactions among participants[29-38]. For example, Hilty et al[33] reported generally high satisfaction among outpatients seeking specialty mental health care but satisfaction scores were statistically higher for rural patients compared to suburban patients. Studies that used qualitative (n = 6) qualitative methods to better understand patients’ experiences with telepsychiatry services suggest a less uniform pattern of findings compared to results from studies using quantitative methods only[32,34,37,39,40]. Participants’ responses to individual interviews and focus groups follow themes characterized by both positive and negative reactions to telepsychiatry. Prominent positive themes include ease of use and decreased burden of transportation to and from appointments; whereas prominent negative themes include privacy concerns, challenging to establish patient-doctor relationship, and technical challenges. Finally, some studies (n = 7) used an experimental design to assess patient satisfaction[10,21,26,36,41-43]. In a study that compared reactions of 48 outpatients randomized to telepsychiatry or face-to-face (FTF) psychiatric consultation, telepsychiatry patients reported comfort in disclosing the same information they would disclose in FTF consultation but reported slightly lower levels of satisfaction regarding feeling supported and encouraged than did FTF patients[36].

Provider satisfaction: Based on 11 studies that use qualitative self-report methods, providers tend to have mixed reactions to telepsychiatry[25,33,34,44-52]. Some studies have shown that adult and child psychiatrists reported adequate to high satisfaction with telepsychiatry[53] and one study demonstrated that mental health providers prefer telepsychiatry over telephone-based consultation[51]. Allied providers such as primary care providers (PCPs)[49] and emergency room providers[25,44] have also expressed satisfaction with telepsychiatry. Other studies have yielded mixed results as rural PCPs are more satisfied with telepsychiatry than are PCPs based in suburban locations[33], providers perceive patients to be less satisfied with telepsychiatry services than actual patient report[41] and provider concerns that their lack of experience delivering telepsychiatry may result in lower levels of care[54]. Finally, some investigators have documented negative reactions to telepsychiatry as providers are resistant to use telepsychiatry[55] and concern that perceived technological challenges associated with telepsychiatry may hinder doctor-patient interactions[56].

Qualitative interviews yield similarly mixed results as some allied health providers report satisfaction with telepsychiatry services[44], while others[45,46] express concerns about the potential adverse impacts of telepsychiatry on the therapeutic alliance and cited barriers such as difficulties incorporating telepsychiatry into their practice, including difficulty accessing trainings for telepsychiatry, and lack of resources. Experimental studies have confirmed the finding that providers are not uniformly satisfied with telepsychiatry services, based in large part about concerns over therapeutic rapport. For example, in an RCT comparing cognitive behavioral therapy (CBT) for bulimia nervosa delivered FTF and via telepsychiatry, patients and psychotherapists completed ratings of the therapeutic alliance[57]. Results showed that psychotherapists generally reported lower alliance scores with telepsychiatry participants than FTF participants whereas there no differences between alliance scores among FTF and telepsychiatry participants’ own evaluations of the therapeutic alliance.

Summary of patient and provider satisfaction: The evidence to date on patient and provider satisfaction is generally positive as most studies demonstrate that providers and patients find telepsychiatry acceptable. In general, (1) patients tend to report higher satisfaction than providers, and this appears to be especially true among parents seeking services for their children; (2) patients acknowledge potential changes to the therapeutic alliance inherent in providing services remotely but are less concerned than providers; (3) allied health professionals (e.g., PCPs and ED providers) report high satisfaction with telepsychiatry provided the technology works well and does not majorly interfere with workflows; and (4) study design appears to influence outcome as studies that used purely descriptive methods tend to report more positive outcomes than studies using qualitative and experimental methods.

Reliability

The reliability of assessments conducted via telepsychiatry compared to assessments conducted via the gold-standard of FTF interviews is a key area of research. The primary variables of interest are inter-rater reliability and inter-method reliability. Inter-rater reliability refers to the degree of agreement between two raters assessing the same patient with the same assessment tools, whereas inter-method reliability refers to the degree of agreement between one rater using one assessment tool with two different methods (i.e., telepsychiatry vs FTF). This area of research represents a hybrid between inter-rater and inter-method reliability.

Several statistical approaches are used to quantify the degree of agreement between two different ratings. Most relevant to evaluating reliability in telepsychiatry assessments are the kappa statistic, correlation coefficients, and the intra-class correlation coefficient (ICC). The kappa statistic assesses agreement on categorical outcomes (i.e., whether or not a patient receives a diagnosis of bipolar disorder) and Pearson’s r is a correlation coefficient that assesses agreement on continuous outcomes such as total scores on the Hamilton Depression Rating Scale (HAMD)[58]. Finally, the ICCs is a more sophisticated version of Pearson’s r because it accounts for variance in assessments due to true between-subject variability in addition to variance due to disagreement between the raters.

We identified 21 studies that evaluated reliability of clinical assessments conducted via telepsychiatry. Two studies compared the accuracy of diagnoses made using the SCID[59,60], two studies evaluated reliability of child assessments[42,61], four studies evaluated reliability of neuropsychological assessments[62-65], and three studies evaluated reliability of measures of depressive symptom severity[66-68]. The remaining studies evaluated a range of targets such as alcohol use severity[69], diagnostic accuracy[70-72], competency to stand trial[73], psychosis[26], and adult autism[21]. The majority of studies report moderate to high level of agreement between raters using telepsychiatry and FTF regardless of instrument, provider, or setting type.

Several authors used clever study designs to evaluate reliability at a more nuanced level. For example, two studies assessed reliability estimates between telepsychiatry conducted with varying levels of bandwidth to determine if connection quality impacted reliability. One report observed ICCs greater than 0.95 across all conditions of varying bandwidth quality among two psychiatrists rating eight psychiatric outpatients and nine health controls using the BPRS[74]. On the other hand, Yoshino et al[75] observed statistically significantly lower ICC scores in the narrow bandwidth condition compared to the broad bandwidth in for BPRS scores. In a rigorous comparison of inter-rater reliability that reduced artificial inflation of reliability estimates by requiring interviewers to generate their own clarifying questions and probes, Kobak et al[66-68] observed very high reliability estimates between four psychiatrists at three different locations using the HAMD via telepsychiatry or FTF[67]. And although diagnostic assessments via telepsychiatry appear to be reliable even when using interpreters[60], there are mixed results of the reliability of telepsychiatry assessments when visual information is required. For example, Jones et al[76] found that raters assessing a geriatric population on a psychiatric inpatient unit achieved lower reliability on the observational items based on participants’ behavior than the subjective items based on participants’ self-report. This difference in Kappa scores was statistically significant for assessments conducted via telepsychiatry, but not for assessments conducted FTF[76]. Conversely, Amarendran et al[62] found that assessment of abnormal movements among patients with at least 10 years of antipsychotic medication exposure was not significantly less reliable when using telepsychiatry compared to FTF.

Summary of reliability estimates in telepsychiatry: The studies included in this review suggest that, in general, assessments made via telepsychiatry are comparable to FTF assessments in terms of reliability. No studies provided strong evidence that telepsychiatry assessments are significantly less reliable than FTF assessments. Studies that have not reported uniformly high reliability estimates for telepsychiatry assessments represent important caveats. First, adequate reliability may be contingent on bandwidth quality as the quality of observations required for a given assessment decreases as video and audio quality deteriorate. Second, the use of interpreters does not appear to reduce the reliability of telepsychiatry assessments but more studies that attempt to replicate this finding are needed. Third, assessments that require objective observations (i.e., the Brief Psychiatric Rating Scale) may be more difficult to conduct via telepsychiatry.

Treatment outcomes

The crucial question in telepsychiatry research is whether or not mental health interventions delivered via telepsychiatry can achieve similar outcomes compared to interventions delivered FTF. We identified 13 RCTs that evaluated treatment outcomes for mental health interventions delivered via telepsychiatry (Table 1). Seven studies targeted depression[77-83], two targeted symptoms of PTSD[11,84,85], two targeted ADHD[86], one targeted bulimia nervosa[87], and two targeted common psychiatric disorders presented in an outpatient medical and mental health settings.

Table 1 Review of telepsychiatry randomized controlled trials.
Ref.Participants
Interventions
Results
nRecruitment sourceMean age (SD)Target disorderConditionsDuration (# of visits)ProviderAttritionFindings
Nelson et al[82]38Urban schools10.3 (2.0)Childhood depressionCBT TP8dnr26%Child Depression Inventory scores reduced from 14.36 (SD = 9.85) at baseline to 6.71 (SD = 4.78) at post-treatment for CBT TP and from 13.57 (SD = 8.75) to 11.64 (SD = 4.78) for CBT FTF [Wilks’ L (1, 26) = 0.83; Eta2 = 0.17]
CBT FTF8dnr26%
Ruskin et al[83]119VA outpatient mental health clinics49.6 (12.8)DepressionPharmacotherapy TP8Psychiatrist27%Mean scores not reported. Differences between response rates according to the Hamilton Rating Scale for Depression for TP (49%) and FTF (43%) were not statistically significant (χ2 = 0.4, P > 0.05)
Pharmacotherapy FTF8Psychiatrist30%
Fortney et al[79]395VA community-based outpatient clinics59.2 (12.2)DepressionStepped collaborative care TPFlexible number of visits up to 12 moOn-site PCP + Off-site psychiatrist, care manager, PharmD10%At 12 mo, TP participants had greater odds of qualifying for remission than usual care participants (OR = 2.4, P = 0.04) but were not more likely to qualify for treatment response (OR = 1.4, P = 0.18) using the Hopkins Symptom Checklist
Usual care in primary care settingFlexible number of visits up to 12 moPCP9%
Hilty et al[80]94Rural primary care clinics46DepressionPsychiatric consultation TP, PCP training, disease management modules5 with psychiatrist 5 with PCPPsychiatrist, PCPdnrMean scores not reported. Differences between response rates according to the Beck Depression Inventory-13 for TP (42%) and augmented usual care (42%) were equivalent and not analyzed with odds ratios. Similarly, response rates according to the Hopkins Symptom Checklist-90 for TP (53%) and augmented usual care (42%) were not analyzed with odds ratios
Disease management modules, usual care in primary care setting5 with PCPPCPdnr
Chong et al[77]167Community health center43.0 (12.0)DepressionPharmacotherapy via TP + integrated primary care7 with psychiatrist, no limit on other visitsPsychiatrist, PCP, mental health specialist13.8%Patient Health Questionnaire-9 scores reduced from 17.3 (SD = 4.9) at baseline to 6.8 (SD = 6.0) at post-treatment for TP and from 18.3 (SD = 4.5) to 4.7 (SD = 5.1) for FTF (F =1.1, P > 0.05. Eta2 = 0.17)
Integrated primary careNo limitPCP, mental health specialist10.3%
Moreno et al[81]167Community health center43.2 (11.9)DepressionPharmacotherapy via TP + integrated primary care7 with psychiatrist, no limit on other visitsPsychiatrist, PCP, mental health specialistdnrPatient Health Questionnaire-9 scores reduced from 17.6 (SD = 7.6) at baseline to 5.1 (SD = 6.8) at post-treatment for TP and from 18.4 (SD = 4.9) to 4.5 (SD = 5.3) for FTF (t =2.30, P < 0.05. Eta2 =0 .11)
Integrated primary careNo limitPCP, mental health specialistdnr
Fortney et al[79]364Federally qualified health centers47.2 (12.6)DepressionEnhanced collaborative care TPFlexible number of visits in 12 moOn-site PCP + Off-site psychiatrist, care manager, behavioral health, PharmD23%At 12 mo, TP participants had greater odds of qualifying for remission than usual care participants (25.8% vs 9.9%; OR = 3.2, P < 0.001) and were more likely to qualify for treatment response (47.7% vs 21.9%; OR=3.3, P < 0.001) using the Hopkins Symptom Checklist-20
Collaborative care in primary care settingFlexible number of visits in 12 moPCP, care manager19%
Fortney et al[11]265VA community-based outpatient clinics52.2 (13.8)PTSDEnhanced collaborative care TPFlexible number of visits in 12 moOn-site PCP + Off-site psychiatrist, care manager, psychologist, PharmD16%At 12 mo, Posttraumatic Diagnostic Scale scores decreased 4.17 (SD = 9.8) for TP and 1.32 (SD = 8.8) for FTF (t = 2.30, P < 0.05. Cohen’s d = 0.31)
Collaborative care in primary care settingFlexible number of visits in 12 moPCP, care manager, social worker11%
Morland et al[84]125VA clinical sites and VA Vet Centers54.7 (9.6)PTSDGroup CBT TP12Clinical psychologist10%In a non-inferiority trial, State-Trait Anger Expression scores reduced from 56.7 (SD = 12.0) to 46.6 (SD = 12.2) in TP and from 55.0 (SD = 10.3) at baseline to 46.6 (SD = 12.2) at post-treatment for FTF. Using CIs and a priori cut-offs, criteria for non-inferiority met (Cohen d = 0.44 in favor of CBT TP)
Group CBT FTF12Clinical psychologist11%
Morland et al[85]125VA clinical sites and VA Vet Centers55.3 (12.5)PTSDCPT-C TP12Clinical psychologist or master’s level social worker18%In a non-inferiority trial, Clinician-Administered PTSD Scale scores reduced from 72.0 (SD = 14.6) to 55.6 (SD = 18.8) in CPT-C TP and from 68.9 (SD = 13.0) at baseline to 58.7 (SD = 21.0) at post-treatment for CPT-C FTF. Using CIs and a priori cut-offs, criteria for non-inferiority met (Cohen d = 0 .27 in favor of CBT TP)
CPT-C FTF12Clinical psychologist or master’s level social worker14%
Myers et al[86]233Primary care9.2 (2)ADHDPharmacotherapy via TP + caregiver training6Psychiatrist, master’s level therapist13%At 12 mo, TP participants had greater odds of no longer meeting diagnostic criteria for ADHD-inattentive subtype according to Vanderbilt ADHD Rating Scale at post-treatment (12% vs 26%; OR = 0.149, P < 0.001)
Psychiatric consultation with PCP + caregiver training1Psychiatrist, PCP5%
Mitchell et al[87]128Patient panels of rural physicians and therapists29.0 (10.7)Bulimia nervosaCBT TP20Clinical psychologist34%At post-treatment, abstinence from binge-eating episodes, purging episodes, and combined episodes ranged from 27%-50% for TP CBT and 29%-50% for FTF CBT with non-significant trend in favor of FTF. TP participants reported significantly more binge episodes (M = 6.2, SD = 12.3) than FTF participants (M = 3.7, SD = 11.2) at post-treatment (F = 6.76; P < 0.05)
CBT FTF20Clinical psychologist41%
De Las Cuevas et al[88]140Community mental health centerAdultsPsychiatric disordersPharmacotherapy, CBT TP8Psychiatrist6%Differences between improvement rates according to the Clinical Global Impressions scale for TP (67.2%) and FTF (62.5%) were not statistically significant (P > 0.05)
Pharmacotherapy, CBT FTF8Psychiatrist7%
O’Reilly et al[89]495Rural hospital and primary care clinicsAdultsPsychiatric disordersPsychiatric consultation TPFlexible number of visits in 4 moPsychiatrist7%In a non-inferiority trial, 22% of TP participants and 20% of FTF participants returned to functional status at post-treatment according to the Brief Symptom Inventory. Using CIs and a priori cut-offs, criteria for non-inferiority met
Psychiatric consultation FTFFlexible number of visits in 4 moPsychiatrist3%

Telepsychiatry compared to usual care: RCTs are required to determine if telepsychiatry interventions are comparable to FTF interventions in terms of outperforming usual care. Of the seven studies that made this comparison, six studies compared psychotropic medication management via telepsychiatry to FTF delivery of usual care[78-81,83,88]. One study included delivery of psychotherapy via telepsychiatry[11]. Four studies demonstrated superiority of telepsychiatry over FTF usual care[11,78,79,86], while telepsychiatry failed to achieve superior outcomes in three studies[77,80,81]. Fortney et al[11] revealed that although cognitive processing therapy (CPT) for PTSD was available in both the collaborative care delivered via telepsychiatry and FTF, participants randomized to telepsychiatry were 18 times more likely to initiate CPT and suggested that long travel distances discouraged weekly psychotherapy.

Telepsychiatry compared to FTF: A second and distinct question is whether or not mental health services delivered via telepsychiatry generate outcomes that are equivalent to FTF services. Of the seven studies making this comparison[82-85,88-90], only Mitchell et al[87] - who evaluated CBT for Bulimia Nervosa - found FTF to be superior to telepsychiatry. It is important to note that this finding held despite a high attrition rate of over 33% in both arms that reduced statistical power. The remaining six studies[81-83,88,90,91] reported that telepsychiatry was equivalent to FTF but only three conducted a non-inferiority trial that was explicitly designed to determine equivalency between telepsychiatry and FTF[84,85,89].

Summary: Overall, mental health interventions delivered via telepsychiatry and FTF resulted in similar treatment outcomes. To summarize: (1) telepsychiatry appears to be better than usual care, except possibly in the case of depression treatment in primary care where telepsychiatry has failed to show superior treatment outcomes to usual care in multiple studies; (2) There were no differences in the patterns of findings for the delivery of pharmacotherapy or psychotherapy delivered via telepsychiatry; (3) With the exception of one study, current data suggest telepsychiatry interventions produce outcomes that are statistically equivalent to outcomes produced by FTF interventions; and (4) The treatment outcome data on telepsychiatry is strongest for the treatment of depression, whereas the compelling data for psychiatric disorders other than depression is strongest in publications that focus on program descriptions and retrospective single cohort designs.

Implementation

There are two primary approaches to developing frameworks that guide the implementation of telepsychiatry. First, “purist” approaches use a comprehensive review of theory, basic science, and expert consensus to develop theoretically- and empirically-derived implementation frameworks (CFIR, RE-AIM, PRISM). Second, “pragmatic” approaches emerge from a practical need to organize implementation efforts for a given type of intervention or intervention delivery method for a defined population (i.e., pharmacotherapy via telepsychiatry for primary care patients with mental illness). For the scope of this review, pragmatic approaches such as the “Lexicon of Assessment and Outcome Measures for Telemental Health” are most appropriate[92]. This lexicon, derived by consensus from 26 established telepsychiatry experts, contains 36 implementation variables to consider. We review access, utilization, and the impact on clinical skill and workflows as particularly relevant variables in the implementation of telepsychiatry services.

Studies from the implementation of telepsychiatry in the VA provide data documenting the power of telepsychiatry to reach great numbers of people. Since 2003, the VA has documented nearly 500000 telepsychiatry encounters[93]. In an assessment of telepsychiatry services for over 100000 patients between 2006 and 2010, researchers found that hospitalization utilization decreased by approximately 25%[94]. However, patients in other settings do not always use telepsychiatry services, even when they are freely available. For example, in the Fortney et al[79] trial reviewed above in which free psychotherapy was available via telepsyschiatry, Deen et al[30] demonstrated 76% of patients reported that psychotherapy was acceptable, 38% scheduled a telepsychiatry psychotherapy session, 17% attended a session and 8% attended at least 8 session. Out of a range of possible patient sociedemographic and clinical factors, perceptions depression would remit on its own and low treatment efficacy predicted treatment utilization. On the other hand, there is some evidence that telepsychiatry may be more efficient that FTF. In a clinic that compared utilization data for 7523 telepsychiatry appointments and 115148 FTF appointments, patients kept more telepsychiatry appointments than FTF appointments (92% telepsychiatry vs 87% FTF). Also, patients were less likely to cancel telepsychiatry appointments (3.5% vs 4.8%) and were significantly less likely to be no-shows (4.2% vs 7.8%)[95]. Finally, it appears that telepsychiatry delivered in primary care does not increase PCP or mental health provider burden[96]. Hilty et al[97] demonstrated successful uptake of skills among PCPs treating anxiety and depression following consultations with telepsychiatrists. In this study, PCPs’ ability to appropriately dose medications for depression and anxiety improved from 47% to 64% (P < 0.001).

Cost-effectiveness

The cost of telepsychiatry is widely debated and discussed. Several methods to estimate the cost of telepsychiatry compared to FTF mental health services in traditional clinic settings as well as specialty environments (e.g., emergency departments) exist in the literature. One of the simplest assessments of cost data compares the collection of direct and indirect costs associated with patient encounters in telepsychiatry and FTF psychiatry visits. Studies that assessed the cost effectiveness of telepsychiatry have evaluated the direct costs of provider time[98], medical supplies[96], technology[53], and reimbursement[99]. Measurement of indirect costs include resources that facilitate patient encounters, such as clinic space[9], administrative support[96], and transportation[53,100-103]. Other studies compare healthcare utilization, such as visits to the emergency department[101] or primary care encounters[96] as a proxy measure of cost associated with telepsychiatry of FTF treatment. Return on investment (ROI) is another approach to calculating cost comparisons between telepsychiatry and FTF consolations. Simply stated, ROI is a cost-benefit ratio that assesses the cost of the service relative to an outcome measure, such as quality adjusted life years (e.g., QALY) or disability adjusted life years (e.g., DALY). Typically, QALY and DALY are measured by taking the difference between number of days symptom free days and days with clinical symptoms over a set period (12-18 mo). That difference is then divided by 365 d to create a range of time spent fully symptomatic to symptom-free.

Assessment of direct and indirect costs: Several studies (n = 18) compared the direct or indirect costs associated with providing telepsychiatry services. The majority of these studies (n = 13) found that telepsychiatry was associated with less direct and indirect costs than FTF services. A handful of studies utilized expenses associated with patient travel time as a cost outcome measure and found telepsychiatry reduced costs associated with travel compared to FTF[53,99,102,104-107]. The literature reviewed suggests that telepsychiatry may have greater up-front costs associated with service delivery when compared to FTF; however, there appears to be “tipping points” at which telepsychiatry begins to eclipse the cost-effectiveness of FTF interventions. Several studies identified the number of consultations at which telepsychiatry became more cost effective than FTF, with the number of consultations delivered ranging from 131[99] to 249[98,100] to 379[103]. For example, Butler et al[98] found that the cost savings of telepsychiatry occur after the health center or provider delivers 249 consultations. In more rural settings, the costs saving effects of telepsychiatry for the provider and patient could be established in as little as 6 consultation sessions[102]. Of note, two studies found that telepsychiatry was more expensive than FTF[108,109]. Pyne et al[109] hypothesized that telepsychiatry may prompt patients to seek additional specialty care because the telepsychiatry intervention delivered promotes a more integrated approach to care than traditional FTF delivery. Modai et al[108] suggested that telepsychiatry leaded to more hospitalizations than FTF; however, it should be noted that Modai et al[108] study was limited by a small sample size (n = 49). When excluding hospitalization from the analysis, Modai et al[108] found telepsychiatry was associated with decreased travel costs when compared to FTF.

Return on investment: A smaller number of studies (n = 4) utilized ROI methodology to determine cost effectiveness of telepsychiatry compared to FTF. Three of these studies utilized the QALY as an outcome measure[109-111] and the remaining study utilized the DALY as an outcome measure[112]. Results using these more sophisticated methods to evaluate cost-effectiveness found telepsychiatry to be more cost effective than FTF in underserved primary care populations[111], management of pain and depression in cancer populations[110], and in depressed populations[112]. However, Pyne et al[109] found that telepsychiatry was not cost effective compared to FTF in treating depression in rural primary care settings; while telepsychiatry was effective in treating depression in rural populations, there were greater costs associated with the delivery of services and greater utilization of outpatient services among patients receiving telepsychiatry compared to FTF consultations.

Summary: The evidence to date on the cost effectiveness of telepsychiatry is generally positive as most studies demonstrate that telepsychiatry reduces direct and indirect costs and increases quality of life adjusted years when compared to FTF. For example, in one study the authors estimated that utilizing telepsychiatry to deliver 16 sessions of CBT for over 20 wk could save a clinic approximately $2025 per patient compared to FTF services[104]. In general, (1) telepsychiatry reduces costs associated with patient travel; (2) there are likely more upfront costs associated with telepsychiatry but the costs are recovered after 6-379 sessions depending in the population being served; (3) cost effectiveness of telepsychiatry may differ depending on setting (e.g., rural or urban). Significant differences emerged specific to operationalizing cost effectiveness and the method of assessing cost effectiveness. It appears that studies that utilized a simple approach to cost effectiveness by comparing direct and indirect costs had similar cost saving findings, whereas the handful of studies that utilized a more sophisticated approach suggest telepsychiatry is more cost-effective than FTF.

Legal issues

Despite the promise of telepsychiatry to reduce barriers associated with accessing mental health services and increase access to services, legal issues specific to telepsychiatry remain an extant and significant barrier. We identified two primary themes concerning legal issues associated with providing mental health services via telepsychiatry - licensing regulations and risks to patient confidentiality.

Several publications highlighted the legal challenges that telepsychiatry presents concerning provider licensure. Telepsychiatry increases providers’ catchment area such that patients living outside of the state where a provider is licensed to practice may request their services via telepsychiatry. However, each state has its own licensing boards that establish practice jurisdictions for providers licensed in the state, and some have specific regulations related to telepsychiatry[113]. As the field progresses, the topic of licensure jurisdictions within the United States will continue to be discussed on the National level[114]. Until a resolution is reached, telepsychiatry providers must have separate licenses for each state in which they provide services and are advised to be familiar with state-specific regulations. For example, Shore et al[115] suggests that a priori arrangements should be made with local law enforcement and social services that are responsible for initiating involuntary commitments.

Patient confidentiality is also a debated topic specific to telepsychiatry. Baker et al[116] outlined several considerations for patient confidentiality when utilizing telepsychiatry: Verification of the patient’s identity, establishing and ensuring privacy on both the provider and patient’s location, disruption of technology, involving others in treatment, and storage of information collected or recorded during the telepsychiatry consultation. Several studies encourage telepsychiatry providers to go above and beyond normal methods of informing patients about limits of confidentiality such that they are aware of possible privacy breaches that are less likely to occur in traditional FTF consultations[116-119].

DISCUSSION

There is a large evidence base for the use of telepsychiatry as a delivery method for mental health services. Contemporary healthcare innovations such as telepsychiatry are commonly evaluated according to the Triple Aim Framework which addresses patient satisfaction, care quality, and cost effectiveness[5]. We reviewed 569 studies that focus on Triple Aim domains and the evidence indicates that patients are satisfied, telepsychiatry is comparable to FTF delivery of mental health interventions, and telepsychiatry can be a cost-effective approach to increasing access to mental health care.

A socioeconomically and clinically diverse patient and provider population has reported on their experience with telepsychiatry. Their responses to self-report questionnaires, qualitative interviews, and mixed-methods assessments suggest that they are comfortable using this technology, appreciate the practical benefit of avoiding travel, and are less concerned than providers about potentially adverse impacts of telepsychiatry on the therapeutic alliance. In terms of care quality, the evidence reviewed suggest that telepsychiatry is comparable to FTF in the reliability of assessment and effective treatment of a range of behavioral and mental health disorders. Importantly, a small number of high quality studies have used non-inferiority designs to demonstrate statistically equivalence in treatment outcomes between telepsychiatry and FTF.

The remaining literature we reviewed focus on practical factors related to the implementation of telepsychiatry such as adaptability of telepsychiatry to routine care settings, cost-effectiveness, and legal issues. Several program descriptions discuss the actual usage of telepsychiatry services in routine clinical settings. Although it is not a foregone conclusion that patients will use telepsychiatry services, even when they are freely available, telepsychiatry is comparable to FTF in terms of service utilization patterns and can help allied health providers develop clinical skill in treating mental illnesses. Using both face-valid and sophisticated approaches to evaluating cost-effectiveness, telepsychiatry can be as cost-effective as FTF services and more studies are needed to determine how cost-effectiveness is affected by rurality, patient sociodemographic and clinical characteristics, provider type, and organizational characteristics. Notwithstanding legal concerns about loss of confidentiality and limited capacity to respond to psychiatric emergencies, we uncovered no published reports of these adverse events in the use of telepsychiatry.

Limitations and future directions

Despite many strengths evident in the evidence base for telepsychiatry, there are important limitations to consider. First, it is important to acknowledge the limitations inherent in assessing consumer attitudes with healthcare services. As reported over a decade earlier[119] there are several factors that continue to limit the quality and generalizability of research on patient satisfaction with telepsychiatry services such as: (1) over-reliance on self-report methodologies; (2) selection biases that over-represent patients amenable to telepsychiatry; (3) insufficient sample sizes; and (4) omission direct comparison of preferences for telepsychiatry vs FTF. Quantitative designs that rely solely on participant self-report are sufficient for demonstrating minimum standards of acceptability, but to obtain a more nuanced understanding of reactions to telepsychiatry, mixed-methods and experimental designs are strongly recommended. In light of the consistent finding that patients are satisfied with telepsychiatry services, we recommend that future studies focus less on assessing satisfaction, and more on clarifying the actual effects of telepsychiatry on therapeutic rapport. For instance, comparing ratings of therapeutic rapport by blind assessors listening to audio recordings of telepsychiatry and FTF mental health services would provide a more objective test of the notion that telepsychiatry impairs rapport. A more thorough understanding of reactions to telepsychiatry, especially as they pertain to provider skepticism based on the assumption that telepsychiatry impairs rapport and intervention quality, has the potential to increase acceptability and uptake of telepsychiatry services.

Second, although the studies reviewed suggest that telepsychiatry is a reliable method for assessment of mental health constructs, it is inappropriate to interpret a null hypothesis without using a non-inferiority design. In this case it is particularly problematic because there is a consistent pattern of higher kappa statistics for FTF assessments than for telepsychiatry assessments, and some authors interpret a lack of statistically significant differences between assessment modality as equivalence in their reliability. To claim that assessments conducted via telepsychiatry are as reliable as assessments conducted FTF, researchers must use study designs that either explicitly test for equivalency, or are adequately powered to detect clinically significant differences between the reliability of telepsychiatry and FTF assessments. We recommend that future studies shift focus from establishing equivalent reliability between telepsychiatry and FTF assessments to identifying which types of assessments are most amenable to the telepsychiatry modality, which types of assessments are most difficult to administer via telepsychiatry, and which types of adaptations help improve the accuracy, efficiency, and consumer experience of telepsychiatry assessments.

Third, we did not perform an intensive quality appraisal of the treatment outcome studies reviewed. Even rigorously designed and executed RCTs are subject to several sources of biases such as selection bias, performance bias, attrition bias, and reporting bias. Furthermore, several studies claim equivalency between telepsychiatry and FTF with using non-inferiority designs. However, we do not recommend more equivalency studies; rather, it is clear that telepsychiatry is comparable to FTF and the field now needs more research focusing on factors that increase telepsychiatry adoption, moderators to determine for which patients in which settings telepsychiatry is most effective, and strategies for integrating telepsychiatry services within the broach context of healthcare service delivery.

Finally, assessing costs associated with the delivery of telepsychiatry is an important, yet often overlooked, factor when evaluating telepsychiatry outcomes. There is room for improvement in cost evaluations of telepsychiatry in order to generate high-quality generalizable findings. Only four of the studies reviewed used sophisticated methods of cost effectiveness. Additionally, cost effectiveness has only been evaluated in a narrow range of patient populations. Future research would benefit from using methodologies that incorporate DALYs or QALYs to examine cost-effectiveness among diverse psychiatric populations, as there is evidence to suggest that telepsychiatry may be more economical than FTF with patients with specific demographic characteristics (e.g., patients seeking services in primary care vs specialty mental health, rural vs urban patients).

The gap between the need for mental health treatment and availability of mental health providers has prompted national and international organizations such as the National Institute of Mental Health (NIMH; 2013) and the World Health Organization (WHO; 2013) to prioritize the development and evaluation of novel service delivery methods. Telepsychiatry represents a highly promising approach to reducing the treatment gap by making it easier for patients, especially those in isolated contexts, to access expert mental health care. Just as the NIMH and WHO have shifted emphasis from efficacy to effectiveness testing, it also time for telepsychiatry researchers to focus less on pure outcome studies to document patient acceptability and high care quality, and more on studies that inform evidence-based approaches to implementing and sustaining telepsychiatry services.

COMMENTS
Background

Innovative approaches to delivering mental health services are urgently needed to increase access to evidence-based care. Telepsychiatry, which in its contemporary use refers to the delivery of mental health services via video-based conferencing, has great potential to address mental health disparities by extending the reach of mental health care to those living in rural areas or to those who otherwise have limited access to care.

Research frontiers

For many years, an important question for telepsychiatry researchers was, “Is telepsychiatry a via ble delivery method for mental health services relative to face-to-face delivery”? As experts in mental health service delivery have shifted emphasis from efficacy to effectiveness testing, so too is it time for telepsychiatry researchers to focus less on pure outcome studies to document patient acceptability and high care quality, and more on studies that inform evidence-based approaches to implementing and sustaining telepsychiatry services.

Innovations and breakthroughs

Telepsychiatry represents a highly promising approach to reducing the treatment gap by making it easier for patients, especially those in isolated contexts, to access expert mental health care.

Applications

This review suggests that telepsychiatry is a via ble and cost-effective method to increase access to mental health services.

Terminology

Telepsychiatry, in its contemporary use, refers to the delivery of mental health services via video-based conferencing.

Peer-review

In this review, the authors have provided a comprehensive commentary on the state of the research for key outcomes in telepsychiatry.

Footnotes

P- Reviewer: Bee PE, Grof P, Hyler SE S- Editor: Qiu S L- Editor: A E- Editor: Wu HL

References
1.  United States Department of Veterans Affairs. Native Domain Telemental Health Services (Edited by Office of Health Reserach. Washington DC, 2015).  Available from: http//www.ruralhealth.va.gov/native/programs/telemental-services.asp.  [PubMed]  [DOI]
2.  Yellowlees P, Chapman M, Fielke K. Telehealth: Anywhere, anytime, achievements and aspirations. Australian and New Zealand Journal of Psychiatry. 2013;47:19-20.  [PubMed]  [DOI]
3.  Monnier J, Knapp RG, Frueh BC. Recent advances in telepsychiatry: an updated review. Psychiatr Serv. 2003;54:1604-1609.  [PubMed]  [DOI]
4.  Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemed J E Health. 2013;19:444-454.  [PubMed]  [DOI]
5.  Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27:759-769.  [PubMed]  [DOI]
6.  Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Eval Program Plann. 1979;2:197-207.  [PubMed]  [DOI]
7.  Bose U, McLaren P, Riley A, Mohammedali A. The use of telepsychiatry in the brief counselling of non-psychotic patients from an inner-London general practice. J Telemed Telecare. 2001;7 Suppl 1:8-10.  [PubMed]  [DOI]
8.  D'Souza R. A pilot study of an educational service for rural mental health practitioners in South Australia using telemedicine. J Telemed Telecare. 2000;6 Suppl 1:S187-189.  [PubMed]  [DOI]
9.  De Las Cuevas C, Artiles J, De La Fuente J, Serrano P. Telepsychiatry in the Canary Islands: user acceptance and satisfaction. J Telemed Telecare. 2003;9:221-224.  [PubMed]  [DOI]
10.  Dobscha SK, Corson K, Solodky J, Gerrity MS. Use of videoconferencing for depression research: enrollment, retention, and patient satisfaction. Telemed J E Health. 2005;11:84-89.  [PubMed]  [DOI]
11.  Fortney JC, Pyne JM, Kimbrell TA, Hudson TJ, Robinson DE, Schneider R, Moore WM, Custer PJ, Grubbs KM, Schnurr PP. Telemedicine-based collaborative care for posttraumatic stress disorder: a randomized clinical trial. JAMA Psychiatry. 2015;72:58-67.  [PubMed]  [DOI]
12.  Frueh BC, Henderson S, Myrick H. Telehealth service delivery for persons with alcoholism. J Telemed Telecare. 2005;11:372-375.  [PubMed]  [DOI]
13.  Holden D, Dew E. Telemedicine in a rural gero-psychiatric inpatient unit: comparison of perception/satisfaction to onsite psychiatric care. Telemed J E Health. 2008;14:381-384.  [PubMed]  [DOI]
14.  Kennedy C, Yellowlees P. A community-based approach to evaluation of health outcomes and costs for telepsychiatry in a rural population: preliminary results. J Telemed Telecare. 2000;6 Suppl 1:S155-S157.  [PubMed]  [DOI]
15.  Manning TR, Goetz ET, Street RL. Signal delay effects on rapport in telepsychiatry. CyberPsychology & Behavior. 2000;3:119-127.  [PubMed]  [DOI]
16.  McCarty CA, Vander Stoep A, Violette H, Myers K. Interventions developed for psychiatric and behavioral treatment in the children’s adhd telemental health treatment study. Journal of Child and Family Studies. 2014;24:1735-1743.  [PubMed]  [DOI]
17.  Mucic D. Transcultural telepsychiatry and its impact on patient satisfaction. J Telemed Telecare. 2010;16:237-242.  [PubMed]  [DOI]
18.  Mucic D. Experiences from the first international telepsychiatry service in Europe: Patient satisfaction survey. European Psychiatry. 2010;25:25.  [PubMed]  [DOI]
19.  Rohland BM. Telepsychiatry in the heartland: if we build it, will they come? Community Ment Health J. 2001;37:449-459.  [PubMed]  [DOI]
20.  Santa Ana EJ, Stallings DL, Rounsaville BJ, Martino S. Development of an in-home telehealth program for outpatient veterans with substance use disorders. Psychol Serv. 2013;10:304-314.  [PubMed]  [DOI]
21.  Schutte JL, McCue MP, Parmanto B, McGonigle J, Handen B, Lewis A. Usability and reliability of a remotely administered adult autism assessment, the Autism Diagnostic Observation Schedule (ADOS) Module 4. Telemedicine and e-Health. 2015;21:176-184.  [PubMed]  [DOI]
22.  Shore JH, Brooks E, Novins D. In-home monitoring for American Indian Veterans with posttraumatic stress disorder. Telemedicine and e-Health. 2008;14:77.  [PubMed]  [DOI]
23.  Simpson S. The provision of a telepsychology service to Shetland: client and therapist satisfaction and the ability to develop a therapeutic alliance. J Telemed Telecare. 2001;7 Suppl 1:34-36.  [PubMed]  [DOI]
24.  Simpson S, Knox J, Mitchell D, Ferguson J, Brebner J, Brebner E. A multidisciplinary approach to the treatment of eating disorders via videoconferencing in north-east Scotland. J Telemed Telecare. 2003;9 Suppl 1:S37-S38.  [PubMed]  [DOI]
25.  Sorvaniemi M, Ojanen E, Santamäki O. Telepsychiatry in emergency consultations: a follow-up study of sixty patients. Telemed J E Health. 2005;11:439-441.  [PubMed]  [DOI]
26.  Stain HJ, Payne K, Thienel R, Michie P, Carr V, Kelly B. The feasibility of videoconferencing for neuropsychological assessments of rural youth experiencing early psychosis. J Telemed Telecare. 2011;17:328-331.  [PubMed]  [DOI]
27.  Werner P. Willingness to use telemedicine for psychiatric care. Telemed J E Health. 2004;10:286-293.  [PubMed]  [DOI]
28.  Williams A, Larocca R, Chang T, Trinh NH, Fava M, Kvedar J, Yeung A. Web-based depression screening and psychiatric consultation for college students: a feasibility and acceptability study. Int J Telemed Appl. 2014;2014:580786.  [PubMed]  [DOI]
29.  Brodey BB, Claypoole KH, Motto J, Arias RG, Goss R. Satisfaction of forensic psychiatric patients with remote telepsychiatric evaluation. Psychiatr Serv. 2000;51:1305-1307.  [PubMed]  [DOI]
30.  Deen TL, Fortney JC, Schroeder G. Patient acceptance of and initiation and engagement in telepsychotherapy in primary care. Psychiatr Serv. 2013;64:380-384.  [PubMed]  [DOI]
31.  Gawdzik N, Kachnic J, Wojtuszek M, Wutke J, Krysta K. Knowledge about telepsychiatry among patients in Poland. European Psychiatry. 2015;30:315.  [PubMed]  [DOI]
32.  George S, Hamilton A, Baker RS. How Do Low-Income Urban African Americans and Latinos Feel about Telemedicine? A Diffusion of Innovation Analysis. Int J Telemed Appl. 2012;2012:715194.  [PubMed]  [DOI]
33.  Hilty DM, Nesbitt TS, Kuenneth CA, Cruz GM, Hales RE. Rural versus suburban primary care needs, utilization, and satisfaction with telepsychiatric consultation. J Rural Health. 2007;23:163-165.  [PubMed]  [DOI]
34.  May C, Gask L, Ellis N, Atkinson T, Mair F, Smith C, Pidd S, Esmail A. Telepsychiatry evaluation in the north-west of England: preliminary results of a qualitative study. J Telemed Telecare. 2000;6 Suppl 1:S20-S22.  [PubMed]  [DOI]
35.  Tucker W, Olfson M, Simring S, Goodman W, Bienenfeld S. A pilot survey of inmate preferences for on-site, visiting consultant, and telemedicine psychiatric services. CNS Spectr. 2006;11:783-787.  [PubMed]  [DOI]
36.  Urness D, Wass M, Gordon A, Tian E, Bulger T. Client acceptability and quality of life--telepsychiatry compared to in-person consultation. J Telemed Telecare. 2006;12:251-254.  [PubMed]  [DOI]
37.  Wagnild G, Leenknecht C, Zauher J. Psychiatrists’ satisfaction with telepsychiatry. Telemed J E Health. 2006;12:546-551.  [PubMed]  [DOI]
38.  Ye J, Shim R, Lukaszewski T, Yun K, Kim SH, Ruth G. Telepsychiatry services for Korean immigrants. Telemed J E Health. 2012;18:797-802.  [PubMed]  [DOI]
39.  May C, Gask L, Atkinson T, Ellis N, Mair F, Esmail A. Resisting and promoting new technologies in clinical practice: the case of telepsychiatry. Soc Sci Med. 2001;52:1889-1901.  [PubMed]  [DOI]
40.  Tucker WM, Segal G, Hyler SE. Psychiatric telemedicine for rural New York. J Psychiatr Pract. 2001;7:279-281.  [PubMed]  [DOI]
41.  Shore JH, Brooks E, Savin D, Orton H, Grigsby J, Manson SM. Acceptability of telepsychiatry in American Indians. Telemed J E Health. 2008;14:461-466.  [PubMed]  [DOI]
42.  Elford R, White H, Bowering R, Ghandi A, Maddiggan B, St John K, House M, Harnett J, West R, Battcock A. A randomized, controlled trial of child psychiatric assessments conducted using videoconferencing. J Telemed Telecare. 2000;6:73-82.  [PubMed]  [DOI]
43.  Greenwood J, Chamberlain C, Parker G. Evaluation of a rural telepsychiatry service. Australas Psychiatry. 2004;12:268-272.  [PubMed]  [DOI]
44.  Saurman E, Kirby SE, Lyle D. No longer ‘flying blind’: how access has changed emergency mental health care in rural and remote emergency departments, a qualitative study. BMC Health Serv Res. 2015;15:156.  [PubMed]  [DOI]
45.  Pangka KR, Chandrasena R, Wijeratne N, Mann M. Exploring the views of emergency department staff on the use of videoconferencing for mental health emergencies in southwestern Ontario. Stud Health Technol Inform. 2015;209:114-120.  [PubMed]  [DOI]
46.  Sinclair C, Holloway K, Riley G, Auret K. Online mental health resources in rural Australia: clinician perceptions of acceptability. J Med Internet Res. 2013;15:e193.  [PubMed]  [DOI]
47.  Alexander J, Lattanzio A. Utility of telepsychiatry for Aboriginal Australians. Aust N Z J Psychiatry. 2009;43:1185.  [PubMed]  [DOI]
48.  Dzara K, Sarver J, Bennett JI, Basnet P. Resident and medical student viewpoints on their participation in a telepsychiatry rotation. Acad Psychiatry. 2013;37:214-216.  [PubMed]  [DOI]
49.  Hilty DM, Yellowlees PM, Nesbitt TS. Evolution of telepsychiatry to rural sites: changes over time in types of referral and in primary care providers’ knowledge, skills and satisfaction. Gen Hosp Psychiatry. 2006;28:367-373.  [PubMed]  [DOI]
50.  Kopel H, Nunn K, Dossetor D. Evaluating satisfaction with a child and adolescent psychological telemedicine outreach service. J Telemed Telecare. 2001;7 Suppl 2:35-40.  [PubMed]  [DOI]
51.  Mitchell SA, MacLaren AT, Morton M, Carachi R. Professional opinions of the use of telemedicine in child & amp; adolescent psychiatry. Scott Med J. 2009;54:13-16.  [PubMed]  [DOI]
52.  Pesämaa L, Ebeling H, Kuusimäki ML, Winblad I, Isohanni M, Moilanen I. Videoconferencing in child and adolescent psychiatry in Finland--an inadequately exploited resource. J Telemed Telecare. 2007;13:125-129.  [PubMed]  [DOI]
53.  Elford DR, White H, St John K, Maddigan B, Ghandi M, Bowering R. A prospective satisfaction study and cost analysis of a pilot child telepsychiatry service in Newfoundland. J Telemed Telecare. 2001;7:73-81.  [PubMed]  [DOI]
54.  Wynn R, Bergvik S, Pettersen G, Fossum S. Clinicians’ experiences with videoconferencing in psychiatry. Stud Health Technol Inform. 2012;180:1218-1220.  [PubMed]  [DOI]
55.  Grealish A, Hunter A, Glaze R, Potter L. Telemedicine in a child and adolescent mental health service: participants’ acceptance and utilization. J Telemed Telecare. 2005;11 Suppl 1:53-55.  [PubMed]  [DOI]
56.  Gibson K, O’Donnell S, Coulson H, Kakepetum-Schultz T. Mental health professionals’ perspectives of telemental health with remote and rural First Nations communities. J Telemed Telecare. 2011;17:263-267.  [PubMed]  [DOI]
57.  Ertelt TW, Crosby RD, Marino JM, Mitchell JE, Lancaster K, Crow SJ. Therapeutic factors affecting the cognitive behavioral treatment of bulimia nervosa via telemedicine versus face-to-face delivery. Int J Eat Disord. 2011;44:687-691.  [PubMed]  [DOI]
58.  Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry. 1960;23:56-62.  [PubMed]  [DOI]
59.  Shore JH, Savin D, Orton H, Beals J, Manson SM. Diagnostic reliability of telepsychiatry in American Indian veterans. Am J Psychiatry. 2007;164:115-118.  [PubMed]  [DOI]
60.  Yellowlees PM, Odor A, Iosif AM, Parish MB, Nafiz N, Patrice K, Xiong G, McCaron R, Sanchez R, Ochoa E. Transcultural psychiatry made simple--asynchronous telepsychiatry as an approach to providing culturally relevant care. Telemed J E Health. 2013;19:259-264.  [PubMed]  [DOI]
61.  Yellowlees PM, Hilty DM, Marks SL, Neufeld J, Bourgeois JA. A retrospective analysis of a child and adolescent eMental Health program. J Am Acad Child Adolesc Psychiatry. 2008;47:103-107.  [PubMed]  [DOI]
62.  Amarendran V, George A, Gersappe V, Krishnaswamy S, Warren C. The reliability of telepsychiatry for a neuropsychiatric assessment. Telemed J E Health. 2011;17:223-225.  [PubMed]  [DOI]
63.  Turner TH, Horner MD, Vankirk KK, Myrick H, Tuerk PW. A pilot trial of neuropsychological evaluations conducted via telemedicine in the Veterans Health Administration. Telemed J E Health. 2012;18:662-667.  [PubMed]  [DOI]
64.  Timpano F, Pirrotta F, Bonanno L, Marino S, Marra A, Bramanti P, Lanzafame P. Videoconference-based mini mental state examination: a validation study. Telemed J E Health. 2013;19:931-937.  [PubMed]  [DOI]
65.  Munro Cullum C, Hynan LS, Grosch M, Parikh M, Weiner MF. Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc. 2014;20:1028-1033.  [PubMed]  [DOI]
66.  Kobak KA, Williams JB, Jeglic E, Salvucci D, Sharp IR. Face-to-face versus remote administration of the Montgomery-Asberg Depression Rating Scale using videoconference and telephone. Depress Anxiety. 2008;25:913-919.  [PubMed]  [DOI]
67.  Kobak KA, Williams JB, Engelhardt N. A comparison of face-to-face and remote assessment of inter-rater reliability on the Hamilton Depression Rating Scale via videoconferencing. Psychiatry Res. 2008;158:99-103.  [PubMed]  [DOI]
68.  Kobak KA. A comparison of face-to-face and videoconference administration of the Hamilton Depression Rating Scale. J Telemed Telecare. 2004;10:231-235.  [PubMed]  [DOI]
69.  Kirkwood KT, Peck DF, Bennie L. The consistency of neuropsychological assessments performed via telecommunication and face to face. J Telemed Telecare. 2000;6:147-151.  [PubMed]  [DOI]
70.  Singh SP, Arya D, Peters T. Accuracy of telepsychiatric assessment of new routine outpatient referrals. BMC Psychiatry. 2007;7:55.  [PubMed]  [DOI]
71.  Malhotra S, Chakrabarti S, Shah R, Gupta A, Mehta A, Nithya B, Kumar V, Sharma M. Development of a novel diagnostic system for a telepsychiatric application: a pilot validation study. BMC Res Notes. 2014;7:508.  [PubMed]  [DOI]
72.  Seidel RW, Kilgus MD. Agreement between telepsychiatry assessment and face-to-face assessment for Emergency Department psychiatry patients. J Telemed Telecare. 2014;20:59-62.  [PubMed]  [DOI]
73.  Manguno-Mire GM, Thompson JW, Shore JH, Croy CD, Artecona JF, Pickering JW. The use of telemedicine to evaluate competency to stand trial: a preliminary randomized controlled study. J Am Acad Psychiatry Law. 2007;35:481-489.  [PubMed]  [DOI]
74.  Matsuura S, Hosaka T, Yukiyama T, Ogushi Y, Okada Y, Haruki Y, Nakamura M. Application of telepsychiatry: a preliminary study. Psychiatry Clin Neurosci. 2000;54:55-58.  [PubMed]  [DOI]
75.  Yoshino A, Shigemura J, Kobayashi Y, Nomura S, Shishikura K, Den R, Wakisaka H, Kamata S, Ashida H. Telepsychiatry: assessment of televideo psychiatric interview reliability with present- and next-generation internet infrastructures. Acta Psychiatr Scand. 2001;104:223-226.  [PubMed]  [DOI]
76.  Jones BN, Johnston D, Reboussin B, McCall WV. Reliability of telepsychiatry assessments: subjective versus observational ratings. J Geriatr Psychiatry Neurol. 2001;14:66-71.  [PubMed]  [DOI]
77.  Chong J, Moreno F. Feasibility and acceptability of clinic-based telepsychiatry for low-income Hispanic primary care patients. Telemed J E Health. 2012;18:297-304.  [PubMed]  [DOI]
78.  Fortney JC, Pyne JM, Edlund MJ, Williams DK, Robinson DE, Mittal D, Henderson KL. A randomized trial of telemedicine-based collaborative care for depression. J Gen Intern Med. 2007;22:1086-1093.  [PubMed]  [DOI]
79.  Fortney JC, Pyne JM, Mouden SB, Mittal D, Hudson TJ, Schroeder GW, Williams DK, Bynum CA, Mattox R, Rost KM. Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial. Am J Psychiatry. 2013;170:414-425.  [PubMed]  [DOI]
80.  Hilty DM, Marks S, Wegelin J, Callahan EJ, Nesbitt TS. A randomized, controlled trial of disease management modules, including telepsychiatric care, for depression in rural primary care. Psychiatry (Edgmont). 2007;4:58-65.  [PubMed]  [DOI]
81.  Moreno FA, Chong J, Dumbauld J, Humke M, Byreddy S. Use of standard Webcam and Internet equipment for telepsychiatry treatment of depression among underserved Hispanics. Psychiatr Serv. 2012;63:1213-1217.  [PubMed]  [DOI]
82.  Nelson EL, Barnard M, Cain S. Treating childhood depression over videoconferencing. Telemed J E Health. 2003;9:49-55.  [PubMed]  [DOI]
83.  Ruskin PE, Silver-Aylaian M, Kling MA, Reed SA, Bradham DD, Hebel JR, Barrett D, Knowles F, Hauser P. Treatment outcomes in depression: comparison of remote treatment through telepsychiatry to in-person treatment. Am J Psychiatry. 2004;161:1471-1476.  [PubMed]  [DOI]
84.  Morland LA, Greene CJ, Rosen CS, Foy D, Reilly P, Shore J, He Q, Frueh BC. Telemedicine for anger management therapy in a rural population of combat veterans with posttraumatic stress disorder: a randomized noninferiority trial. J Clin Psychiatry. 2010;71:855-863.  [PubMed]  [DOI]
85.  Morland LA, Mackintosh MA, Greene CJ, Rosen CS, Chard KM, Resick P, Frueh BC. Cognitive processing therapy for posttraumatic stress disorder delivered to rural veterans via telemental health: a randomized noninferiority clinical trial. J Clin Psychiatry. 2014;75:470-476.  [PubMed]  [DOI]
86.  Myers K, Vander Stoep A, Zhou C, McCarty CA, Katon W. Effectiveness of a telehealth service delivery model for treating attention-deficit/hyperactivity disorder: a community-based randomized controlled trial. J Am Acad Child Adolesc Psychiatry. 2015;54:263-274.  [PubMed]  [DOI]
87.  Mitchell JE, Crosby RD, Wonderlich SA, Crow S, Lancaster K, Simonich H, Swan-Kremeier L, Lysne C, Myers TC. A randomized trial comparing the efficacy of cognitive-behavioral therapy for bulimia nervosa delivered via telemedicine versus face-to-face. Behav Res Ther. 2008;46:581-592.  [PubMed]  [DOI]
88.  De Las Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, Meise U. Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemed J E Health. 2006;12:341-350.  [PubMed]  [DOI]
89.  O'Reilly R, Bishop J, Maddox K, Hutchinson L, Fisman M, Takhar J. Is telepsychiatry equivalent to face-to-face psychiatry? Results from a randomized controlled equivalence trial. Psychiatr Serv. 2007;58:836-843.  [PubMed]  [DOI]
90.  Mohr DC, Carmody T, Erickson L, Jin L, Leader J. Telephone-administered cognitive behavioral therapy for veterans served by community-based outpatient clinics. J Consult Clin Psychol. 2011;79:261-265.  [PubMed]  [DOI]
91.  Nelson EL. Cognitive behavioral therapy for childhood depression: A comparison of face-to-face and interactive televideo settings. The Sciences and Engineering. 2004;65:1558.  [PubMed]  [DOI]
92.  Shore J, Kaufmann LJ, Brooks E, Bair B, Dailey N, Richardson WJ, Floyd J, Lowe J, Nagamoto H, Phares R. Review of American Indian veteran telemental health. Telemed J E Health. 2012;18:87-94.  [PubMed]  [DOI]
93.   Available from: http//www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/869-notes.pdf.  [PubMed]  [DOI]
94.  Godleski L, Darkins A, Peters J. Outcomes of 98,609 U.S. Department of Veterans Affairs patients enrolled in telemental health services, 2006-2010. Psychiatr Serv. 2012;63:383-385.  [PubMed]  [DOI]
95.  Leigh H, Cruz H, Mallios R. Telepsychiatry appointments in a continuing care setting: kept, cancelled and no-shows. J Telemed Telecare. 2009;15:286-289.  [PubMed]  [DOI]
96.  Fortney JC, Maciejewski ML, Tripathi SP, Deen TL, Pyne JM. A budget impact analysis of telemedicine-based collaborative care for depression. Med Care. 2011;49:872-880.  [PubMed]  [DOI]
97.  Hilty DM, Yellowlees PM, Cobb HC, Bourgeois JA, Neufeld JD, Nesbitt TS. Models of telepsychiatric consultation--liaison service to rural primary care. Psychosomatics. 2006;47:152-157.  [PubMed]  [DOI]
98.  Butler TN, Yellowlees P. Cost analysis of store-and-forward telepsychiatry as a consultation model for primary care. Telemed J E Health. 2012;18:74-77.  [PubMed]  [DOI]
99.  Smith AC, Armfield NR, Croll J, Gray LC. A review of Medicare expenditure in Australia for psychiatric consultations delivered in person and via videoconference. J Telemed Telecare. 2012;18:169-171.  [PubMed]  [DOI]
100.  D'Souza R. Telemedicine for intensive support of psychiatric inpatients admitted to local hospitals. J Telemed Telecare. 2000;6 Suppl 1:S26-S28.  [PubMed]  [DOI]
101.  Jong M. Managing suicides via videoconferencing in a remote northern community in Canada. Int J Circumpolar Health. 2004;63:422-428.  [PubMed]  [DOI]
102.  Harley J. Economic evaluation of a tertiary telepsychiatry service to an island. J Telemed Telecare. 2006;12:354-357.  [PubMed]  [DOI]
103.  Smith AC, Scuffham P, Wootton R. The costs and potential savings of a novel telepaediatric service in Queensland. BMC Health Serv Res. 2007;7:35.  [PubMed]  [DOI]
104.  Crow SJ, Mitchell JE, Crosby RD, Swanson SA, Wonderlich S, Lancanster K. The cost effectiveness of cognitive behavioral therapy for bulimia nervosa delivered via telemedicine versus face-to-face. Behav Res Ther. 2009;47:451-453.  [PubMed]  [DOI]
105.  Grady BJ. A comparative cost analysis of an integrated military telemental health-care service. Telemed J E Health. 2002;8:293-300.  [PubMed]  [DOI]
106.  Spaulding R, Belz N, DeLurgio S, Williams AR. Cost savings of telemedicine utilization for child psychiatry in a rural Kansas community. Telemed J E Health. 2010;16:867-871.  [PubMed]  [DOI]
107.  Whitacre BE, Hartman PS, Boggs SA, Schott V. A community perspective on quantifying the economic impact of teleradiology and telepsychiatry. J Rural Health. 2009;25:194-197.  [PubMed]  [DOI]
108.  Modai I, Jabarin M, Kurs R, Barak P, Hanan I, Kitain L. Cost effectiveness, safety, and satisfaction with video telepsychiatry versus face-to-face care in ambulatory settings. Telemed J E Health. 2006;12:515-520.  [PubMed]  [DOI]
109.  Pyne JM, Fortney JC, Tripathi SP, Maciejewski ML, Edlund MJ, Williams DK. Cost-effectiveness analysis of a rural telemedicine collaborative care intervention for depression. Archives of General Psychiatry. 2010;67:812-821.  [PubMed]  [DOI]
110.  Choi Yoo SJ, Nyman JA, Cheville AL, Kroenke K. Cost effectiveness of telecare management for pain and depression in patients with cancer: results from a randomized trial. Gen Hosp Psychiatry. 2014;36:599-606.  [PubMed]  [DOI]
111.  Pyne JM, Fortney JC, Mouden S, Lu L, Hudson TJ, Mittal D. Cost-effectiveness of on-site versus off-site collaborative care for depression in rural FQHCs. Psychiatr Serv. 2015;66:491-499.  [PubMed]  [DOI]
112.  Lokkerbol J, Adema D, Cuijpers P, Reynolds CF, Schulz R, Weehuizen R, Smit F. Improving the cost-effectiveness of a healthcare system for depressive disorders by implementing telemedicine: a health economic modeling study. Am J Geriatr Psychiatry. 2014;22:253-262.  [PubMed]  [DOI]
113.  American Psychiatric Association Telepsychology 50-State Review. 2013;.  [PubMed]  [DOI]
114.  Ferrer DC, Yellowlees PM. Telepsychiatry: licensing and professional boundary concerns. Virtual Mentor. 2012;14:477-482.  [PubMed]  [DOI]
115.  Shore JH, Bloom JD, Manson SM, Whitener RJ. Telepsychiatry with rural American Indians: issues in civil commitments. Behav Sci Law. 2008;26:287-300.  [PubMed]  [DOI]
116.  Baker DC, Bufka LF. Preparing for the telehealth world: Navigating legal, regulatory, reimbursement, and ethical issues in an electronic age. Professional Psychology: Research and Practice. 2011;42:405-411.  [PubMed]  [DOI]
117.  Gunter TD, Srinivasaraghavan J, Terry NP. Misinformed regulation of electronic medicine is unfair to responsible telepsychiatry. J Am Acad Psychiatry Law. 2003;31:10-14.  [PubMed]  [DOI]
118.  Jefee-Bahloul H. Telemental health in the middle East: overcoming the barriers. Front Public Health. 2014;2:86.  [PubMed]  [DOI]
119.  Williams TL, May CR, Esmail A. Limitations of patient satisfaction studies in telehealthcare: a systematic review of the literature. Telemed J E Health. 2001;7:293-316.  [PubMed]  [DOI]