Editorial Open Access
Copyright ©The Author(s) 2021. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Psychiatr. Jul 19, 2021; 11(7): 265-270
Published online Jul 19, 2021. doi: 10.5498/wjp.v11.i7.265
Cognitive screening for adult psychiatric outpatients: Comparison of the Cognivue® to the Montreal Cognitive Assessment
Amanda F Rose, Alan F Gilbertson, Constance Cottrell, Rajesh R Tampi
Amanda F Rose, Alan F Gilbertson, Rajesh R Tampi, Department of Psychiatry and Behavioral Sciences, Cleveland Clinic Akron General, Akron, OH 44307, United States
Constance Cottrell, Office of Nursing Research and Innovation, Cleveland Clinic, Cleveland, OH 44106, United States
ORCID number: Amanda F Rose (0000-0003-3387-7362); Alan F Gilbertson (0000-0001-7276-9546); Constance Cottrell (0000-0001-8820-9746); Rajesh R Tampi (0000-0001-6754-6567).
Author contributions: Rose AF designed the overall project concept and study design, analyzed data, and outlined the manuscript; Gilbertson AF contributed to the content of the research study, interpretation of data, and manuscript; Cottrell C contributed to study design, review of literature, and revising manuscript; Tampi RR contributed to study design and revising manuscript.
Conflict-of-interest statement: There is no conflict of interest.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Amanda F Rose, PhD, Doctor, Department of Psychiatry and Behavioral Sciences, Cleveland Clinic Akron General, 1 Akron General Blvd., Akron, OH 44307, United States. rosea@ccf.org
Received: December 16, 2020
Peer-review started: December 16, 2020
First decision: April 6, 2021
Revised: May 10, 2021
Accepted: June 28, 2021
Article in press: June 28, 2021
Published online: July 19, 2021

Abstract

In this editorial we comment on the article by Cahn-Hidalgo D published in a recent issue of the World Journal of Psychiatry 2020; 10(1); 1-11. We focus on the importance of utilizing psychometrically valid cognitive screening tools when assessing for cognitive decline in older adults in a psychiatric outpatient setting. We compared the use of Cognivue® to use of the montreal cognitive assessment (MoCA) as a cognitive screening tool. A total of 58 patients aged 55 and over participated in this comparison study. Patients completed cognitive screening on Cognivue®, a new Food and Drug Administration-cleared computer screening device, and the MoCA. The results of patient performance using these two instruments were analyzed. Sixteen (28%) patients screened negative for cognitive impairment on both assessments. Forty-two (72%) patients screened positive on one or both of the assessments. There was 43% agreement between Cognivue® and the MoCA in identifying patients with cognitive impairment, and individual subtests were weakly correlated. The MoCA was determined to be the preferred instrument due to its high sensitivity and specificity (100% and 87%, respectively) when screening for cognitive impairment. We propose that the use of Cognivue® cognitive screening tool be closely reviewed until more research proves that the test meets the standards for reliability and validity. It is important for clinicians to remember that screeners should not be used to diagnosis patients with neurocognitive disorders; instead, they should be used to determine whether further evaluation is warranted. Additionally, misdiagnosing of neurocognitive disorders can pose unnecessary psychological and emotional harm to patients and their families and also lead to incorrect treatment and undue healthcare costs.

Key Words: Dementia, Cognitive screening test, Cognitive impairment, Psychological assessment, Neurocognitive disorder, Geriatric psychiatry, Cognitive decline

Core Tip: Practicing clinicians should utilize validated measures when screening for cognitive impairment among older adults. Based on their findings they should make recommendations for further evaluation and not use cognitive screening tools as diagnostic tools.



INTRODUCTION

Cognitive decline is the leading cause of functional impairment among older adults[1]. As the population of older adults in the United States continues to rise, recognition and prevention of neurocognitive disorders becomes increasingly important. Screening for cognitive deficits facilitates early identification of these disorders, which, in turn, helps providers determine when to refer patients to neurology, psychology, or geriatric specialists for more extensive evaluation. Further, recognition of cognitive impairments allows clinicians to more effectively monitor safety and adherence to treatment, determine when to include family in treatment/decision making, and make accommodations during visits (such as providing materials and instructions the patient can understand and remember). Early and accurate diagnosis enables clinicians to educate patients and their families on symptoms and prognoses and to advise on treatment and support options.

There are different types of neurocognitive disorders (i.e., mild and major) that vary in symptom presentation, degree of impairment, prognosis, and treatment. Screening for and differentiating among these conditions can be a challenge. When testing for cognitive impairments, an ideal screening tool should sample the various cognitive domains that are most often compromised. These domains include executive functioning, visuospatial skills, language, processing speed, attention, memory, abstraction, and psychomotor skills[2]. In addition to using the cognitive screening tool, direct observation of the patient and collection of collateral information from a close family member, friend, or caregiver can provide important details regarding symptoms and level of functioning. This information will assist clinicians in making informed decisions on how to proceed with further evaluation and treatment. Clinicians should not rely solely on cognitive screening tools to diagnose patients with neurocognitive disorders, as gathering additional information is imperative in confirming a diagnosis and providing the most appropriate treatment for patients. Incorrectly diagnosing any type of neurocognitive disorder can lead to mismanagement of symptoms, improper use of medications, anxiety and distress for patients and their families, and unnecessary health care costs.

CRITICAL EVALUATION OF SCREENING INSTRUMENTS

It is important to critically evaluate screening tools to ensure they are psychometrically valid. Currently, there are a number of readily available screening instruments from which to choose[3]. Among the more widely researched and utilized screeners are the montreal cognitive assessment (MoCA), saint louis missouri mental status (SLUMS), and mini-mental state examination (MMSE). The MoCA detects symptoms of dementia with 100% sensitivity and 87% specificity[4]. It has been shown to evaluate many cognitive domains that are impacted in the various types of neurocognitive disorders. The pen-and-paper tool is administered by a clinician and takes about 10 min to complete. Scores range from 0-30 (+1 for 12 or fewer years of education); a score of 26 or higher indicates “normal” cognitive functioning, while a score of 25 or lower indicates “impaired” functioning.

Cognivue® is a recently introduced screening tool that is administered using a standalone computer and onscreen instructions. The instrument has been clearedby the Food and Drug Administration (FDA), signifying that the administration does not perceive it to pose any danger to patients when used as directed[5]. Notably, clearance of a “de novo medical device” implies there is no comparable instrument and, thus, imposes few, if any, requirements for comparative analyses. Cognivue® provides scores ranging from 0-100, with a score of 75 or higher signifying “normal” cognitive functioning, a score of 51-74 signifying low-moderate cognitive impairment, and a score of 50 or lower signifying severe cognitive impairment. There are a few research studies on this device which have been company-funded and focused on comparing Cognivue® results with that of the SLUMS and several other neuropsychological assessment tools. This research by Cahn-Hidalgo et al[6] was published in a recent issue of the World Journal of Psychiatry [2020; 10(1); 1-11].

In the Cahn-Hidalgo et al[6] article they noted correlations between various neuropsychological tests and the “components” of the Cognivue®; however, it is unclear which subtests of the Cognivue® fell under each of the five “components”. It did label the components as verbal processing, manual dexterity and speed, visual acuity, visuospatial and executive function, and speed and sequencing, which doesn’t align with the domains on the clinician report generated by the Cognivue® (i.e., Visuospatial, Executive Function/Attention, Naming/Language, Memory, Delayed Recall, and Abstraction). The results from Table of their article highlight strong correlations (0.529 to 0.902) between verbal processing and the SLUMS naming task and Rey Auditory Verbal Learning Test; manual dexterity and speed with Groove Peg Board Task; visuospatial and executive function with Trails B and Judgment of Line Orientation; and speed and sequencing with Trails A (Cahn-Hidalgo et al[6], 2020). Correlations with other Cognivue® components and neuropsychological tests administered had low to moderate correlations (0.003 to 0.408) also outlined in Table. There was no good indication in this research that the Cognivue® tapped into the domains of attention, immediate memory, delayed recall, or abstraction, which are important areas to consider when screening for neurocognitive disorders. For example Cognivue® presentation of stimuli is all visual, which is a limitation. After initial exposure a few seconds pass before the participant is given a multiple choice paradigm to recognize and respond. This brief delay can be categorized as a short-term memory process, but not a long-term memory one. Additionally when considering models of memory, recognition of stimuli in a multiple choice format is easier than free recall of information or encoding the stimuli to long-term memory[2]. Since recognition can be intact in some individuals with neurocognitive disorders, such as with vascular dementia or mild cognitive impairment, presentation of information in this way could lead to false-negatives. Additionally it is unclear how the Cognivue® subtests measure executive functioning skills even though Cahn-Hidalgo et al[6] research suggests correlations with Trails B, an executive function test.

It is also important to highlight that correlations between subtests do not necessarily mean that they are valid or even that they measure the intended variables. For example, the naming task on the SLUMS had a strong correlation (0.529) with the “language” measures on Cognivue®[6]. The SLUMS naming task requires the subject to verbally generate as many animals as they can in one minute and is intended to screen for aphasia and other language/speech disturbances. In comparison, the language section on Cognivue® does not have a verbal component. The tasks consist of single letters or simple three-letter words being visually displayed on a screen and then presents the subject with a visual multiple-choice paradigm, which requires them to select what was previously presented from items that were not. While this task involves some elements of language, it does not assess the same area of the brain as a naming task that requires verbal fluency and word finding skills, which are commonly observed deficits in neurocognitive disorders like Alzheimer’s disease[2]. The cognitive domains measured by the Cognivue® are not well defined or researched in comparison to other screeners and neuropsychological measures.

Importantly, there are potential conflicts of interest with the aforementioned article. The research was funded by the makers of Cognivue® and the authors were employees or consultants for the company. Therefore it is important that studies with larger sample sizes are completed by unaffiliated researchers for validation of the Cognivue®. Additionally the company did not use trained psychologists or clinicians to administer the neuropsychological assessments in their research, calling the validity of the results into question. These authors concluded that the Cognivue® is either the equivalent or superior to the SLUMS when screening for cognitive impairment and “superior” for test-retest reliability[6,7]. They do admit more comparison studies are warranted; however they go on to infer that the Cognivue® will be equivalent in terms of its sensitivity, specificity, and psychometric validity to commonly used screeners like the MoCA and MMSE[6]. Without more research, with larger sample sizes, it is not appropriate to suggest the Cognivue® is more useful or accurate than other screening instruments. Furthermore the researchers claim the Cognivue® reduces “costs” associated with screening for cognitive impairments; however the cost saving advantage of this device vs other tools has not been established.

Unfortunately there are limited validation studies of the Cognivue®, especially ones that are not associated with or funded by the company. There has been research examining the use of the Cognivue® with a small sample of MS patients, which was coauthored by the founder and chief executive officer of Cerebral Assessment Systems and inventor of Cognivue®. This study compared Cognivue® total scores to the paced auditory serial addition test (PASAT) (which assesses auditory information processing speed, attention, and flexibility) and symbol digit modalities test (SDMT) (which assesses visual processing speed and attention)[8]. The PASAT and SDMT are commonly used cognitive screeners and research tools when working with Multiple Sclerosis (MS) patients[9]. Smith et al[8] found strong correlation between the Cognivue® Total Score and SDMT (0.79) and the PASAT (0.61)[8]. In 2020 Bomprezzi expanded this research and found moderate correlations (0.67) between the Cognivue® Total Score and SDMT results in a small sample of MS patients[10]. The finding of these studies suggests the Total Cognivue® score correlates with tests that are measuring elements of attention and processing speed.

Digital and computer based screeners and tests show promise for detecting cognitive impairments[11]. In addition to the Cognivue® there has been development of different computerized cognitive screeners. For example the historical Clock Drawing Test has been transformed into a digital version. The five minute Digital Clock Drawing Test is registered as a FDA Class II medical device for cognitive screening[12]. The tablet uses a digitizing pen that captures and analyzes the drawing. One Harvard research study concluded the DCT clock showed “excellent discrimination” between individuals with cognitive impairment and controls[12]. Unfortunately much of the technology and test adaptations for these devices are new, with few studies, small sample sizes, and lack of evidence, making it risky to suggest that computerized testing should be used clinically for the detection, diagnosis, and monitoring of neurocognitive disorders without complete and validated research[11].

We compared the Cognivue® to the MoCA to assess its ability to screen for cognitive deficits among older adults in a mental health outpatient clinic. Both instruments were administered to 58 adult clinic outpatients aged 55-89 years by trained personnel. The results showed 28% agreement between tests for patients who did not screen positive for cognitive impairment according to their scores. In contrast, 42 (72%) patients screened positive on one or both measures. Of all patients who screened positive, the tests showed only 43% agreement in terms of identifying patients who may benefit from further assessment. Both Cognivue® and the MoCA independently identified 12 different patients as being positive for cognitive impairment. Demographics as well as MoCA and Cognivue® scores are described in Table 1. As can be seen here, there may be particular risks for false positive results among older women using Cognivue® and among patients who score close to the cutoff (24 or 25) using the MoCA.

Table 1 Demographics and montreal cognitive assessment and Cognivue® Scores.

Positive MoCA score, n = 12
Positive Cognivue® score, n = 12
Gender, n (%); Male; Female5, (41.7); 7 (58.3)0, (0.0); 12 (100.0)
Age, yr63.1 (5.0)68.0 (7.2)
Length of education, yr15.5 (2.2)15.4 (2.4)
MoCA score24.3 (0.8)27.1 (1.3)
Cognivue® score81.3 (4.7)62.7 (11.6)

Given the lack of agreement between these measures, we then determined whether correlations exist between domains of the MoCA and Cognivue® and whether Cognivue® measures the same or similar domains as the well-established MoCA. The results, presented in Table 2, suggest there are a few low to moderate correlations between subtests of the two instruments, in terms of ability to assess visuospatial abilities, naming ability, and attention. The results also indicated that most subtests, which purportedly measure the same domains, do not demonstrate commensurate correlations.

Table 2 Correlations between Subtests of the montreal cognitive assessment and Cognivue®.
MoCA subtests
Cognivue® subtests
Correlation score
P value
Executive function/visuospatialCognivue visual salience0.248150.0604
Executive function/visuospatialCognivue shape discrimination0.260590.0482
Executive function/visuospatialCognivue motion discrimination0.305700.0196
Executive function/visuospatialCognivue word memory0.190580.1519
Executive function/visuospatialCognivue shape memory0.357600.00591
AttentionCognivue visual salience0.439440.00061
AttentionCognivue share discrimination0.197400.1375
AttentionCognivue motion discrimination0.340350.0089
AttentionCognivue word memory0.257630.0509
AttentionCognivue shape memory0.423190.00091
MoCA namingCognivue letter discrimination0.444210.00051
MoCA namingCognivue word discrimination0.358210.00581
MoCA languageCognivue letter discrimination0.289870.0273
MoCA languageCognivue word discrimination0.097390.4670
MoCA delayed memoryCognivue word memory0.309070.0182
MoCA delayed memoryCognivue shape memory0.216640.1024
MoCA delayed memoryCognivue letter memory0.270640.0399
MoCA delayed memoryCognivue motion memory0.298310.0229
MoCA delayed memoryCognivue word discrimination0.199720.1328
MoCA delayed memoryCognivue shape discrimination0.303080.0207
MoCA abstractionCognivue shape discrimination-0.038960.7715
MoCA abstractionCognivue motion discrimination0.002760.9836
CONCLUSION

The findings of this limited study raise questions regarding the utility of Cognivue® for its intended purposes. We compared instruments and based on our findings and previous research which determined that the MoCA is the preferable screening tool. While both instruments seemed comparable with regard to their acceptability to patients, the MoCA does require more time from trained personnel to administer. In addition, the use of MoCA is now restricted to trained users as there were significant variations observed in the quality of the tests that were administered and the potential liability that this issue causes to its users[13]. The training to administer and score the MoCA has been deemed necessary starting September 1, 2019. The users will have 1 year to complete their training and will continue during that time to access the test without any restriction. After September 1, 2020, the access to the test has been restricted to certified users. We believe the requirement of additional time is offset by the extensive body of research supporting its psychometric properties and the significant risks to patients when screeners result in misdiagnosis.

Our findings call into question claims pertaining to the domains that Cognivue® measures, which are crucial for correctly identifying potential neurocognitive deficits. Most Cognivue® subtests appear to place cognitive demands in the domains of visual ability, motor control, attention, processing speed, visual discrimination, and short-term memory/recognition. These areas are important; however, if patients have deficits in one or more of these domains, it may impact their performance on some, if not all, subtests of the Cognivue® given the way tasks are presented. Therefore, results may be skewed, potentially creating false positive outcomes. Additionally the subtests do not appear to assess long-term memory, executive functioning, language, or abstraction. Clearly defining the subtests of the Cognivue® is crucial in determining its efficacy as a screening tool. More research by unaffiliated researchers, on large samples of participants, is needed to determine what specifically the Cognivue® subtests are measuring and what modifications can be made to improve its screening capabilities.

Practicing clinicians should be aware of the importance of identifying cognitive impairments among older adults. Screening tools may play an important role in the identification of cognitive impairments and should not be seen as an inconvenience, but as an essential part of optimizing patient care. A cognitive screening tool should not be chosen because it is new and easily administered, but because it is the most efficacious way to accomplish the task. We recommend that clinicians primarily use the MoCA for this purpose. Further, we propose that use of Cognivue® be evaluated carefully, until its subtests are modified or more research proves that the test meets standards for reliability and validity. Inadequate evaluation and misdiagnosis of neurocognitive disorders can be distressing for patients and their families and lead to inappropriate treatment and unnecessary healthcare costs. It is important to remember that a cognitive screening tool should not be used in isolation to establish a diagnosis of neurocognitive disorder, rather, it should be used to assist clinicians in determining when further evaluation is indicated.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Psychiatry

Country/Territory of origin: United States

Peer-review report’s scientific quality classification

Grade A (Excellent): 0

Grade B (Very good): B

Grade C (Good): 0

Grade D (Fair): 0

Grade E (Poor): 0

P-Reviewer: Chakrabarti S S-Editor: Zhang L L-Editor: A P-Editor: Li JH

References
1.  American Psychological Association. Guidelines for the evaluation of dementia and age-related cognitive change. Am Psychol. 2012;67:1-9.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 29]  [Cited by in F6Publishing: 6]  [Article Influence: 2.9]  [Reference Citation Analysis (0)]
2.  Lee GP  The Little Black Book of Neuropsychology: A Syndrome-Based Approach. In: Mike RS, James GS. Archives of Clinical Neuropsychology 2011, Springer, New York.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 2]  [Article Influence: 0.2]  [Reference Citation Analysis (0)]
3.  Ranjit E, Sapra A, Bhandari P, Albers CE, Ajmeri MS. Cognitive Assessment of Geriatric Patients in Primary Care Settings. Cureus. 2020;12:e10443.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1]  [Cited by in F6Publishing: 1]  [Article Influence: 1.0]  [Reference Citation Analysis (0)]
4.  Segal-Gidan F. Cognitive screening tools. Clin Rev. 2013;23:12-18.  [PubMed]  [DOI]  [Cited in This Article: ]
5.  United State Food and Drug Administration  Device Approvals, Denials and Clearances [cited 20 April 2021]. Available from: https://www.fda.gov/medical-devices/products-and-medical-procedures/device-approvals-denials-and-clearances.  [PubMed]  [DOI]  [Cited in This Article: ]
6.  Cahn-Hidalgo D, Estes PW, Benabou R. Validity, reliability, and psychometric properties of a computerized, cognitive assessment test (Cognivue®). World J Psychiatry. 2020;10:1-11.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 4]  [Cited by in F6Publishing: 3]  [Article Influence: 4.0]  [Reference Citation Analysis (0)]
7.  Andrefsky J, Cahn-Hidalgo D, Benabou R. Superior test-retest reliability of a computerized cognitive assessment vs SLUMS during an 18-month longitudinal study. Am J Geriatr Psychiatry. 2020;28:S106-S107.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1]  [Article Influence: 1.0]  [Reference Citation Analysis (0)]
8.  Smith AD 3rd, Duffy C, Goodman AD. Novel computer-based testing shows multi-domain cognitive dysfunction in patients with multiple sclerosis. Mult Scler J Exp Transl Clin. 2018;4:2055217318767458.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 3]  [Cited by in F6Publishing: 2]  [Article Influence: 1.0]  [Reference Citation Analysis (0)]
9.  Benedict RH, DeLuca J, Phillips G, LaRocca N, Hudson LD, Rudick R;  Multiple Sclerosis Outcome Assessments Consortium. Validity of the Symbol Digit Modalities Test as a cognition performance outcome measure for multiple sclerosis. Mult Scler. 2017;23:721-733.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 309]  [Cited by in F6Publishing: 102]  [Article Influence: 77.3]  [Reference Citation Analysis (0)]
10.  Bomprezzi R  Cognitive impairment in patients with multiple sclerosis as assessed by objective computerized testing [cited 20 April 2021]. Scholars Literature 2020. Available from: https://www.scholarsliterature.com/journals/neurological-sciences-and-neurosurgery/fulltext/cognitive-impairment-in-patients-with-multiple-sclerosis-as-assessed-by-objective-computerized-testing.  [PubMed]  [DOI]  [Cited in This Article: ]
11.  Aslam RW, Bates V, Dundar Y, Hounsome J, Richardson M, Krishan A, Dickson R, Boland A, Fisher J, Robinson L, Sikdar S. A systematic review of the diagnostic accuracy of automated tests for cognitive impairment. Int J Geriatr Psychiatry. 2018;33:561-575.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 13]  [Cited by in F6Publishing: 5]  [Article Influence: 4.3]  [Reference Citation Analysis (0)]
12.  Brooks M  Five-minute digital clock test may speed early alzheimer's diagnosis [cited 20 April 2021]. Available from: https://www.medscape.com/viewarticle/949410.  [PubMed]  [DOI]  [Cited in This Article: ]
13.  Nasreddine ZS. MoCA Test Mandatory Training and Certification: What Is the Purpose? J Am Geriatr Soc. 2020;68:444-445.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 4]  [Cited by in F6Publishing: 3]  [Article Influence: 2.0]  [Reference Citation Analysis (0)]