Cognitive screening for adult psychiatric outpatients: Comparison of the Cognivue® to the Montreal Cognitive Assessment

Abstract

In this editorial we comment on the article by Cahn-Hidalgo D published in a recent issue of the World Journal of Psychiatry 2020; 10(1); 1-11. We focus on the importance of utilizing psychometrically valid cognitive screening tools when assessing for cognitive decline in older adults in a psychiatric outpatient setting. We compared the use of Cognivue^® to use of the montreal cognitive assessment (MoCA) as a cognitive screening tool. A total of 58 patients aged 55 and over participated in this comparison study. Patients completed cognitive screening on Cognivue^®, a new Food and Drug Administration-cleared computer screening device, and the MoCA. The results of patient performance using these two instruments were analyzed. Sixteen (28%) patients screened negative for cognitive impairment on both assessments. Forty-two (72%) patients screened positive on one or both of the assessments. There was 43% agreement between Cognivue^® and the MoCA in identifying patients with cognitive impairment, and individual subtests were weakly correlated. The MoCA was determined to be the preferred instrument due to its high sensitivity and specificity (100% and 87%, respectively) when screening for cognitive impairment. We propose that the use of Cognivue^® cognitive screening tool be closely reviewed until more research proves that the test meets the standards for reliability and validity. It is important for clinicians to remember that screeners should not be used to diagnosis patients with neurocognitive disorders; instead, they should be used to determine whether further evaluation is warranted. Additionally, misdiagnosing of neurocognitive disorders can pose unnecessary psychological and emotional harm to patients and their families and also lead to incorrect treatment and undue healthcare costs.

Key Words: Dementia, Cognitive screening test, Cognitive impairment, Psychological assessment, Neurocognitive disorder, Geriatric psychiatry, Cognitive decline

Core Tip: Practicing clinicians should utilize validated measures when screening for cognitive impairment among older adults. Based on their findings they should make recommendations for further evaluation and not use cognitive screening tools as diagnostic tools.

INTRODUCTION

Cognitive decline is the leading cause of functional impairment among older adults[1]. As the population of older adults in the United States continues to rise, recognition and prevention of neurocognitive disorders becomes increasingly important. Screening for cognitive deficits facilitates early identification of these disorders, which, in turn, helps providers determine when to refer patients to neurology, psychology, or geriatric specialists for more extensive evaluation. Further, recognition of cognitive impairments allows clinicians to more effectively monitor safety and adherence to treatment, determine when to include family in treatment/decision making, and make accommodations during visits (such as providing materials and instructions the patient can understand and remember). Early and accurate diagnosis enables clinicians to educate patients and their families on symptoms and prognoses and to advise on treatment and support options.

There are different types of neurocognitive disorders (i.e., mild and major) that vary in symptom presentation, degree of impairment, prognosis, and treatment. Screening for and differentiating among these conditions can be a challenge. When testing for cognitive impairments, an ideal screening tool should sample the various cognitive domains that are most often compromised. These domains include executive functioning, visuospatial skills, language, processing speed, attention, memory, abstraction, and psychomotor skills[2]. In addition to using the cognitive screening tool, direct observation of the patient and collection of collateral information from a close family member, friend, or caregiver can provide important details regarding symptoms and level of functioning. This information will assist clinicians in making informed decisions on how to proceed with further evaluation and treatment. Clinicians should not rely solely on cognitive screening tools to diagnose patients with neurocognitive disorders, as gathering additional information is imperative in confirming a diagnosis and providing the most appropriate treatment for patients. Incorrectly diagnosing any type of neurocognitive disorder can lead to mismanagement of symptoms, improper use of medications, anxiety and distress for patients and their families, and unnecessary health care costs.

CRITICAL EVALUATION OF SCREENING INSTRUMENTS

It is important to critically evaluate screening tools to ensure they are psychometrically valid. Currently, there are a number of readily available screening instruments from which to choose[3]. Among the more widely researched and utilized screeners are the montreal cognitive assessment (MoCA), saint louis missouri mental status (SLUMS), and mini-mental state examination (MMSE). The MoCA detects symptoms of dementia with 100% sensitivity and 87% specificity[4]. It has been shown to evaluate many cognitive domains that are impacted in the various types of neurocognitive disorders. The pen-and-paper tool is administered by a clinician and takes about 10 min to complete. Scores range from 0-30 (+1 for 12 or fewer years of education); a score of 26 or higher indicates “normal” cognitive functioning, while a score of 25 or lower indicates “impaired” functioning.

Cognivue^® is a recently introduced screening tool that is administered using a standalone computer and onscreen instructions. The instrument has been “cleared” by the Food and Drug Administration (FDA), signifying that the administration does not perceive it to pose any danger to patients when used as directed[5]. Notably, clearance of a “de novo medical device” implies there is no comparable instrument and, thus, imposes few, if any, requirements for comparative analyses. Cognivue^® provides scores ranging from 0-100, with a score of 75 or higher signifying “normal” cognitive functioning, a score of 51-74 signifying low-moderate cognitive impairment, and a score of 50 or lower signifying severe cognitive impairment. There are a few research studies on this device which have been company-funded and focused on comparing Cognivue^® results with that of the SLUMS and several other neuropsychological assessment tools. This research by Cahn-Hidalgo et al[6] was published in a recent issue of the World Journal of Psychiatry [2020; 10(1); 1-11].

In the Cahn-Hidalgo et al[6] article they noted correlations between various neuropsychological tests and the “components” of the Cognivue^®; however, it is unclear which subtests of the Cognivue^® fell under each of the five “components”. It did label the components as verbal processing, manual dexterity and speed, visual acuity, visuospatial and executive function, and speed and sequencing, which doesn’t align with the domains on the clinician report generated by the Cognivue^® (i.e., Visuospatial, Executive Function/Attention, Naming/Language, Memory, Delayed Recall, and Abstraction). The results from Table of their article highlight strong correlations (0.529 to 0.902) between verbal processing and the SLUMS naming task and Rey Auditory Verbal Learning Test; manual dexterity and speed with Groove Peg Board Task; visuospatial and executive function with Trails B and Judgment of Line Orientation; and speed and sequencing with Trails A (Cahn-Hidalgo et al[6], 2020). Correlations with other Cognivue^® components and neuropsychological tests administered had low to moderate correlations (0.003 to 0.408) also outlined in Table. There was no good indication in this research that the Cognivue^® tapped into the domains of attention, immediate memory, delayed recall, or abstraction, which are important areas to consider when screening for neurocognitive disorders. For example Cognivue^® presentation of stimuli is all visual, which is a limitation. After initial exposure a few seconds pass before the participant is given a multiple choice paradigm to recognize and respond. This brief delay can be categorized as a short-term memory process, but not a long-term memory one. Additionally when considering models of memory, recognition of stimuli in a multiple choice format is easier than free recall of information or encoding the stimuli to long-term memory[2]. Since recognition can be intact in some individuals with neurocognitive disorders, such as with vascular dementia or mild cognitive impairment, presentation of information in this way could lead to false-negatives. Additionally it is unclear how the Cognivue^® subtests measure executive functioning skills even though Cahn-Hidalgo et al[6] research suggests correlations with Trails B, an executive function test.

It is also important to highlight that correlations between subtests do not necessarily mean that they are valid or even that they measure the intended variables. For example, the naming task on the SLUMS had a strong correlation (0.529) with the “language” measures on Cognivue^®[6]. The SLUMS naming task requires the subject to verbally generate as many animals as they can in one minute and is intended to screen for aphasia and other language/speech disturbances. In comparison, the language section on Cognivue^® does not have a verbal component. The tasks consist of single letters or simple three-letter words being visually displayed on a screen and then presents the subject with a visual multiple-choice paradigm, which requires them to select what was previously presented from items that were not. While this task involves some elements of language, it does not assess the same area of the brain as a naming task that requires verbal fluency and word finding skills, which are commonly observed deficits in neurocognitive disorders like Alzheimer’s disease[2]. The cognitive domains measured by the Cognivue^® are not well defined or researched in comparison to other screeners and neuropsychological measures.

Importantly, there are potential conflicts of interest with the aforementioned article. The research was funded by the makers of Cognivue^® and the authors were employees or consultants for the company. Therefore it is important that studies with larger sample sizes are completed by unaffiliated researchers for validation of the Cognivue^®. Additionally the company did not use trained psychologists or clinicians to administer the neuropsychological assessments in their research, calling the validity of the results into question. These authors concluded that the Cognivue^® is either the equivalent or superior to the SLUMS when screening for cognitive impairment and “superior” for test-retest reliability[6,7]. They do admit more comparison studies are warranted; however they go on to infer that the Cognivue^® will be equivalent in terms of its sensitivity, specificity, and psychometric validity to commonly used screeners like the MoCA and MMSE[6]. Without more research, with larger sample sizes, it is not appropriate to suggest the Cognivue^® is more useful or accurate than other screening instruments. Furthermore the researchers claim the Cognivue^® reduces “costs” associated with screening for cognitive impairments; however the cost saving advantage of this device vs other tools has not been established.

Unfortunately there are limited validation studies of the Cognivue^®, especially ones that are not associated with or funded by the company. There has been research examining the use of the Cognivue^® with a small sample of MS patients, which was coauthored by the founder and chief executive officer of Cerebral Assessment Systems and inventor of Cognivue^®. This study compared Cognivue^® total scores to the paced auditory serial addition test (PASAT) (which assesses auditory information processing speed, attention, and flexibility) and symbol digit modalities test (SDMT) (which assesses visual processing speed and attention)[8]. The PASAT and SDMT are commonly used cognitive screeners and research tools when working with Multiple Sclerosis (MS) patients[9]. Smith et al[8] found strong correlation between the Cognivue^® Total Score and SDMT (0.79) and the PASAT (0.61)[8]. In 2020 Bomprezzi expanded this research and found moderate correlations (0.67) between the Cognivue^® Total Score and SDMT results in a small sample of MS patients[10]. The finding of these studies suggests the Total Cognivue^® score correlates with tests that are measuring elements of attention and processing speed.

Digital and computer based screeners and tests show promise for detecting cognitive impairments[11]. In addition to the Cognivue^® there has been development of different computerized cognitive screeners. For example the historical Clock Drawing Test has been transformed into a digital version. The five minute Digital Clock Drawing Test is registered as a FDA Class II medical device for cognitive screening[12]. The tablet uses a digitizing pen that captures and analyzes the drawing. One Harvard research study concluded the DCT clock showed “excellent discrimination” between individuals with cognitive impairment and controls[12]. Unfortunately much of the technology and test adaptations for these devices are new, with few studies, small sample sizes, and lack of evidence, making it risky to suggest that computerized testing should be used clinically for the detection, diagnosis, and monitoring of neurocognitive disorders without complete and validated research[11].

We compared the Cognivue^® to the MoCA to assess its ability to screen for cognitive deficits among older adults in a mental health outpatient clinic. Both instruments were administered to 58 adult clinic outpatients aged 55-89 years by trained personnel. The results showed 28% agreement between tests for patients who did not screen positive for cognitive impairment according to their scores. In contrast, 42 (72%) patients screened positive on one or both measures. Of all patients who screened positive, the tests showed only 43% agreement in terms of identifying patients who may benefit from further assessment. Both Cognivue^® and the MoCA independently identified 12 different patients as being positive for cognitive impairment. Demographics as well as MoCA and Cognivue^® scores are described in Table 1. As can be seen here, there may be particular risks for false positive results among older women using Cognivue^® and among patients who score close to the cutoff (24 or 25) using the MoCA.

Table 1 Demographics and montreal cognitive assessment and Cognivue^® Scores.

	Positive MoCA score, n = 12	Positive Cognivue® score, n = 12
Gender, n (%); Male; Female	5, (41.7); 7 (58.3)	0, (0.0); 12 (100.0)
Age, yr	63.1 (5.0)	68.0 (7.2)
Length of education, yr	15.5 (2.2)	15.4 (2.4)
MoCA score	24.3 (0.8)	27.1 (1.3)
Cognivue® score	81.3 (4.7)	62.7 (11.6)

Presented as mean (SD), unless otherwise indicated. MoCA: Montreal cognitive assessment.

Given the lack of agreement between these measures, we then determined whether correlations exist between domains of the MoCA and Cognivue^® and whether Cognivue^® measures the same or similar domains as the well-established MoCA. The results, presented in Table 2, suggest there are a few low to moderate correlations between subtests of the two instruments, in terms of ability to assess visuospatial abilities, naming ability, and attention. The results also indicated that most subtests, which purportedly measure the same domains, do not demonstrate commensurate correlations.

Table 2 Correlations between Subtests of the montreal cognitive assessment and Cognivue^®.

MoCA subtests	Cognivue® subtests	Correlation score	P value
Executive function/visuospatial	Cognivue visual salience	0.24815	0.0604
Executive function/visuospatial	Cognivue shape discrimination	0.26059	0.0482
Executive function/visuospatial	Cognivue motion discrimination	0.30570	0.0196
Executive function/visuospatial	Cognivue word memory	0.19058	0.1519
Executive function/visuospatial	Cognivue shape memory	0.35760	0.00591
Attention	Cognivue visual salience	0.43944	0.00061
Attention	Cognivue share discrimination	0.19740	0.1375
Attention	Cognivue motion discrimination	0.34035	0.0089
Attention	Cognivue word memory	0.25763	0.0509
Attention	Cognivue shape memory	0.42319	0.00091
MoCA naming	Cognivue letter discrimination	0.44421	0.00051
MoCA naming	Cognivue word discrimination	0.35821	0.00581
MoCA language	Cognivue letter discrimination	0.28987	0.0273
MoCA language	Cognivue word discrimination	0.09739	0.4670
MoCA delayed memory	Cognivue word memory	0.30907	0.0182
MoCA delayed memory	Cognivue shape memory	0.21664	0.1024
MoCA delayed memory	Cognivue letter memory	0.27064	0.0399
MoCA delayed memory	Cognivue motion memory	0.29831	0.0229
MoCA delayed memory	Cognivue word discrimination	0.19972	0.1328
MoCA delayed memory	Cognivue shape discrimination	0.30308	0.0207
MoCA abstraction	Cognivue shape discrimination	-0.03896	0.7715
MoCA abstraction	Cognivue motion discrimination	0.00276	0.9836

Bolded numbers represent significant correlations between the subtests.

¹Notes it was significant at the P < 0.005 level.

MoCA: Montreal cognitive assessment.

CONCLUSION

The findings of this limited study raise questions regarding the utility of Cognivue^® for its intended purposes. We compared instruments and based on our findings and previous research which determined that the MoCA is the preferable screening tool. While both instruments seemed comparable with regard to their acceptability to patients, the MoCA does require more time from trained personnel to administer. In addition, the use of MoCA is now restricted to trained users as there were significant variations observed in the quality of the tests that were administered and the potential liability that this issue causes to its users[13]. The training to administer and score the MoCA has been deemed necessary starting September 1, 2019. The users will have 1 year to complete their training and will continue during that time to access the test without any restriction. After September 1, 2020, the access to the test has been restricted to certified users. We believe the requirement of additional time is offset by the extensive body of research supporting its psychometric properties and the significant risks to patients when screeners result in misdiagnosis.

Our findings call into question claims pertaining to the domains that Cognivue^® measures, which are crucial for correctly identifying potential neurocognitive deficits. Most Cognivue^® subtests appear to place cognitive demands in the domains of visual ability, motor control, attention, processing speed, visual discrimination, and short-term memory/recognition. These areas are important; however, if patients have deficits in one or more of these domains, it may impact their performance on some, if not all, subtests of the Cognivue^® given the way tasks are presented. Therefore, results may be skewed, potentially creating false positive outcomes. Additionally the subtests do not appear to assess long-term memory, executive functioning, language, or abstraction. Clearly defining the subtests of the Cognivue^® is crucial in determining its efficacy as a screening tool. More research by unaffiliated researchers, on large samples of participants, is needed to determine what specifically the Cognivue^® subtests are measuring and what modifications can be made to improve its screening capabilities.

Practicing clinicians should be aware of the importance of identifying cognitive impairments among older adults. Screening tools may play an important role in the identification of cognitive impairments and should not be seen as an inconvenience, but as an essential part of optimizing patient care. A cognitive screening tool should not be chosen because it is new and easily administered, but because it is the most efficacious way to accomplish the task. We recommend that clinicians primarily use the MoCA for this purpose. Further, we propose that use of Cognivue^® be evaluated carefully, until its subtests are modified or more research proves that the test meets standards for reliability and validity. Inadequate evaluation and misdiagnosis of neurocognitive disorders can be distressing for patients and their families and lead to inappropriate treatment and unnecessary healthcare costs. It is important to remember that a cognitive screening tool should not be used in isolation to establish a diagnosis of neurocognitive disorder, rather, it should be used to assist clinicians in determining when further evaluation is indicated.