Culturally adapted pictorial screening tool for autism spectrum disorder: A new approach

Abstract

AIM

To assess the performance of a newly designed, culturally adapted screening tool for autism spectrum disorder (ASD).

METHODS

Items for the screening tool were modeled from already documented checklists and diagnostic criteria for ASD. Each item in text was paired with a photograph that illustrated the written content, which was in the 2 main local languages. The final product had 21 items and was named the pictorial autism assessment schedule (PAAS). Performance of PAAS was tested on a clinical sample of 18-48 mo old children, diagnosis naïve, presenting with developmental deficits. Mothers completed PAAS checklist. Based on clinical diagnosis, which was taken as the gold standard, children were later grouped into ASD (Group 1) and non-ASD developmental disorders (Group 2). Mothers of a control sample of typically developing children also completed PAAS (Group 3).

RESULTS

A total of 105 children (Group 1-45, Group 2-30, Group 3-30) participated in the study. Mean age of Group 1 and Group 2 were 36 and 40 mo respectively. Majority were male in all 3 groups. Performance of PAAS in discriminating between ASD and non-ASD developmental disorders was sensitivity 88.8%, specificity 60.7%, positive predictive value (PPV) 78.4%, negative predictive value (NPV) 77.2%, likelihood ratio (LR+) 2.26, and LR- 0.18. Performance of PAAS in discriminating between ASD and typical development was sensitivity 88.0%, specificity 93.3%, PPV 95.2%, NPV 84.0%, LR+ 13.3 and LR- 0.12. The results indicated that that a positive result from PAAS was 2.26 times more likely to be found in a child with ASD than in a child with non-ASD developmental disorder. A positive result from PAAS was 13.3 times more likely to be found in a child with ASD than in a child with typical development.

CONCLUSION

PAAS is an effective tool in screening for ASD. Further study is indicated to evaluate the feasibility of using this instrument for community screening for ASD.

Key Words: Autism spectrum disorder, Screening tool, Culture, Ethnicity, Parent self-assessment, Pictorial

Core tip: Two previous studies in Sri Lanka showed that mothers had difficulty in comprehending and accurately responding to symptom of autism spectrum disorder (ASD) given in written text in a screening tool. The possible reason was cultural, where mothers did not perceive social deficits. To overcome this barrier, a screening tool was designed where each item on the checklist of key features of ASD was paired with a compatible photograph to improve comprehension. The new tool was tested on children with ASD, non-ASD developmental disorders and typically developing children. The new tool showed high sensitivity in discriminating between ASD and the other 2 groups.

INTRODUCTION

Autism spectrum disorder (ASD) is neurodevelopmental in nature, where deficit in social interaction and social communication is the most prominent behavioral feature. ASD is a biological disorder and therefore, the diagnostic criteria are similar irrespective of ethnic and racial differences. However, description of behavioral symptoms of ASD, their interpretation, and the level of acceptance is known to vary widely across cultures[1]. As a result, culture can influence the outcome in surveillance and screening and arriving at a valid diagnosis. It has been suggested that some cultures may not recognize ASD as a disorder or may group individuals with ASD under another diagnostic category[1]. Although supportive evidence is available in this regard, systematic research is limited[1].

ASD is considered the commonest developmental disorder and the importance of early identification and intervention is well accepted. Several screening and diagnostic tools for early detection are available for use in community and clinical settings. Almost all these instruments were developed in high-income countries and were not designed to consider cultural and ethnic variables or influences in using them[2]. Hence, their use in culturally diverse populations has been a challenge, with an added risk in adversely affecting the true estimates in epidemiological studies[1,2]. Even in the United States, marked disparity has been shown in diagnosis of ASD in different ethnic groups on community based screening. This difference in rates was attributed to parental reporting, level of availability of services for ethnic minorities, socioeconomic status and heterogeneity of presentation[3-5]. Other similar studies concluded that ascertainment issues, environmental risk factors, and genetic susceptibility may have influenced the observed differences[1]. Findings that contradict these facts are also available. A long term follow up of birth cohorts in the United States failed to find any differences in prevalence rates between racial and ethnic groups[6]. Also, early childcare providers in under-served communities using a screening tool, effectively identified young children for ASD in preschool/daycare settings, thus providing early diagnosis and access to intervention[7]. Among Asian populations, delay in seeking help for a child with ASD is explained on cultural beliefs of parents and family. When compared to the median age of 15-19 mo for seeking treatment in the West, that for a Sri Lankan and Indian clinical cohorts were 35.8 mo and 25.7 mo respectively[8,9]. In a four country (United States, United Kingdom, South Korea, Israel) study that compared relationship between culture and symptoms of comorbid psychopathology in those with ASD, the authors concluded that cultural factors, such as views about typical behaviour should be taken into account when examining symptoms of comorbidity in children with ASD[10].

Comparison of sensitivity and specificity of screening tools for ASD have shown that their accuracy is moderate, which limits their use in isolation and in making decisions on diagnosis[11]. At the same time, formal screening tools and general developmental testing provide critical data as brief clinical observations may not reliably detect ASD risk where atypical behaviors are present[12]. Although diagnostic and screening instruments have been translated into many languages and used in ethnically diverse populations, the effect of potential cultural confounds on the validity and reliability of these instrument have not been thoroughly assessed[1]. Also, most widely available development, communication and behavior screening tools for young children, such as Ages and Stages Questionnaire (ASQ-3), Parents Evaluation of Developmental Status (PEDS) and Child Development Inventory among others, lack the sensitivity to screen for ASD.

The Modified Autism Checklist for Toddlers (M-CHAT) is the most commonly studied screening instrument[13]. M-CHAT is a 23 item checklist of symptoms and is used for screening of 16-30 mo olds. It is available as a free on-line version and is self-administered to parents/guardians and interpreted by pediatric providers in the context of developmental surveillance[14]. More recent revised and follow-up version (M-CHAT-R/F) claims to reduce false positive rates and better detection than the original M-CHAT[15]. A positive predictive value of 50% has been found with M-CHAT-R/F when used in community settings. M-CHAT has been translated to several languages. Some have shown satisfactory reliability when used in culturally varied populations while some did not[16-23]. For example, in a large community survey, an Arabic validation of M-CHAT failed to identify a substantial proportion of children, 18-24 mo of age suspected to have ASD[16]. However, a Chinese version showed sensitivity as high as 93% on a similar age group[21]. Training of professionals to use existing tools was feasible and effective in an Iranian study for screening of preschool children[24]. Q-CHAT is another related tool found to be effective and reliable in screening at 18-24 mo[25]. Adapted versions of existing tools have been criticized for their flawed research methods as a reason for poor performance. Differences between the psychometric properties of the original and adapted versions were common, indicating the need to obtain normative data on populations to increase the utility of the translated tools[26]. Also, in a systematic review on the cultural adaptation of autism screening in ten languages, it was found that the cultural adaptation process was not always clearly outlined and often did not follow the recommended guidelines[1].

Our study investigated the sensitivity and specificity of a new screening tool for ASD. Justification for development of such a tool came from two previous research outcomes. First was in using a translated M-CHAT in a total population screening at 18-24 mo of age[27]. The study found that sensitivity of M-CHAT was only 25%. Second was a study on the presenting complaints in a clinical cohort of children later diagnosed with ASD. Abnormal play behavior and social un-connectedness were presented as key problems in only in 1.2%, which was in marked contrast to 82.3% seeking help for delayed speech development[8]. Possible reasons for both findings were considered as socio-cultural influence where parents attributed less importance or failed to notice deficits in social behavior. The alternative was to develop a tool that may reduce these cultural barriers to screening.

MATERIALS AND METHODS

Development of the screening tool

The checklist of items for the screening tool was adopted from several sources, but was not directly translated. The main source was diagnostic criteria for ASD in the Diagnostic and Statistical Manual of Mental Disorders 5^th edition (DSM V)[28]. Others were selected from M-CHAT[13] and “First Signs” from American Academy of Neurology and the Child Neurology Society[29]. The applicability to an age group of 18-48 mo and cultural factors that may influence the responses to ASD related enquiry were given due consideration when selecting and constructing the items of the checklist. Each item on the scale was worded as a question, for example: “Does your child bring over things to show you?” These items were originally written in Sinhala language and also translated into Tamil. Sinhala and Tamil are the 2 main languages spoken in Sri Lanka. All items were checked for accuracy of meaning, clarity and cultural appropriateness by experts in English and Sinhala/Tamil languages.

Design of pictorial scale

Each item on the checklist was paired with a photograph that illustrated the message in text. All photographs were taken on typically developing children together with an adult, either mother or father, where relevant. Informed written consent was obtained from the parent following which, the children and adults were initially coached on participating for the requirement of the photograph.

The written items and the matching photographs (facing each other on the opposing pages) were compiled into a manual. The end product was a list of 21 items where 20 items carried photographs to match. The last item was in text only (item 21) as it was difficult to convey meaning in a photograph. The self-assessment responses by parent were recorded on a separate sheet giving the numbered items in text with a choice of “yes” or “no” as the response. A “Yes” response to items 15, 16 and 21, and a “No” response to all other items was taken as positive indicators for ASD. Four or more positive indicators according the above scoring were taken as positive for ASD. Appropriate response to each item in support of ASD was counted as one.

Table 1 gives the English translation of items of PAAS. Clarity and comprehensibility of the text items and the compatibility of the text to the accompanying photograph in the compiled manual was further reviewed by a random sample of different grades of healthcare personnel and members of the general public. This was done on request and changes were made as necessary.

Table 1 English translation of items of Pictorial autism assessment schedule.

Does your child bring over things to show you

Does your child enjoy being thrown up and down on your lap

Does your child enjoy playing hide and seek

Does your child show pretend play

Does your child point to request

Does your child play with toys appropriately rather than mostly mouth or break them

Does your child attempt to imitate your actions

Does your child show an interest in other children

Does your child show willingness to share toys with others

Does your child look at your face when you hold an object in front of you

Does your child imitate your facial gestures

Does your child reciprocate affectionate gestures from you

Does your child look directly at your face on request

If you point at something far away, does your child look in that direction

Does your child watch rotating objects such as a fan or wheels for long periods

Does your child show repetitive purposeless finger movements

Does your child respond when called by name

If you point at something nearby, does your child look in that direction

Does your child join in a play of another child

Does your child point and show something that interests him

Does your child often appear as if he is in his own world

The completed scale was named Pictorial autism assessment schedule (PAAS).

Assessment of performance of scale

Performance of PAAS was evaluated on children selected from consecutive new referrals to a specialist developmental and child mental health outpatient service in a tertiary care pediatric hospital. They were either self-referrals or referrals from other pediatric services. The child was included in the sample if he/she was: (1) 18-48 mo of age; (2) was seeking help for a developmental problem; and (3) not had a developmental assessment or intervention prior to entry into the study. The last inclusion criterion was applied to avoid bias in responding to PAAS from exposure to ASD specific information from previous assessments. Mothers of all children included in the sample completed PAAS prior to the clinical assessment of the child. The mothers were unaware of the child’s diagnosis at the time. Detailed information gathering from mother, observation of child’s behavior and DSM V criteria was used to include or exclude ASD. The clinical diagnosis of ASD or non-ASD developmental disorder, which was carried out by a team of senior clinicians, was taken as the gold standard. In addition, those children diagnosed with ASD were assessed on Childhood autism Rating Scale (CARS) to further establish the clinical decision and estimate the severity. Diagnosis of developmental disorder was also made on clinical assessment and laboratory investigations where indicated.

Study and control samples

The sample was divided into 2 groups, based on the diagnosis. Group 1 - consisted of children who earned a diagnosis of ASD; Group 2 - were children diagnosed with a developmental disorder but not ASD. A control group of children, 18-48 mo of age, with typical development (Group 3) were randomly selected from general pediatric outpatient clinics at the same hospital. Absence of developmental problems in Group 3 children was established from their clinical records. Children with any doubt about their developmental status in the records were excluded from the sample. Mothers of the control group children also completed PAAS.

Statistical analysis

Frequency distribution of data was analyzed using SPSS version 16. Sensitivity, specificity, positive and negative predictive values and positive and negative likelihood ratios was calculated on performance of PAAS in detecting ASD and discriminating from non-ASD. Approval was obtained from the institutional ethical review committee of the Lady Ridgeway Hospital for Children, where the study was conducted.

RESULTS

Characteristics of the study population

A total of 105 children whose parents completed PAAS, 45 were later diagnosed with ASD (Group 1), 30 with non-autistic developmental delay (Group 2), and 30 had typical development (Group 3).

Table 2 gives the characteristics of children who participated in the study.

Table 2 Characteristics of children who participated in the study.

Variable	Group 1 n = 45	Group 2 n = 30	Group 3 n = 30
Mean age in months (range, SD)	36.5 (18-48, 12.02)	40.2 (18-48, 12.82)	31.8 (19-44, 9.77)
Sex
Male	38 (84.4)	22 (73.3)	17 (56.7)
Female	7 (15.6)	8 (26.7)	13 (43.3)
Screening positive (%)	40 (88.8)	9 (30.0)	1 (3.33)
Screening negative (%)	5 (11.1)	21 (70.0)	29 (96.7)
Maternal education
Primary education	3 (6.7)	2 (6.7)	6 (20.0)
Secondary education	19 (42.2)	22 (73.3)	22 (73.3)
Tertiary education	23 (51.1)	6 (20.0)	2 (6.7)

Performance of the scale

Table 3 gives the performance of PAAS in discriminating between ASD, non-ASD developmental delay/disorder and typical development

Table 3 Performance of Pictorial autism assessment schedule in discriminating between autism spectrum disorder, non-autism spectrum disorder developmental delay/disorder and typical development.

Result	Group 1 (n = 45) vs Group 2 (n = 30)	Group 1 (n = 45) vs Group 3 (n = 30)
Sensitivity	88.80%	88.00%
Specificity	60.70%	93.30%
PPV	78.40%	95.20%
NPV	77.20%	84.00%
LR+	2.26	13.3
LR-	0.18	0.12

These results indicate that a positive outcome from PAAS was 2.26 times more likely to be found in a child with ASD than in a child with non-ASD developmental disorder. A positive result from PAAS was 13.3 times more likely to be found in a child with ASD than in a child with typical developmental. Similarly, a negative result from PAAS was 0.18 time and 0.12 times more likely to be found with ASD than with non-ASD developmental disorder and typical development respectively.

CARS scores on Group 1 ranged from 33-45 (mean 39.4, SD 3.988) indicating that all were in moderate to severe range for ASD. Test-retest reliability of PAAS was 95.7%. The time taken by mothers to complete PAAS was 15-20 min.

DISCUSSION

PAAS was an attempt to overcome a cultural barrier to identifying symptoms of ASD by adding a visual aid to facilitate recognition. When compared to a text only scale, pairing the item with a photograph improved the comprehension and identification of the symptom and better accuracy of responses by mothers. Also, PAAS had an administration time of 15-20 min, which increased its user friendliness. The nature of the cultural barriers that were overcome in PAAS is not entirely clear. However, it was inferred in a previous study on community-based screening in Sri Lanka that a disregard for social interactional deficits, stigma, and an over-riding wish for the child to be normal may all contribute to a false-negative result in screening for ASD[27].

Our results showed that PAAS performed well in identifying ASD. This was evident from a sensitivity of 88.8% in discriminating between ASD and non-ASD developmental disorder, and 88.0% between ASD and typical development. In comparison, in the previous community based study, the respective results for M-CHAT were sensitivity of 25%, specificity of 71%, PPV of 0.13 and NPV of 0.85[27]. These discrepancies were evident despite the fact that item lists in both M-CHAT and PAAS are directly related to core behaviors of ASD. The high LR+ of 13.3 in the current study indicated a good discriminatory power of PAAS between ASD and typical development, with a specificity of 93.3%. In comparison, at a specificity of 60.7% and LR of 2.26, performance of PAAS was less satisfactory in discriminating between ASD and non-ASD developmental disorders. The possible explanation is that some children with other developmental disorders in the sample may have had comorbid ASD like behavior.

In developing the tool, cultural adaptation was implemented in several ways. Firstly, the photographs used were that of local children. Secondly, the items for the checklist were conceptualized and worded in the local language in the first instance, rather than translated from English. All items were ASD specific, giving the tool the required face validity. Also, the high sensitivity of the tool indicated satisfactory construct validity against the gold standard of clinical assessment. Other similar adaptations to improve performance of ASD screening in different cultural settings are known. In a Japanese study, using a list of most discriminative in a short form screener improved performance of M-CHAT[30]. Also, the Indian scale for assessment of autism (ISAA) and INCLEN diagnostic tool for autism Spectrum disorder (INDT-ASD) are culturally adapted new screening tools[31,32].

Use of picture based illustrations in assessment and screening tools to facilitate comprehension is well known. For example, line-drawings and clip-art are used in scales for assessment of pain, body-image and anxiety, especially in children. However, such scales are few when compared to the vast number in written text alone. More recent studies have used pictorial scales where accurate comprehension is compromised by literacy level of the respondents, or the traditional instrument took too long to complete[33-35]. Some of these scales have used only pictures and no text. With regard to ASD, a pictorial scale is available for assessment of joint attention in infants and preschoolers, which shows good validity[36].

Using an arbitrary rather than a calculated cut-off score of 4 positive items is a limitation of our scale. However, in M-CHAT, positive response for only 2 critical items or 3 of the others is taken as positive for ASD. Similarly, in keeping with M-CHAT and other similar screening tools, “Yes/No” responses were implemented rather than a Likert scale. The tool was tested on an age group of 18-48 mo. This does not guarantee its performance on older children. The reason is that although core symptoms are that of social communication and social interaction, there is a wide variation in symptoms and behavior according to age, cognitive level, and severity. The sample being hospital based and small in size are other shortcomings. To be useful as a screening tool, PAAS should ideally be tested in primary healthcare setting and on a larger sample. The mother’s education level especially of the Group 1 children was relatively high, which may not match that of a community-based sample. Hence, its performance and feasibility in using in the community is yet to be examined, although high sensitivity and PPV indicate good potential value.

In conclusion, PAAS is an effective tool in screening for ASD. The addition of a visual aid in the form of photographs improved its sensitivity. Further study is indicated to evaluate the feasibility of using this instrument for community screening for autism.

COMMENTS

Background

Inconsistency of performance of screening tools for autism spectrum disorder (ASD) in culturally diverse populations is a challenge to accurate estimation of epidemiological data. There is limited research on applicability of screening instruments in such varied groups.

Research frontiers

Use of suitable modifications based on the knowledge about specific cultural values and beliefs in order to improve performance of screening tools is indicated.

Innovations and breakthroughs

The use of photographs to illustrate the text items in this screening instrument for ASD produced a high sensitivity and discriminatory power up to 13 times with non-ASD developmental disorders and typical development.

Applications

This tool was used in a hospital-based setting. In view of its high sensitivity, positive predictive value and positive likelihood ratio, the possibility exists in using it in hospital-based developmental clinics to screen ASD. However, further study is indicated before use in community setting.

Terminology

Screening for ASD in young children is actively promoted due to clear befit of early intervention. However, culturally diverse populations do not always respond accurately to existing screening tools. Hence, developing culturally adapted tools is useful.

Peer-review

The paper is very interesting.