Cognitive screening for children: Piloting a new battery of interactive games in 4- to 8-year-old children

Introduction

Cognitive Assessment in Children

Measuring cognitive skills at a younger age is essential for a number of reasons: e.g. early detection of cognitive deficits, assessment of school readiness (the readiness of children to enter school in terms of physical, cognitive, social and emotional development), evaluation of school performance or revealing the strengths and weaknesses of an individual child. Despite this, freely available interactive cognitive screening tools for children are still limited for our language region. Widely used standard methods often require trained examiners (Collie & Maruff, 2003; Engler & Alfonso, 2020) or depend on teachers’ and parents’ reports (e.g., BRIEF or BRIEF-P; Ezpeleta et al., 2015; Roth et al., 2014). In addition, the task selection can be limited by age (tasks are often intended for children over the age of six, e.g., Wisconsin Card Sorting test, Grant & Berg, 1948).

The most frequently employed methods include the use of developmental scales and intelligence tests, such as the Cognitive Assessment System (CAS 2) or the Wechsler Intelligence Scale for Children (WISC) (Wechsler, 1949), complex neuropsychological batteries assessing three or more cognitive functions (e.g., Intelligence and Development Scale for children IDS, or the preschool version IDS-P; Grob et al., 2018), or tests assessing specific target abilities, such as attention, or executive functions (most often the Trail Making Test for children from 5 to 7 years old; Rosin & Levett, 1989). Importantly, it is critical to evaluate the maturation of executive functions in preschool screening as the beginning of school attendance requires flexible adaptation of behavior to different social contexts and rules. However, most of the preschool methods available in the Czech Republic are aimed at language acquisition, motor and graphomotor skills, and laterality (e.g., Edfeldt’s Reversal Test; Laterality test, Orientation test of school maturity), or are in the form of rating scales designed for teachers or parents (Matejcek & Zlab, 1972; Schürerova, 1977). Some of the available screening tools are also not suitable for the target preschool age group (e.g., Bayley III for children from 1month to 4yrs; Bayley, 2005).

Game-Based Assessment

With the progress of technology, there is an increasing effort to create new types of assessments that are computerized and self-administered. Such game-based assessment (GBA) present new possibilities, simplifying cognitive evaluation in children. GBA is portable, with digital measurement capabilities, engaging the children and reducing test anxiety (Hawkins et al., 2013; Lumsden et al., 2016; Mavridis & Tsiatsos, 2017). GBA studies are traditionally conducted in schools, concentrating on student’s evaluation, with the aim to improve the assessment of competencies, skills, or learning outcomes (Gomez et al., 2022). GBA also offers the option of online assessment which allows for widespread access and remote testing if required.

There are three types of GBA: (1) the measurement of game-related success criteria (such as the achievement of targets, completion time, etc.), (2) external assessment using tools such as pre-post questionnaires, debriefing interviews, essays, knowledge maps, and test scores, and (3) embedded assessment based on player response data (the use of click streaming, log file analysis, etc.). There are more than 1,000 computer-assisted interventions for children, but their application is more often used in education, rehabilitation, or training (e.g., Neuro-World) than in assessing cognitive skills (Amukune et al., 2022).

The availability of touchscreen technology has facilitated the adaptation of assessment procedures to children of younger age. The majority of four year olds (and a minority of two to three year olds) are nowadays able to use a tablet with simple gestures (tapping or swiping) and to follow animated instructions (Yadav et al, 2020). A current scoping review by Amukune et al., (2022) provides a good overview of these methods available for three to eight year-old-children. Among GBAs aimed at testing cognition we can list for example the Early Years Toolbox for iPad (http://www.eytoolbox.com), measuring emerging cognitive, self-regulatory, language, numeracy and social development (Howard & Melhuish, 2017), or EF Touch (https://eftouch.fpg.unc.edu/index.html; (M. T. Willoughby et al., 2010) concentrating on the assessment of executive functions. Some GBAs are available in the form of mobile applications, such as CoCon, a serious game assessing cognitive control, or the neuropsychological screening Towi, a series of games based on standardized tests evaluating cognitive performance also serving as a cognitive training tool (Rosetti et al., 2017). Despite the large number of existing GBA worldwide, to our knowledge there are no GBAs methods targeting the children population available in the Czech language. As the proposed online games were developed in the local language, there is no requirement for adaptation or translation of foreign applications, which may entail additional costs

Current Study

The goal of this study was to evaluate the usability (as defined by Quintana et al., 2020) of a newly developed child-friendly interactive screening tool, potentially applicable in the form of an online assessment with gamification features. The battery of tests was designed as a part of a broader project aimed at identifying potential risk factors predicting school success at preschool age. This tool should allow preliminary screening of children’s cognitive abilities in a fast and enjoyable way both in preschool and older-age children.

The screening tool aimed to address cognitive functions that cover cognitive domains traditionally assessed in school and preschool age and are associated with school performance and school success prediction (Alloway et al., 2005; St Clair-Thompson & Gathercole, 2006), which include attentional control, inhibitory control and working memory. These domains partially overlap with the executive functions as defined by Diamond (2013). The term "executive functions" is used to describe a broad range of cognitive processes that are involved in the control and coordination of information processing during goal-directed behavior (Fuster, 1997; Miller & Cohen, 2001). The ontogenetic development of executive functions relies on the maturation of associated brain areas, but also requires stimulation in a social context (Huizinga et al., 2018). These processes are particularly important in situations that are novel or demanding (e.g., the beginning of school attendance), and thus requiring flexible adjustment of behavior. Therefore, appropriate screening of development (maturity) of executive functions is essential for assessing the potential for school performance.

The tested prototype of the screening tool therefore contains three games, assessing the key cognitive functions: selective attention, inhibitory control, and working memory (specifically, the episodic buffer of working memory).

To address (1) selective attention, we used a nonverbal performance test based on traditional cancellation tasks (e g., Bates & Lemay, 2004; Hokken et al., 2023; Theeuwes, 1993), measuring visual attention. As (2) inhibitory control has been repeatedly shown to be associated with early childhood education (Irvan & Tsapali, 2020), we chose a go/no-go task paradigm with the main focus on inhibition and cognitive flexibility. To assess (3) short-term episodic memory, we based the memory game on the episodic-like memory (ELM) model (Clayton & Dickinson, 1998). This is regarded as a model of episodic memory, which involves the encoding of contextual information (i.e., "What", "Where", "When"). This concept was already successfully used in our previous studies with adults and a pilot study in children. The task targets primarily the episodic buffer of working memory as described by Baddeley (2012), that represents temporary storage integrating information from other components of working memory and preserves a sense of time so that events occur in a coherent sequence.

The cognitive tasks, which are presented as individual games, are situated within the context of an interactive city map that employs the use of a guide character. The tasks were created predominantly for preschool children and the first grades of primary schools, therefore each game contains several levels with increasing difficulty. The games are designed in such a way that they can be easily managed, even by children as young as four years old, for example, when instructions are provided on the basis of spoken language.

The screening tool has been developed using the Unity game engine in the form of a desktop application available online and is compatible both with touch screens or mouse. The online platform was designed in order to provide remote access to the assessment and to continuously collect data for further refinement of age-appropriate standards of cognitive tests.

In this pilot study, we primarily aimed to test the usability of the proposed set of cognitive games. Beyond this primary objective, we have focused on preliminary testing the functionality of individual games, the comprehensibility of instructions, and the attractiveness of the games to children of different ages. Moreover, the effect of age is studied in means of cognitive performance. Finally, we aimed to test the preliminary construct validity of the games using direct comparison with selected traditional short paper and pencil tasks.

Methods

Participants

The participants were selected based on several criteria, including age (4 to 8 years old), Czech as the native language, the absence of significant visual or hearing impairments, and parental consent to participate in the study. The sample was recruited using the snowball method and the dataset was collected from March 2021 to June 2021 through in person visits to the children’s home. In person meetings took place with 18 out of the 28 initially contacted parents. The total sample size recruited was 24 children, and this sample also included some siblings (6 families). Two of the children had to be excluded from the sample due to acute medical issues (hand injury not associated with the study). Thus, the final sample consisted of 22 children (50% boys, 50% girls) without serious psychiatric or neurological disorders. Some basic demographic and personal information was gathered using a questionnaire created for parents including questions about children’s age, gender, kindergarten or school attendance, number of siblings, and health issues. Descriptives of the sample can be found in Table 1.

Procedure

The participants were recruited through contact with parents using the snowball method. Communication was established and in person meetings were arranged. All the assessments took place in a home setting, directly at the place of residence of the children. The testing was done under the supervision of an experimenter/psychologist. Signed written informed consent (approved by the Ethical Committee of the National Institute of Mental Health in the Czech Republic) was obtained before the beginning of the study from each parent, and each child was verbally asked if he/she was willing to participate. In case the child was unmotivated or tired, the testing was interrupted or terminated as necessary. During the in-person meeting, the procedure was explained to the parents, the written consent was signed, and necessary information about the children was gathered. The first interaction included the parent(s) to establish a safe atmosphere, then the assessment continued without the parent’s assistance. The whole assessment lasted about 45 minutes including instructing.

For the testing using interactive games a tablet was placed in front of the child, so the child could follow instructions given by the guide persona (visualized in the form of a friendly animal (Fig.1) and could respond by touching the touch screen. The aim of the pilot testing was to verify that the instructions are comprehensible and the child is capable of performing the tasks independently. Therefore, the experimenter did not interfere with the child’s actions during the assessment (only in case the instructions needed to be repeated or explained, especially to the youngest children).

The testing procedures (for a detailed description of individual tasks and games see below) followed a specific order. The assessment started with the Sentence repetition task, followed by the Bells test and interactive games. Three interactive games were tested in the following order: (1) the Book game (2) the Animal Feeding game, (3) the Toy Store game. The Verbal fluency test was applied in between each interactive game with randomized categories of listing (Animals or Vegetables between the first and the second game or the second and the third game). At the end of the assessment, the child received a small reward (e.g., coloring books). The order of tests was chosen to prevent the potential effect of fatigue in the tasks with higher attention demands and also to prevent potential interference between the two memory tasks, therefore applied in the beginning and the end. The standard paper and pencil tasks were applied in the beginning as we wanted to maintain the motivation of the children for these non-game tasks. Only short verbal fluency tasks were inserted between the cognitive games, serving as a short break after each cognitive game to prevent visual fatigue.

Measures

Game-Based Assessment

This screening tool was developed at the National Institute of Mental Health in Czech Republic. The prototype has been designed as an interactive city map (Fig. 1) with a virtual character guiding the child through the assessment using three different games to test selected cognitive functions. The screening tool is based on 2D animations and is suitable primarily for touchscreen (on a tablet or a touch notebook) but can also be used on a computer with a regular mouse. Each game was created as a prototype and tested repeatedly before the beginning of the study. The application is launched via a link (http://hry0421.skolniuspesnost.cz/www/) with a login code protecting the anonymity of the child, and all the data is automatically saved during task completion. Tasks were designed based on general cognitive experimental paradigms such as visual search, go/no-go, and episodic-like memory tasks. Particular games are displayed in the given order during the assessment. All tasks have multiple levels with increasing difficulty.

Figure 1. Main Screen of the Application—Interactive City Map.

Book Game

The first game is the Book game, inspired by the selective attention tasks (e g., Bates & Lemay, 2004; Hokken et al., 2023; Theeuwes, 1993) designed as an exercise book (Figure 2) with lists filled with symbols (48 symbols on 1 page). The goal is to select as many symbols as possible according to the visible template during a limited time period (60 seconds per level). This task is based on the principle of cancellation tasks (Matier et al., 1994; Pradhan & Nagendra, 2008), where the participant has to search for and cross out target stimuli embedded among distractor stimuli. The target symbol is displayed in the upper corner of the page throughout the entirety of the gaming session. This ensures that selective attention is assessed without the need for participants to recall the target symbol.The overall duration of the game is approximately 7 minutes. Target symbols differ on each level (Figure 2):

• Level 0: training
• Level 1 to 3: 1 target stimuli + 2 distractors (stimuli are more difficult to discriminate with increasing game level due to increasing similarity of the stimuli)
• Level 4: 1 target stimuli + 3 distractors

Figure 2. The Book Game - The Instructions and Correct/Incorrect Responses Shown during the Training Level 0; Game Levels 0 - 4 (described from the left)

The following outputs are recorded during the game:

• The number of correct responses (correctly marked symbols) per given time limit (60 seconds), indicating the search speed without the need for corrections;
• The number of incorrect responses (marked symbols which do not match the sample—errors), reflecting the inability to pay attention to the task according to the given rules.

For this pilot study it is not reliable to express the success rates in form of ratios of correct responses or errors with the total number of stimuli presented, as it was possible for the child to skip some pages in the book during the task (which disproportionately increases the total number of stimuli).

Animal Feeding Game

The second game, the Animal Feeding game, was inspired by the go/no-go task (e.g., Simpson & Riggs, 2006) used to assess executive functions, in particular inhibition and cognitive flexibility. This task is designed as an animal house with different types of animals used as stimuli (Figure 3). The instruction is to feed animals (target stimuli) by clicking on them, once they appear one after another in one of the visible windows. The goal is to react as fast as possible to the target stimuli (correct “go” animal) and ignore the distractors (incorrect “no-go” animal). The overall duration of the game is approximately 5 minutes with 30 stimuli per level (“go“ and “no-go“ stimuli in total). The target animals change with increasing level complexity:

• Level 0: training
• Level 1: 1 “go“ / 0 “no-go“ stimuli
• Level 2: 1 “go“ / 1 “no-go“ stimulus
• Level 3: 1 “go“ / 1 “no-go“ stimulus that was a target in the previous level
• Level 4: 2 “go“ / 1 “no-go“ stimulus

Figure 3. Animal Feeding Game Design (Main Screen on the Left, the Target Stimulus on the Right).

In this task, 30 stimuli were presented to the participant at each level. Stimuli were divided into target (“go”) stimuli and distractors (“no-go”; for level description see above). During the game, the following outputs were recorded:

• The total number of stimuli presented for each level
• The total number of “go” and “no-go“ stimuli for each level
• The total number of correct reactions to “go“ stimuli
• The total number of omitted “no-go“ stimuli
• The average reaction time of correct reactions across levels.

Two parameters that are indicators of success rate are calculated from these outputs: (1) the ratio of correct reactions out of the total number of target objects/distractors and (2), the ratio of correctly stopped reactions out of the total number of target objects/distractors. The parameters are expressed as values between zero and one (where 0 means absolute failure and 1 absolute success).

The following three parameters were utilized in the current study:

• Success rate of the correct reaction to the target stimuli (success rate ”go”).
• Success rate in stopping the incorrect reaction (success rate “no-go”). This parameter does not apply to level 1 containing only “go” stimuli.
• The average reaction time of correct reactions across levels.

These parameters represent different cognitive functions—the first reflects (success rate “go”) the correct reaction to the target stimuli associated with attention ability, and the second (success rate for “no-go”) reflects inhibitory control (the ability to stop the planned reaction when the “no-go“ stimulus is presented). The reaction time parameter represents the processing speed associated with the decision process.

Toy Store Game

The third game, Toy Store game, is designed to assess at short-term episodic memory and is based on the WWW (“What”, “Where”, “When”) model for the evaluation of episodic-like memory (Clayton & Dickinson, 1998; Clayton et al., 2001), as was applied in similar tasks (Eacott & Norman, 2004; Fajnerova et al., 2017; Vlcek et al., 2006). The task is aimed at the episodic memory, specifically at the episodic buffer of working memory (as was suggested by Baddeley, 2012) dedicated to linking information across domains to form integrated units of information with time sequencing (or episodic chronological ordering). Moreover, the episodic buffer is also assumed to have links to long-term memory (Baddeley, 2012).

The game is designed as a shelf in a toy shop, where different items (toys) are placed in specific positions. The goal is to remember which objects were selected, in what order, and what their original positions on the shelf were. This task has three phases (Figure 4): in the first phase, the children are asked to pick toys from the shelf in a specified order following visual and verbal instructions and put them in a box (Step 1). In the second phase, the child is presented with an expanded set of toys in a box—original toys and distractors (other toys) and they are asked to choose the toys selected previously in the first phase (Step 2 – “What”). The correct set of toys is presented before the last phase, even if the child made mistakes in the second phase. In the last phase, the child is instructed to put the toys back in the correct positions on the shelf in the same order as they were picked at the beginning (Step 3 – “Where”, “When”). The total duration of all game levels is approximately 10 minutes. The number of objects and distractors increases at each level:

• Level 0: training
• Level 1: 3 targets / 3 distractors / 6 positions (3 out of 6 correct)
• Level 2: 4 targets / 4 distractors / 8 positions (4 out of 8 correct)
• Level 3: 5 targets / 5 distractors / 10 positions (5 out of 10 correct)

Figure 4. Toy Store Game Design: Step 1–3 (From Left to Right).

Based on the described steps (1–3), the following parameters are recorded during the game:

• Objects (“What”): number of correctly selected objects (toys remembered as originally selected) among the distractors;
• Positions (“Where”): number of objects placed in the correct positions among the distractor positions;
• Order (“When”): the precision of responses within the "Order" parameter (expressed as 1 = all correct or 0 = mistakes made) for each level.

The correct order of objects was automatically recorded only as numbers of “all correct” or “mistakes made” for each level, hence the relevant number of correct responses is not available due to potential subsequent errors in order position. This is why this parameter was analyzed differently from the other two. To allow direct comparison, the categories of “What” and “When” were also transformed to “all correct” levels (the highest level where no error was made).

Paper and Pencil (Standard) Tasks

The following standard (paper and pencil) tasks were used in the study:

Bells test is a cancellation task used for the assessment of selective attention and executive functions and also for the diagnostics of unilateral spatial agnosia (Mancuso et al., 2019; Pureza et al., 2013). The goal is to find and mark all target stimuli (bells) placed among other distractors as fast as possible. The test material is in the form of a template containing 35 bells spread among 264 distractors on a paper sheet (299 black and white symbols depicting, in addition to bells, also horses, houses, and others). The tested person has no information about the total number of bells. The task ends once the person announces that she/he can no longer find any bells. For our purposes, there was no time limit, but the task can also be presented as time-limited when working with adults. The Bells test was used to test the preliminary construct validity of the Book game.

The Verbal fluency test was selected as a widespread tool, commonly used to assess cognitive performance, both in pedagogical and neuropsychological practice. It is part of various test batteries assessing cognitive functioning. Executive functions such as cognitive flexibility and inhibition control, as well as processing speed, have been reported as the strongest predictors of Verbal fluency test performance (Amunts et al., 2020). For that reason, we used this test to address the association with performance tested in the Animal Feeding game (“go” success rate). In the Czech Republic two types of verbal fluency tests are traditionally used – phonemic word fluency, requiring the participant to list words starting with a certain letter (most often N, K, and P), and semantic word fluency, where the task is to list words of a given category. The enumeration period is limited to one minute. For the purposes of this study, semantic word fluency with the categories Animals and Vegetables was used. The instructions were modified so that they were understandable for younger children (Cigler & Durmekova, 2018; Nikolai et al., 2015).

Sentence repetition task is a simple verbal memory test based on sentence repetition that was used to test the association with the Toy Store game. Following the initial instructions, the administrator then reads sentences of increasing difficulty, presented in a clear and audible manner, to the child. The task is to repeat the sentence in the exact same words. The length and complexity increase with every sentence. Each correctly repeated sentence is awarded two points and one point if there was a mistake corrected by the child. The task ends if the child makes mistakes without correction in two sentences in a row. The list of sentences used for our purposes contained 17 sentences and was adopted from NEPSY battery (Davis & Matthews, 2010) and translated into the Czech language independently by two persons. This test was used in our previous study concentrating on memory assessment in preschool children (Pisova, 2020).

The above listed methods were selected to address the overall cognitive performance of the tested children, would not be overly demanding and are manageable on top of the tested cognitive games. For this reason, the tasks were not selected to directly match the newly-designed interactive cognitive games and the targeted cognitive functions, as the evaluation of construct validity was not the primary aim of the presented study. Moreover, some more fitting test methods were not available as validated tests with normative data for children (especially in the local region).

Data Analysis

Analysis was performed using Microsoft Excel and Jamovi 2.3.19.0. Demographic data were analyzed by computing means, standard deviations, and frequencies. Verification of normal distribution was made using the Shapiro-Wilk test. Since the data were not normally distributed, non-parametric methods were chosen for further analysis. Descriptive statistics describing the performance of children in individual tests were applied. Friedman’s test for repeated measures and subsequent post hoc tests (Durbin-Conover) were used to assess the effect of increasing difficulty levels of cognitive games on tested performance. A Spearman correlation analysis was used to assess the effect of age and for comparison of the performance tested using interactive games and standard test methods. The results were evaluated at the 0.05 significance level.

Results

The descriptive statistics of the study sample is presented in Table 1. Other results are presented in the following order: the GBA (descriptives of interactive games presented separately for each task); Standard test assessment (the results of the standardized paper and pencil tasks) and the results based on comparisons between the interactive games and standard tasks. Finally, we present the effect of age on the performance in both sets of individual games and standard methods.

Table 1. Descriptives of the Sample.

Game-Based Assessment

Book Game

In the Book game, we analyzed the numbers of correct and incorrect responses at individual levels (for a detailed description see Methods).

Figure 5. Average Number of Correct and Incorrect Responses in Levels 1–4 of the Book Game.

Note. Error bars represent standard errors.

Descriptive statistics of the listed parameters can be found in Appendix in Table A1. The average number of correct responses decreased with every level, indicating increasing difficulty across levels. This assumption was verified using the nonparametric Friedman’s test, χ²(3) = 43.3, p < .001. The post-hoc analysis (Durbin-Conover) confirmed the difference between particular levels (for a detailed description of post hoc pairwise comparisons see Table A2). The increasing difficulty was also verified using Friedman’s test in the average number of incorrect responses, see Figure 5. The results were significant for this parameter as well, χ²(3) = 24.1, p < .001, but the post-hoc analysis (Table A2) didn’t confirm a significant difference between levels 2 and 3 in this cohort (for a more detailed description of test levels see Methods).

Animal Feeding Game

In this task, we analyzed the success rates of the reactions to “go” and “no-go” stimuli and the average reaction times calculated separately for each level (for detailed description see Methods).

Descriptive statistics for the selected parameters can be seen in Table A3. The average success rate (“go“ and “no-go”) decreased with increasing levels (see Figure 6), and we can therefore assume increasing difficulty across levels. This assumption was confirmed using the nonparametric Friedman’s test. The results were significant, Success rate “go“: χ²(3) = 23.1, p < .001; “no-go“: χ²(2) = 12.2, p = .002, indicating a difference in difficulty between levels. The post-hoc analysis (see Table A4) confirmed significant differences only between level 1 (level without “no-go“ stimuli) and the other levels (levels 2 to 4 with a different number of “go“ and “no-go“ stimuli), and between levels 2 and 4.

Figure 6. The Success Rate of “Go“ and “No-Go“ Reactions in Levels 1–4 of the Animal Feeding Game.

Note. Error bars represent standard errors.

On the other hand, the average reaction time increased with every level, and the difference between levels was verified using the Friedmans test, χ²(3) = 39.9, p < .001, confirmed also by the post-hoc analysis (Table A4).

Toy Store Game

Each level of the Toy Store game contains recollection of the following information: Objects (“What”), Positions (“Where”) and position Order (“When”; detailed description in Methods). The statistical analysis focused on 1) the number of correctly recollected information in each level and 2) number of levels that were performed without recollection error for each category (“What”, “Where”, “When”).

Table 2. Descriptives of the Correct Responses for Parameters What and Where of the Toy Store Game.

The average number of correct responses in remembering the designated objects (“What”) and their spatial position (“Where”) for levels 1 to 3 is shown in Table 2. The difference in the difficulty of levels was also analyzed in this task (using the nonparametric Friedman’s test) for both parameters separately. The difference between the tested game levels was significant for the parameter “What”, χ²(2) = 29.5; p < .001, while for the parameter “Where” the difference was not confirmed, χ²(2) = 3.77; p = .152.

To compare children’s performances in all parameters (“What”, “Where”, “When”), it was necessary to convert the results to categorical values as was the case with the parameter “When” (see above). The three types of errors were evaluated separately for each level, depending on whether the child managed to correctly determine all objects, their position, or their order (0 = at least one mistake made / 1 = all correct). For the purposes of further analysis, the number of the level reached without an error was assigned to each child. The results can be seen in Figure 7 showing the percentage of children who were able to finish the particular level (1–3) for the given parameter without errors.

The difficulty of the three parameters was analyzed separately using the Wilcoxon signed-rank test, and the post-hoc comparison confirmed a difference between the parameter “What” and other two parameters “Where” and “When”. However, no difference was found between the “Where” and “When” parameters (Table A5).

Figure 7. The Success Rate for Parameters “What”, “Where”, “When” in Levels 1–3 of the Toy Store Game.
The Percentage Scores Indicate the Proportion of Children Who Meet the Criteria (“All Correct“) for the Targeted Parameter.

Standard Cognitive Assessment

The descriptives of paper and pencil tasks used in the study sample are presented in Table 3.

Table 3. Descriptives of Paper and Pencil Tasks.

Comparison Between Game-Based and Standard Methods

Bells test: The number of correct visual target symbols (bells) that the child was able to mark was calculated (descriptives in Table 3). As described above, the maximum score was 35. The achieved score was correlated with the number of correct responses (marking the target stimuli) in the Book game as both tasks are designated to assess selective attention and are based on similar principles (searching target stimuli among distractors). The identified moderate correlation between these two tasks was statistically significant (r = .587, p = .007).

Verbal fluency test: Success in this task is evaluated according to the total number of words that the child was able to list within the time limit. Based on studies of semantic fluency, it can be assumed that the category Vegetables is more complicated than the category Animals (Nikolai et al., 2015). This was also confirmed by our data (descriptives in Table 3). Performance on this task was compared with the success rates (“go“ and “no-go“) in the Animal Feeding game (using the Spearman correlation coefficient. The results (Table 4) were significant (p < .05) for the success rate “go“ and the category Vegetables in all levels of the Animal Feeding game (level 1: r = .453; level 2: r = .465; level 3: r = .456; level 4: r = .542), but not for the other category. For the category Animals, the only significant result was the correlation with the success rate “no-go“ in level 4 of the Animal Feeding game (level 1: r = .214, p = .365; level 2: r = .124, p = .726; level 3: r = .202, p = .410; level 4: r = .304, p = .383).

Table 4. Correlation of Total Success Rate (“Go“ & “No-Go“) in Levels 1–4 of the Animal Feeding Game
With the Results of the Verbal Fluency Test (Categories “Animals” and “Vegetables”).

In addition, a possible association between the results of the Book game (the number of correctly marked symbols) and the Animal Feeding game (Success rate “go”) was analyzed, as both of the games are performance-type tasks employing attention. Correlation results (using the Spearman correlation coefficient) were significant supporting the association between the two tasks’ results (r = .759, p < .001).

Sentence repetition task: In this task, the maximum number of points that can be achieved by repeating sentences is 34 (see Methods). The number of points achieved was calculated for every participant (descriptives in Table 3). The task results were correlated with the results of the Toy Store game (Step 2—remembering objects), but no significant correlation was found (r = .014, p = .952).

Effect of Age on the Tested Cognitive Performance

The correlation between age and performance tested in individual tasks is presented in Table 5. The main findings were as follows: Correlations with age were significant in all tested methods except for the Toy Store game. The category Animals in the Verbal fluency test, and the success of stopping the reaction during the Animal Feeding game for “no-go“ stimuli were also not associated with age.

Table 5. Correlation of Task Results With Age.

Usability Outcomes

Since one of the study goals was to optimize the functionality of the screening battery as such, all technical errors, misunderstandings of instructions or other complications were recorded by the researcher during the testing. Based on this information, game modifications have been made, including the elimination of functional errors, output recordings (performance log) and game levels adjustments. Also a shortening of instructions (narrated by the in-game guide character) was made based on the children’s verbal feedback. They were too repetitive and some of the children described them as “boring”, or “lengthy”. Overall, the games were well received by the children. This information is based on the short questioning at the end of the testing, where the children were asked about their experience playing the games. Children referred to them mostly with terms such as “fun”, “cute”, wanting to play “one more time”.

Discussion

This pilot study focused on the usability testing of a newly developed battery of interactive games, intended for preliminary screening of selected cognitive functions in preschool children. The battery consists of three interactive games testing selected cognitive abilities: selective attention, inhibitory control, and episodic-like working memory. These targeted cognitive abilities have been selected as they overlap with particular executive functions, the maturation of which is critical for school readiness and performance (Huizinga et al., 2018).

The main outcome of the usability testing in the reported study is that the games were well received by the tested children, both in terms of the comprehensibility of the game instructions and their execution. The games usability was demonstrated in the tested sample of children of all ages, although effect of age was not observed in all tested games due to some study limitations (see Limitations), thus a more rigorous testing of these effects is needed. In addition, the correlations between the newly designed cognitive games and the standard cognitive tests were applied to address whether the developed games meet some of the preliminary requirements of construct validity that should be studied in more detail in future studies.

Effect of Age on Cognitive Performance

The research studies focusing on cognitive assessment in children show that cognitive functions represent a complex interconnected system (Rosenqvist et al., 2014) and that tested cognitive performance generally improves with age (Brüne & Brüne-Cohrs, 2006).

We have confirmed that tested performance improved with age in all standard tests selected for this study, except the Animals subtest of the Verbal fluency test. Based on the average number of listed words in each category and on in person interaction with children during the assessment, we hypothesized that younger children may know more words from the category Animals because it is more attractive and may reflect their interest more than the category Vegetables. The higher variability in the category Animals (independent of age) also suggests that this category can be strongly affected by the child’s interests.

For interactive games, we found the correlation of age and performance in some of the tested parameters, but not in all of them. Two of the tested interactive games, the Book game and the Animal Feeding game, require the child to distinguish between stimuli while searching for a target object following a simple rule. This ability increases around the child’s third year (Atkinson & Braddick, 2012) and attention develops significantly mainly in preschool age (Hanania & Smith, 2010; Zupan et al., 2018). Indeed, improved performance with increasing age has been identified in all measured variables associated with attention control—performance in the Book game and in the correct (“go“) stimuli success rate and reaction time in the Animal Feeding game. However, the successful inhibition of the “no-go“ reaction during the Animal Feeding game was not associated with age (see Table 5). This finding is in line with a study showing that the most significant developmental changes in inhibition control appear between the ages of 6–7 and 8–10 years (Pureza et al., 2013). As our study tested only children up to 8 years, we did not cover the crucial developmental period.

In addition, the Toy Store game performance did not show an association with age in any of the tested categories. We attribute this to the fact that the three tested parameters (“What”, “Where”, “When”) were recorded in the form of errors that are not directly comparable (see above). Therefore, it was necessary to choose a different type of analysis transforming the original error value (for each parameter separately) into the level number that the child achieves without error (“all correct” level), which might not be sensitive to the effect of age.

Relationship of GBA to Standard Methods

In the Book game, the number of correct responses (correctly marked symbols) during the given time limit (60 seconds) has been selected as the main performance variable. The Bells test (Gauthier et al., 1989) is designed in a similar way—the goal is to find as many target symbols as possible among distractors (so-called cancellation task). The moderate association between the number of correctly selected target symbols in the Bells test and the Book game supports the association between the cognitive constructs tested by the designed game aimed at testing a child’s ability to distinguish between various stimuli while searching for a target object following a simple rule (Pureza et al., 2013).

In the case of the Animal Feeding game, we performed a correlation analysis between the number of correctly selected target (“go“) stimuli and the Verbal fluency test results (number of named animals or vegetables). The association between these tasks is not so straightforward. However, it has been suggested that executive functions such as cognitive flexibility and inhibition control, as well as processing speed, are the strongest predictors of Verbal fluency performance (Amunts et al., 2020). This is also supported by our findings, as the correlation coefficients for all tested levels of the Animal Feeding game showed moderate association with the number of vegetables named by the children. Importantly, the number of named animals did not correlate with the Animal Feeding game results. This could be associated with potentially reduced sensitivity of the verbal fluency in naming animals, where higher scores can be observed in children showing personal interests in animals independent on their verbal fluency ability.

Moreover, we did identify a strong positive correlation between the correct scores of the Animal Feeding game (“go” stimuli) and performance (correct symbols) in the Book game. This supports the idea that performance in these two games requires not only selective attention but also effective visual search strategies and identification of rules-defined correct stimuli. This is also confirmed by a study (Pureza et al., 2013) showing that scores in the Bells test correlate with the maturation of executive functions.

Finally, we did not find a correlation between performance in the Toy Store game and the Sentence repetition task testing verbal memory in our sample. This result can be attributed to the fact that these two tasks tested different memory subtypes (episodic-like vs. verbal working memory), even though we originally assumed they contained a common working memory component. In the Toy Store game, the task is to select the correct objects and return them to the correct positions and in the same order in which they were previously selected (episodic buffer of working memory), whereas sentence repetition tests the child’s ability to remember and repeat sentences with increasing complexity (verbal working memory, so called phonological loop component in the Baddeley’s model of working memory).

We chose an episodic memory paradigm for a few reasons – this test mimics real-life situations, and it has high ecological validity. The instructions of this task are easy to understand. Moreover, we also used the same paradigm in our previous experiments focusing on adults which allows us to compare the task performance by means of ontogeny.

The results of episodic memory research using WWW tasks (Bauer et al., 2012; Bauer & Leventon, 2013; Hayne & Imuta, 2011) showed that it is easier for children to remember an object’s identity than to remember its position and the most difficult appears to be the order. However, an error in the order or position automatically generates another error—compared to an incorrectly selected object. This can lead to the impression that the order or position (parameters “When” and “Where”) are more difficult than the “What” parameter. In order to avoid such an artifact, we evaluate the level the child reached without any errors. Even after applying this data conversion, the difference in the difficulty of parameters “What”, “Where”, “When” was confirmed by our study, showing higher scores in the object identity recall in comparison to position and order. This can be explained by greater complexity in remembering the order and position than in the simple memory of the objects presented. Please note that in the Toy Store game the object identity is selected in a separate phase, while the order and position are selected as a combination, which makes the task more demanding.

Serious Games in Cognitive Screening in Children—Future Directions

We aimed to develop a set of interactive games for cognitive screening suitable both for preschool and younger school-age children. As children were able to complete all the games without any problems, we can assume that the games were manageable for the children of the targeted age and the instructions were clear to them. Moreover, children described the games as engaging and attractive. This was also confirmed by interviewing individual children. Verbal feedback from children allowed us to improve some game features (e.g., we changed the length of particular levels in the Book game (attention), reduced the number of stimuli in the Animal Feeding game for the next stage of testing, and also modified the recording of some data parameters).

Finally, the gradually changing performance associated with the increasing game level confirmed in all tested cognitive games suggests that they were designed well in terms of their difficulty for the targeted age group. This feature is extremely important to address the cognitive performance screened in children of variable ages. The above-reported correlations between the cognitive games and the standard test methods suggests that our interactive battery is testing targeted cognitive domains, however the construct validity of individual games has to be yet properly addressed in a future study.

The next step will be the optimization of the screening battery and validation through widespread testing in normative samples in schools and kindergartens, and also expanding the battery to assess more complex cognitive functions. For that reason, the final form of this cognitive battery already incorporates an additional game focused on the assessment of social cognition that is being tested in an ongoing study to address this developing cognitive skill.

Limitations

One of the limitations of this pilot study is its small sample size. The study was conducted during the Covid-19 pandemic lockdowns with limited possibility of testing in kindergarten or elementary schools and, for that reason, the testing was carried out individually, in a more time-consuming way. As this study was conducted with preschool children, some of the results may have been affected by fluctuating motivation, or by interaction with an unfamiliar person (researcher), a larger sample would therefore be beneficial.

Another limitation is that the children’s previous experience with touch screens wasn’t recorded. For subsequent testing this question is included. Current research (Jin & Lin, 2022) indicates that children who spend more time on a touch screen had a faster reaction time and greater accuracy in a game-like attention task (the Attentional Network Task ANT-C). On the other hand, their performance was weaker when it came to the executive function in the same task.

In addition, the parent’s highest level of education, and the number of siblings of the tested child, including sibling order were recorded, but this information was not processed in detail, given the research objectives and the small sample size. However, we consider it appropriate to explore the possible relationship between these factors and children’s performance in the consecutive study.

Some studies find gender differences, for example in tests of episodic and semantic memory (K. A. Willoughby et al., 2012) or in tests of verbal fluency, verbal comprehension, and graphical reproduction (Kraft & Nickel, 1995), while other studies show that gender differences in cognitive development are only minimal (Ardila et al., 2011; Hyde, 2016). Unbalanced distribution of age between girls and boys prevented us from analyzing the possible effect of gender on cognitive performance. Even though gender differences in performance were not our priority in this pilot study, this may be addressed in future studies, as the findings of previous studies are ambiguous.

Another study limitation is associated with the selection of paper and pencil tasks. While we were predominantly guided by the cognitive games tested in this study, other factors also affected method selection, such as the availability of methods validated for the child population and the time demands of the methods used. Therefore, some of the selected methods do not fully correspond with the construct validity of the tested cognitive games and this limitation should be addressed in the following study.

Conclusions

The primary goal of this study was to verify the usability and the comprehensibility of instructions of the newly-designed screening battery of cognitive games intended for cognitive screening in preschool and younger school children. Overall, the tasks are comprehensible and attractive to children, even at the age of 4 years. Although correlations of game results with standard tests suggest that our interactive battery is evaluating targeted cognitive domains, the construct validity of individual games should be thoroughly addressed in future studies. In conclusion, the preliminary results of this pilot study indicate that the battery is feasible in its current form, representing an easy and attractive tool for children’s cognitive screening.

Conflict of Interest

The authors have no conflicts of interest to declare.

Acknowledgement

The research leading to the presented results was conducted with the financial support of the Technology Agency of the Czech Republic project TL02000561, by the Grant Agency of the Czech Republic project nr. 22-10493S and with institutional support from the Ministry of Health CZ – DRO (NUDZ, 00023752). The game design of the presented tasks has been developed by the team of programmers and designers of the interactive studio 3dsense, namely Jiří Wild, Filip Vorel, and Jakub Řidký, with the financial assistance provided by this funding.

Appendix

Table A1. Descriptives for the Number of Correct and Incorrect Responses in Levels 1 to 4 of the Book Game.

Table A2. Pairwise Comparison of Levels 1-4 for the Number of Correct and Incorrect Responses in the Book Game.

Table A3. Descriptive Statistic for Success Rates ('Go' and 'No-Go') and Reaction time in the Animal Feeding game.

Table A4. Pairwise Comparison of the Success Rate of 'Go' Reactions and Average Reaction Time in Levels 1-4 of the
Animal Feeding Game.

Table A5. Pairwise Comparison of Parameters What, Where, When of the Toy Store Game. Wilcoxon W Paired Samples T-Test.