|
1
|
Morleo F, Anselmi P, Vitanza A. Interactive, computer-based, and situated design for innovative formative assessment approaches. PLoS One 2025; 20:e0323301. [PMID: 40445897 PMCID: PMC12124497 DOI: 10.1371/journal.pone.0323301] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2024] [Accepted: 04/04/2025] [Indexed: 06/02/2025] Open
Abstract
Social and teamwork skills are essential for today's teachers, yet their assessment in authentic contexts is challenging. This study presents the design, development and validation of an innovative computer-based test, designed to assess collaborative and teamwork skills in preprimary and primary school teachers and referred to as the 'CoTeSt' (Collaborative and Teamwork Skill test). Based on Situated Action Theory, the test immerses participants in realistic team interactions using conversational agents within a narrative framework. Teachers are asked to solve problems, collaborate with virtual colleagues, and provide feedback. The test consists of 20 dichotomously scored items in both multiple-choice and short answer, and it was administered to 139 Italian teachers. Test validation involved qualitative and quantitative methods, confirming that the items actually assess the skills they were assumed to evaluate. Post-test interviews and group discussions highlighted the tool's user-friendly design and its potential to foster self-reflection, professional dialogue, and continuous skill development. The CoTeSt represents a meaningful step toward empowering teachers with critical social skills and fostering a culture of collaboration and growth in education.
Collapse
Affiliation(s)
- Federica Morleo
- University of Padua, Department of Philosophy, Sociology, Education and Applied Psychology (FISPPA), Padova, Italy
- Institute of Cognitive Sciences and Technologies, National Research Council (CNR-ISTC), Catania, Italy
| | - Pasquale Anselmi
- University of Padua, Department of Philosophy, Sociology, Education and Applied Psychology (FISPPA), Padova, Italy
| | - Alessandra Vitanza
- Institute of Cognitive Sciences and Technologies, National Research Council (CNR-ISTC), Catania, Italy
| |
Collapse
|
|
2
|
Ge X. Discriminability around polytomous knowledge structures and polytomous functions. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2025; 78:500-521. [PMID: 39555612 DOI: 10.1111/bmsp.12370] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 09/23/2024] [Accepted: 09/30/2024] [Indexed: 11/19/2024]
Abstract
The discriminability in polytomous KST was introduced by Stefanutti et al. (Journal of Mathematical Psychology, 2020, 94, 102306). As the interesting topic in polytomous KST, this paper discusses the discriminability around granular polytomous knowledge spaces, polytomous knowledge structures, polytomous surmising functions and polytomous skill functions. More precisely, this paper gives some equivalences between the discriminability of polytomous surmising functions (resp. polytomous skill functions) and the discriminability of granular polytomous knowledge spaces (resp. polytomous knowledge structures). Such findings open the field to a systematic generalization of the discriminability in KST to the polytomous case.
Collapse
Affiliation(s)
- Xun Ge
- School of Mathematical Sciences, Soochow University, Suzhou, China
| |
Collapse
|
|
3
|
Roselló Atanet I, Tomova M, Sieg M, Sehy V, Mäder P, März M. Generating learning guides for medical education with LLMs and statistical analysis of test results. BMC MEDICAL EDUCATION 2025; 25:458. [PMID: 40158187 PMCID: PMC11955137 DOI: 10.1186/s12909-025-06978-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/06/2024] [Accepted: 03/10/2025] [Indexed: 04/01/2025]
Abstract
BACKGROUND The Progress Test Medizin (PTM) is a formative test for medical students issued twice a year by the Charité-Universitätsmedizin Berlin. The PTM provides a numerical feedback based on a global view of the strengths and weaknesses of students. This feedback can benefit from more fine-grained information, pinpointing the topics where students need to improve, as well as advice on what they should learn in light of their results. The scale of the PTM, taken by more than 10,000 participants every academic semester, makes it necessary to automate this task. METHODS We have developed a seven-step approach based on large language models and statistical analysis to fulfil the purpose of this study. Firstly, a large language model (ChatGPT 4.0) identified keywords in the form of MeSH terms from all 200 questions of one PTM run. These keywords were checked against the list of medical terms included in the Medical Subject Headings (MeSH) thesaurus published by the National Library of Medicine (NLM). Meanwhile, answer patterns of PTM questions were also analysed to find empirical relationships between questions. With this information, we obtained series of questions related to specific MeSH terms and used them to develop a framework that allowed us to assess the performance of PTM participants and compose personalized feedback structured around a curated list of medical topics. RESULTS We used data from a past PTM to simulate the generation of personalized feedback for 1,401 test participants, thereby producing specific information about their knowledge regarding a number of topics ranging from 34 to 243. Substantial knowledge gaps were found in 14.67% to 21.76% of rated learning topics, depending on the benchmarking set considered. CONCLUSION We designed and tested a method to generate student feedback covering up to 243 medical topics defined by MeSH terms. The feedback generated with data from students in later stages of their studies was more detailed, as they tend to face more questions matching their knowledge level.
Collapse
Affiliation(s)
- Iván Roselló Atanet
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, AG Progress Test Medizin, Charitéplatz 1, 10117, Berlin, Germany.
| | - Mihaela Tomova
- Fakultät für Informatik und Automatisierung, Data-Intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Ehrenbergstraße 29, 98693, Ilmenau, Thuringia, Germany
| | - Miriam Sieg
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, AG Progress Test Medizin, Charitéplatz 1, 10117, Berlin, Germany
| | - Victoria Sehy
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, AG Progress Test Medizin, Charitéplatz 1, 10117, Berlin, Germany
| | - Patrick Mäder
- Fakultät für Informatik und Automatisierung, Data-Intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Ehrenbergstraße 29, 98693, Ilmenau, Thuringia, Germany
- Fakultät für Biowissenschaften, Friedrich Schiller Universität Jena, Schloßgasse 10, 07743, Jena, Thuringia, Germany
| | - Maren März
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, AG Progress Test Medizin, Charitéplatz 1, 10117, Berlin, Germany
| |
Collapse
|
|
4
|
de Chiusole D, Granziol U, Spoto A, Stefanutti L. Reliability of a probabilistic knowledge structure. Behav Res Methods 2024; 56:8022-8037. [PMID: 39060860 PMCID: PMC11362261 DOI: 10.3758/s13428-024-02468-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/10/2024] [Indexed: 07/28/2024]
Abstract
Indexes for estimating the overall reliability of a test in the framework of knowledge space theory (KST) are proposed and analyzed. First, the possibility of applying in KST the existing classical test theory (CTT) methods, based on the ratio between the true score variance and the total variance of the measure, has been explored. However, these methods are not suitable because in KST error and true score are not independent. Therefore, two new indexes based on the concepts of entropy and conditional entropy are developed. One index is used to estimate the reliability of the response pattern given the knowledge state, while the second one refers to the reliability of the estimated knowledge state of a person. Some theoretical considerations as well as simulations and an empirical example on real data are provided within a study of the behavior of these indexes under a certain number of different conditions.
Collapse
Affiliation(s)
| | - Umberto Granziol
- Department of General Psychology, University of Padua, Padua, Italy
| | - Andrea Spoto
- Department of General Psychology, University of Padua, Padua, Italy.
- Centro di Ateneo Servizi Clinici Universitari Psicologici SCUP, University of Padua, Padua, Italy.
| | | |
Collapse
|
|
5
|
de Chiusole D, Spinoso M, Anselmi P, Bacherini A, Balboni G, Mazzoni N, Brancaccio A, Epifania OM, Orsoni M, Giovagnoli S, Garofalo S, Benassi M, Robusto E, Stefanutti L, Pierluigi I. PsycAssist: A Web-Based Artificial Intelligence System Designed for Adaptive Neuropsychological Assessment and Training. Brain Sci 2024; 14:122. [PMID: 38391697 PMCID: PMC10886469 DOI: 10.3390/brainsci14020122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 01/15/2024] [Accepted: 01/19/2024] [Indexed: 02/24/2024] Open
Abstract
Assessing executive functions in individuals with disorders or clinical conditions can be challenging, as they may lack the abilities needed for conventional test formats. The use of more personalized test versions, such as adaptive assessments, might be helpful in evaluating individuals with specific needs. This paper introduces PsycAssist, a web-based artificial intelligence system designed for neuropsychological adaptive assessment and training. PsycAssist is a highly flexible and scalable system based on procedural knowledge space theory and may be used potentially with many types of tests. We present the architecture and adaptive assessment engine of PsycAssist and the two currently available tests: Adap-ToL, an adaptive version of the Tower of London-like test to assess planning skills, and MatriKS, a Raven-like test to evaluate fluid intelligence. Finally, we describe the results of an investigation of the usability of Adap-ToL and MatriKS: the evaluators perceived these tools as appropriate and well-suited for their intended purposes, and the test-takers perceived the assessment as a positive experience. To sum up, PsycAssist represents an innovative and promising tool to tailor evaluation and training to the specific characteristics of the individual, useful for clinical practice.
Collapse
Affiliation(s)
- Debora de Chiusole
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Matilde Spinoso
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Pasquale Anselmi
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Alice Bacherini
- Department of Philosophy, Social Sciences and Education, University of Perugia, Piazza G. Ermini 1, 06123 Perugia, Italy
| | - Giulia Balboni
- Department of Philosophy, Social Sciences and Education, University of Perugia, Piazza G. Ermini 1, 06123 Perugia, Italy
| | - Noemi Mazzoni
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Andrea Brancaccio
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Ottavia M Epifania
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Matteo Orsoni
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Sara Giovagnoli
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Sara Garofalo
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Mariagrazia Benassi
- Department of Psychology "Renzo Canestrari", University of Bologna, Piazza Aldo Moro 90, 47521 Cesena, Italy
| | - Egidio Robusto
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Luca Stefanutti
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua, Via Venezia 14, 35131 Padova, Italy
| | - Irene Pierluigi
- Department of Philosophy, Social Sciences and Education, University of Perugia, Piazza G. Ermini 1, 06123 Perugia, Italy
| |
Collapse
|
|
6
|
Brancaccio A, de Chiusole D, Stefanutti L. Algorithms for the adaptive assessment of procedural knowledge and skills. Behav Res Methods 2023; 55:3929-3951. [PMID: 36526887 PMCID: PMC10616228 DOI: 10.3758/s13428-022-01998-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/01/2022] [Indexed: 12/23/2022]
Abstract
Procedural knowledge space theory (PKST) was recently proposed by Stefanutti (British Journal of Mathematical and Statistical Psychology, 72(2) 185-218, 2019) for the assessment of human problem-solving skills. In PKST, the problem space formally represents how a family of problems can be solved and the knowledge space represents the skills required for solving those problems. The Markov solution process model (MSPM) by Stefanutti et al. (Journal of Mathematical Psychology, 103, 102552, 2021) provides a probabilistic framework for modeling the solution process of a task, via PKST. In this article, three adaptive procedures for the assessment of problem-solving skills are proposed that are based on the MSPM. Beside execution correctness, they also consider the sequence of moves observed in the solution of a problem with the aim of increasing efficiency and accuracy of assessments. The three procedures differ from one another in the assumption underlying the solution process, named pre-planning, interim-planning, and mixed-planning. In two simulation studies, the three adaptive procedures have been compared to one another and to the continuous Markov procedure (CMP) by Doignon and Falmagne (1988a). The last one accounts for dichotomous correct/wrong answers only. Results show that all the MSP-based adaptive procedures outperform the CMP in both accuracy and efficiency. These results have been obtained in the framework of the Tower of London test but the procedures can also be applied to all psychological and neuropsychological tests that have a problem space. Thus, the adaptive procedures presented in this paper pave the way to the adaptive assessment in the area of neuropsychological tests.
Collapse
Affiliation(s)
- Andrea Brancaccio
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Via Venezia, 14, 35131, Padova, Italy.
| | - Debora de Chiusole
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Via Venezia, 14, 35131, Padova, Italy
| | - Luca Stefanutti
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Via Venezia, 14, 35131, Padova, Italy
| |
Collapse
|
|
7
|
Spoto A, Stefanutti L. Empirical indistinguishability: From the knowledge structure to the skills. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2023; 76:312-326. [PMID: 36366819 DOI: 10.1111/bmsp.12291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 08/30/2022] [Accepted: 09/13/2022] [Indexed: 06/16/2023]
Abstract
Recent literature has pointed out that the basic local independence model (BLIM) when applied to some specific instances of knowledge structures presents identifiability issues. Furthermore, it has been shown that for such instances the model presents a stronger form of unidentifiability named empirical indistinguishability, which leads to the fact that the existence of certain knowledge states in such structures cannot be empirically tested. In this article the notion of indistinguishability is extended to skill maps and, more generally, to the competence-based knowledge space theory. Theoretical results are provided showing that skill maps can be empirically indistinguishable from one another. The most relevant consequence of this is that for some skills there is no empirical evidence to establish their existence. This result is strictly related to the type of probabilistic model investigated, which is essentially the BLIM. Alternative models may exist or can be developed in knowledge space theory for which this indistinguishability problem disappears.
Collapse
Affiliation(s)
- Andrea Spoto
- Department of General Psychology, University of Padua, Padova, Italy
| | | |
Collapse
|
|
8
|
Zhou Y, Li J, Yang H, Xu Q, Zhou Y. Knowledge structure construction and skill reduction methods based on multi-scale context. J EXP THEOR ARTIF IN 2023. [DOI: 10.1080/0952813x.2023.2183266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/05/2023]
Affiliation(s)
- Yinfeng Zhou
- School of Mathematics and Statistics, Shaanxi Normal University, Xi’an, Shaanxi, China
| | - Jinjin Li
- School of Mathematics and Statistics, Minnan Normal University, Zhangzhou, Fujian, China
- Key Laboratory of Granular Computing and Application in Fujian, Minnan Normal University, Zhangzhou, Fujian, China
| | - Hailong Yang
- School of Mathematics and Statistics, Shaanxi Normal University, Xi’an, Shaanxi, China
| | - Qingyuan Xu
- School of Computer Science, Minnan Normal University, Zhangzhou, Fujian, China
- Key Laboratory of Data Science and Intelligence Application in Fujian Province Universities, Minnan Normal University, Zhangzhou, Fujian, China
| | - Yueli Zhou
- School of Mathematics and Statistics, Minnan Normal University, Zhangzhou, Fujian, China
| |
Collapse
|
|
9
|
Xie X, Xu W, Li J. A novel concept-cognitive learning method: A perspective from competences. Knowl Based Syst 2023. [DOI: 10.1016/j.knosys.2023.110382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/12/2023]
|
|
10
|
Ünlü A. Qualitative motivation with sets and relations. Front Psychol 2023; 13:993660. [PMID: 36760895 PMCID: PMC9907087 DOI: 10.3389/fpsyg.2022.993660] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 12/12/2022] [Indexed: 01/26/2023] Open
Abstract
In self-determination theory (SDT), multiple conceptual regulations of motivation are posited. These forms of motivation are especially qualitatively viewed by SDT researchers, and there are situations in which combinations of these regulations occur. In this article, instead of the commonly used numerical approach, this is modeled more versatilely by sets and relations. We discuss discrete mathematical models from the theory of knowledge spaces for the combinatorial conceptualization of motivation. Thereby, we constructively add insight into a dispute of opinions on the unidimensionality vs. multidimensionality of motivation in SDT literature. The motivation order derived in our example, albeit doubly branched, was approximately a chain, and we could quantify the combinatorial details of that approximation. Essentially, two combinatorial dimensions reducible to one were observed, which could be studied in other more popular scales as well. This approach allows us to define the distinct, including even equally informative, gradations of any regulation type. Thus, we may identify specific forms of motivation that may otherwise be difficult to measure or not be separable empirically. This could help to dissolve possible inconsistencies that may arise in applications of the theory in distinguishing the different regulation types. How to obtain the motivation structures in practice is demonstrated by relational data mining. The technique applied is an inductive item tree analysis, an established method of Boolean analysis of questionnaires. For a data set on learning motivation, the motivation spaces and co-occurrence relations for the gradations of the basic regulation types are extracted, thus, enumerating their potential subforms. In that empirical application, the underlying models were computed within each of the intrinsic, identified, introjected, and external regulations, in autonomous and controlled motivations, and the entire motivation domain. In future studies, the approach of this article could be employed to develop adaptive assessment and training procedures in SDT contexts and for dynamical extensions of the theory, if motivational behavior can go in time.
Collapse
Affiliation(s)
- Ali Ünlü
- School of Social Sciences and Technology, Technical University of Munich, Munich, Germany
| |
Collapse
|
|
11
|
Wang Z, Qi J, Shi C, Ren R, Wei L. Multiview granular data analytics based on three-way concept analysis. APPL INTELL 2022. [DOI: 10.1007/s10489-022-04145-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
|
12
|
Heller J. Complete Q-matrices in conjunctive models on general attribute structures. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2022; 75:522-549. [PMID: 35307821 DOI: 10.1111/bmsp.12266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/11/2020] [Revised: 01/24/2022] [Indexed: 06/14/2023]
Abstract
In cognitive diagnostic assessment a property of the Q-matrix, usually referred to as completeness, warrants that the cognitive attributes underlying the observed behaviour can be uniquely assessed. Characterizations of completeness were first derived under the assumption of independent attributes, and are currently under investigation for interdependent attributes. The dominant approach considers so-called attribute hierarchies, which are conceptualized through a partial order on the set of attributes. The present paper extends previously published results on this issue obtained for conjunctive attribute hierarchy models. Drawing upon results from knowledge structure theory, it provides novel sufficient and necessary conditions for completeness of the Q-matrix, not only for conjunctive models on attribute hierarchies, but also on more general attribute structures.
Collapse
Affiliation(s)
- Jürgen Heller
- Department of Psychology, University of Tübingen, Germany
| |
Collapse
|
|
13
|
Three Algorithms for Grouping Students: A Bridge Between Personalized Tutoring System Data and Classroom Pedagogy. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2022. [DOI: 10.1007/s40593-022-00309-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
Abstract
AbstractComputer-assisted instructional programs such as intelligent tutoring systems are often used to support blended learning practices in K-12 education, as they aim to meet individual student needs with personalized instruction. While these systems have been shown to be effective under certain conditions, they can be difficult to integrate into pedagogical practices. In this paper, we introduce three group formation algorithms that leverage learning data from the adaptive intelligent tutoring system ALEKS to support pedagogical and collaborative learning practices with ALEKS. Each grouping method was devised for different use cases, but they all utilize a fine-grained multidimensional view of student ability measured across several hundred skills in an academic course. As such, the grouping algorithms not only identify groups of students, but they also determine what areas of ALEKS content each group should focus on. We then evaluate each of the three methods against two alternative baseline methods, which were chosen for their plausibility of being used in practice—one that groups students randomly and one that groups students based on a unidimensional course score. To evaluate these methods, we establish a set of practical metrics based on what we anticipate teachers would care about in practice. Evaluations were performed by simulating mock groupings of students at different time periods for real ALEKS algebra classes that occurred between 2017 and 2019. We show that each devised method obtains more favorable results on the specified metrics than the alternative methods under each use-case. Moreover, we highlight examples where our methods lead to more nuanced groupings than grouping based on a unidimensional measure of ability.
Collapse
|
|
14
|
Minimal Generators from Positive and Negative Attributes: Analysing the Knowledge Space of a Mathematics Course. INT J COMPUT INT SYS 2022. [DOI: 10.1007/s44196-022-00123-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022] Open
Abstract
AbstractFormal concept analysis is a data analysis framework based on lattice theory. In this paper, we analyse the use, inside this framework, of positive and negative (mixed) attributes of a dataset, which has proved to represent more information on the use of just positive attributes. From a theoretical point of view, in this paper we show the structure and the relationships between minimal generators of the simple and mixed concept lattices. From a practical point of view, the obtained theoretical results allow us to ensure a greater granularity in the retrieved information. Furthermore, due to the relationship between FCA and Knowledge Space theory, on a practical level, we analyse the marks of a Mathematics course to establish the knowledge structure of the course and determine the key items providing new relevant information that is not evident without the use of the proposed tools.
Collapse
|
|
15
|
Wu R, Lin Y. SOLO taxonomy-based knowledge structure with subjective items. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2022. [DOI: 10.3233/jifs-212176] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
As an important mathematical theory in intelligent learning and assessment system, knowledge space theory merely cares about items are mastered or non-mastered. Thus it needs to be further explored to achieve more precise and interpretable analysis. To this end, this paper mainly focuses on knowledge structures in corporate with Solo taxonomy. Then, fuzzy knowledge structure and fuzzy learning space are gradually developed. The corresponding knowledge base and surmise relation are explored respectively as well. In such case, the induced maximal knowledge space and its properties are further studied sufficiently. And three kinds of skill models are put forward based on skill proficiency. Finally, a case study is presented to illustrate the advantage in learning description.
Collapse
Affiliation(s)
- Rong Wu
- School of Mathematical Sciences, Huaqiao University, Quanzhou, Fujian, P. R. China
| | - Yidong Lin
- School of Mathematics and Statistics, Minnan Normal University, Zhangzhou, P. R. China
- Fujian Key Laboratory of Granular Computing and Application, Minnan Normal University, Zhangzhou, P. R. China
| |
Collapse
|
|
16
|
Matayoshi J. Approximately counting and sampling knowledge states. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2022; 75:293-318. [PMID: 34741466 DOI: 10.1111/bmsp.12257] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 09/28/2021] [Indexed: 06/13/2023]
Abstract
Approximately counting and sampling knowledge states from a knowledge space is a problem that is of interest for both applied and theoretical reasons. However, many knowledge spaces used in practice are far too large for standard statistical counting and estimation techniques to be useful. Thus, in this work we use an alternative technique for counting and sampling knowledge states from a knowledge space. This technique is based on a procedure variously known as subset simulation, the Holmes-Diaconis-Ross method, or multilevel splitting. We make extensive use of Markov chain Monte Carlo methods and, in particular, Gibbs sampling, and we analyse and test the accuracy of our results in numerical experiments.
Collapse
|
|
17
|
Competence-Based Skill Functions and Minimal Sets of Skills. Symmetry (Basel) 2022. [DOI: 10.3390/sym14050884] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Abstract
As we know, there is some relationship, such as precedence relation, among skills. Each precedence relation induces a competence structure. Thus, we study competence-based skill functions, which rely on competence structures and go from somethings observable to somethings invisible. Conversely, competence-based problem functions go from somethings invisible to somethings observable. In fact, these two dual types of functions based on competence structures are symmetry. Remarkably, there are two kinds of special competence-based skill functions: one is disjunctive, while the other is conjunctive. The former delineates knowledge spaces, which are symmetrical to simple closure spaces delineated by the latter. Based on these facts, we shows some theoretical results on competence-based skill functions, then design the corresponding algorithms for delineating knowledge structures. Sometimes for competence-based skill functions, some skills are maybe reducible. Thus, we discuss what kind of skills are reducible and obtain sufficient and some necessary conditions for skills being reducible for competence-based skill functions. Based on this, we design algorithms to reduce reducible skills and get minimal sets of skills. By comparison, for competence-based skill functions, we can find minimal sets of skills with the smallest cardinality whenever sets of skills are finite. For each algorithm, we take a corresponding example to illustrate the detailed procedure.
Collapse
|
|
18
|
A New Approach of Knowledge Reduction in Knowledge Context Based on Boolean Matrix. Symmetry (Basel) 2022. [DOI: 10.3390/sym14050850] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Abstract
Knowledge space theory (KST) is a mathematical framework for the assessment of knowledge and learning in education. An important task of KST is to achieve all of the atoms. With the development of KST, considering its relationship with formal concept analysis (FCA) has become a hot issue. The atoms of the knowledge space with application in knowledge reduction based on FCA is examined in this paper. The knowledge space and its properties based on FCA are first discussed. Knowledge reduction and its relationship with molecules in the knowledge context are then investigated. A Boolean matrix is employed to determine molecules and meet-irreducible elements in the knowledge context. The method of the knowledge-reduction-based Boolean matrix in the knowledge space is also explored. Furthermore, an algorithm for finding the atoms of the knowledge space in the knowledge context is developed using a Boolean matrix.
Collapse
|
|
19
|
Zhou Y, Li J, Wang H, Sun W. Skills and fuzzy knowledge structures. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2022. [DOI: 10.3233/jifs-212018] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
In knowledge space theory (KST), knowledge structure is an effective feature to evaluate individuals’ knowledge and guide future learning. How to construct knowledge structures is one of the key research problems in KST. At present, the knowledge structure has been generalized to the polytomous knowledge structure. This article mainly focuses on the special polytomous knowledge structures delineated by skills, which are called fuzzy knowledge structures. We consider how to construct fuzzy knowledge structures based on the relationship between items and skills, and how to find the learning paths for specific knowledge domains. First, we construct knowledge structures in four models, which are the conjunctive model of skill maps, the disjunctive and conjunctive models of fuzzy skill maps, and the competency model of fuzzy skill multimaps. Second, we assess individuals’ skills and find the learning paths for the specific knowledge domains in the first three models. Finding the learning paths for a specific knowledge domain can guide learning and improve the learning efficiency of individuals. Finally, we analyze some data sets to show that the algorithms proposed are effective and applicable. These works can be applied to adaptive learning systems, which bring great convenience for assessing individuals’ knowledge and guiding future learning.
Collapse
Affiliation(s)
- Yinfeng Zhou
- School of Mathematics and Statistics, Minnan Normal University, Zhangzhou, China
| | - Jinjin Li
- School of Mathematics and Statistics, Minnan Normal University, Zhangzhou, China
- Key Laboratory of Granular Computing and Application in Fujian, Minnan Normal University, Zhangzhou, China
| | | | - Wen Sun
- School of Science, Shantou University, Shantou, China
| |
Collapse
|
|
20
|
Böckle M, Bick M, Novak J. Toward a Design Theory of User-Centered Score Mechanics for Gamified Competency Development. INFORMATION SYSTEMS MANAGEMENT 2021. [DOI: 10.1080/10580530.2021.1975852] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Martin Böckle
- Department of Business Studies/IACS-Institute for Applied Computer Science, University of Applied Sciences Stralsund and Chair of Business Information Systems - ESCP Business School, Berlin, Germany
| | - Markus Bick
- Chair of Business Information Systems, ESCP Business School, Berlin, Germany
| | - Jasminko Novak
- Department of Business Studies/IACS-Institute for Applied Computer Science, University of Applied Sciences Stralsund and European Institute for Participatory Media, Berlin, Germany
| |
Collapse
|
|
21
|
Pedota M, Piscitello L. A new perspective on technology‐driven creativity enhancement in the Fourth Industrial Revolution. CREATIVITY AND INNOVATION MANAGEMENT 2021. [DOI: 10.1111/caim.12468] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Affiliation(s)
- Mattia Pedota
- Department of Management, Economics, and Industrial Engineering Politecnico di Milano Milan Italy
| | - Lucia Piscitello
- Department of Management, Economics, and Industrial Engineering Politecnico di Milano Milan Italy
| |
Collapse
|
|
22
|
Stefanutti L, Spoto A. On the empirical indistinguishability of knowledge structures. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2021; 74:465-486. [PMID: 33782939 DOI: 10.1111/bmsp.12235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/22/2020] [Revised: 01/07/2021] [Indexed: 06/12/2023]
Abstract
In recent years a number of articles have focused on the identifiability of the basic local independence model. The identifiability issue usually concerns two model parameter sets predicting an identical probability distribution on the response patterns. Both parameter sets are applied to the same knowledge structure. However, nothing is known about cases where different knowledge structures predict the same probability distribution. This situation is referred to as 'empirical indistinguishability' between two structures and is the main subject of the present paper. Empirical indistinguishability is a stronger form of unidentifiability, which involves not only the parameters, but also the structural and combinatorial properties of the model. In particular, as far as knowledge structures are concerned, a consequence of empirical indistinguishability is that the existence of certain knowledge states cannot be empirically established. Most importantly, it is shown that model identifiability cannot guarantee that a certain knowledge structure is empirically distinguishable from others. The theoretical findings are exemplified in a number of different empirical scenarios.
Collapse
Affiliation(s)
| | - Andrea Spoto
- Department of General Psychology, University of Padua, Italy
| |
Collapse
|
|
23
|
|
|
24
|
Ge X, Lin S. Notes on attribution functions. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2021; 74 Suppl 1:131-156. [PMID: 33314081 DOI: 10.1111/bmsp.12220] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Revised: 08/01/2020] [Indexed: 06/12/2023]
Abstract
Let Q,Kσ be the knowledge space derived from an attribution function σ on Q. Under an assumption for σ, this paper gives some necessary and sufficient conditions such that Q,Kσ is discriminative. It also discusses the resolubility of σ when Q is an infinite set. More precisely, this paper proves that σ is not resoluble if Q is uncountable, and gives a necessary and sufficient condition such that σ is resoluble when Q,Kσ is ∞ -well-graded. By way of applications of these results, discriminativeness and resolubility are discussed around the merge of skill multimaps and the meshing of the delineated knowledge spaces.
Collapse
Affiliation(s)
- Xun Ge
- School of Mathematical Sciences, Soochow University, Suzhou, China
| | - Shou Lin
- Department of Mathematics, Ningde Normal University, Fujian, China
| |
Collapse
|
|
25
|
Are We There Yet? Evaluating the Effectiveness of a Recurrent Neural Network-Based Stopping Algorithm for an Adaptive Assessment. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2021. [DOI: 10.1007/s40593-021-00240-8] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
|
26
|
Abstract
In practical applications of knowledge space theory, knowledge states can be conceived as partially ordered clusters of individuals. Existing extensions of the theory to polytomous data lack methods for building "polytomous" structures. To this aim, an adaptation of the k-median clustering algorithm is proposed. It is an extension of k-modes to ordinal data in which the Hamming distance is replaced by the Manhattan distance, and the central tendency measure is the median, rather than the mode. The algorithm is tested in a series of simulation studies and in an application to empirical data. Results show that there are theoretical and practical reasons for preferring the k-median to the k-modes algorithm, whenever the responses to the items are measured on an ordinal scale. This is because the Manhattan distance is sensitive to the order on the levels, while the Hamming distance is not. Overall, k-median seems to be a promising data-driven procedure for building polytomous structures.
Collapse
|
|
27
|
Shan N, Wang X. Cognitive Diagnosis Modeling Incorporating Item-Level Missing Data Mechanism. Front Psychol 2020; 11:564707. [PMID: 33329195 PMCID: PMC7733994 DOI: 10.3389/fpsyg.2020.564707] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2020] [Accepted: 10/28/2020] [Indexed: 11/30/2022] Open
Abstract
The aim of cognitive diagnosis is to classify respondents' mastery status of latent attributes from their responses on multiple items. Since respondents may answer some but not all items, item-level missing data often occur. Even if the primary interest is to provide diagnostic classification of respondents, misspecification of missing data mechanism may lead to biased conclusions. This paper proposes a joint cognitive diagnosis modeling of item responses and item-level missing data mechanism. A Bayesian Markov chain Monte Carlo (MCMC) method is developed for model parameter estimation. Our simulation studies examine the parameter recovery under different missing data mechanisms. The parameters could be recovered well with correct use of missing data mechanism for model fit, and missing that is not at random is less sensitive to incorrect use. The Program for International Student Assessment (PISA) 2015 computer-based mathematics data are applied to demonstrate the practical value of the proposed method.
Collapse
Affiliation(s)
- Na Shan
- School of Psychology, Northeast Normal University, Changchun, China
- Key Laboratory of Applied Statistics of the Ministry of Education, Northeast Normal University, Changchun, China
| | - Xiaofei Wang
- Key Laboratory of Applied Statistics of the Ministry of Education, Northeast Normal University, Changchun, China
- School of Mathematics and Statistics, Northeast Normal University, Changchun, China
| |
Collapse
|
|
28
|
de Chiusole D, Stefanutti L, Anselmi P, Robusto E. Stat-Knowlab. Assessment and Learning of Statistics with Competence-based Knowledge Space Theory. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2020. [DOI: 10.1007/s40593-020-00223-1] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
|
29
|
Kirby K, Anwar MN. An application of activity theory to the "problem of e-books". Heliyon 2020; 6:e04982. [PMID: 32995643 PMCID: PMC7505808 DOI: 10.1016/j.heliyon.2020.e04982] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2020] [Revised: 07/24/2020] [Accepted: 09/16/2020] [Indexed: 02/06/2023] Open
Abstract
The “problem of e-books” is defined as the difficulty improving the adoption rates of e-books by students. The adoption rates of e-books for academic use remain low, and research into the reasons for this have resulted in inconclusive findings. Factors such as student perception, and variations in experimental methodology and technology, contribute to difficulties in generalising findings and establishing conclusive causes for this problem. To better understand the causal factors for low adoption rates and the student's experience with ereaders and digital text, an investigation was conducted by the lead researcher as a student enrolled in a postgraduate course. The experiment was designed using e-book and digital text documents on an ereader for academic study and the results analysed with the framework of Activity Theory. This methodology allowed exploration of the problem within the authentic experience of a student to examine the effects of this social environment on ereader and e-book use. Analysis of the work domain was conducted and a comparative assessment of the observed effect of using the digital documents on an ereader compared with the paper book. Findings show that attempts to apply self-regulation and metacognitive learning techniques within the activity using the ereader were abandoned due to breakdowns in operations, and that this resulted in a perceived lower quality of achievement. The effect on the processes used by the student were extreme and were observed to be highly dependent on the student's use of specific learning strategies. The experimental methodology employed in this investigation enabled identification of the role of the social environment in the use of course documents on an ereader for academic study. The functionality of the ereader was such an extremely poor fit with the observed academic processes that a redesign approach for ereader and e-book technology is proposed as a solution to the low adoption rates of e-books.
Collapse
Affiliation(s)
- Karen Kirby
- Department of Computer and Information Sciences, Northumbria University, UK
| | - Muhammad N Anwar
- Department of Computer and Information Sciences, Northumbria University, UK
| |
Collapse
|
|
30
|
Stefanutti L, de Chiusole D, Anselmi P, Spoto A. Extending the Basic Local Independence Model to Polytomous Data. PSYCHOMETRIKA 2020; 85:684-715. [PMID: 32959202 PMCID: PMC7599199 DOI: 10.1007/s11336-020-09722-5] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2019] [Revised: 06/19/2020] [Indexed: 05/28/2023]
Abstract
A probabilistic framework for the polytomous extension of knowledge space theory (KST) is proposed. It consists in a probabilistic model, called polytomous local independence model, that is developed as a generalization of the basic local independence model. The algorithms for computing "maximum likelihood" (ML) and "minimum discrepancy" (MD) estimates of the model parameters have been derived and tested in a simulation study. Results show that the algorithms differ in their capability of recovering the true parameter values. The ML algorithm correctly recovers the true values, regardless of the manipulated variables. This is not totally true for the MD algorithm. Finally, the model has been applied to a real polytomous data set collected in the area of psychological assessment. Results show that it can be successfully applied in practice, paving the way to a number of applications of KST outside the area of knowledge and learning assessment.
Collapse
Affiliation(s)
- Luca Stefanutti
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Padua, Italy
| | - Debora de Chiusole
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Padua, Italy
| | - Pasquale Anselmi
- Department of Philosophy, Sociology, Pedagogy, and Applied Psychology, University of Padua, Padua, Italy
| | - Andrea Spoto
- Department of General Psychology, Via Venezia, 8, 35131, Padua, Italy.
| |
Collapse
|
|
31
|
Abbakumov D, Desmet P, Van den Noortgate W. Psychometrics of MOOCs: Measuring Learners' Proficiency. Psychol Belg 2020; 60:115-131. [PMID: 32477583 PMCID: PMC7243831 DOI: 10.5334/pb.515] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2019] [Accepted: 04/23/2020] [Indexed: 12/03/2022] Open
Abstract
Massive open online courses (MOOCs) generate learners' performance data that can be used to understand learners' proficiency and to improve their efficiency. However, the approaches currently used, such as assessing the proportion of correct responses in assessments, are oversimplified and may lead to poor conclusions and decisions because they do not account for additional information on learner, content, and context. There is a need for theoretically grounded data-driven explainable educational measurement approaches for MOOCs. In this conceptual paper, we try to establish a connection between psychometrics, a scientific discipline concerned with techniques for educational and psychological measurement, and MOOCs. First, we describe general principles of traditional measurement of learners' proficiency in education. Second, we discuss qualities of MOOCs which hamper direct application of approaches based on these general principles. Third, we discuss recent developments in measuring proficiency that may be relevant for analyzing MOOC data. Finally, we draw directions in psychometric modeling that might be interesting for future MOOC research.
Collapse
Affiliation(s)
- Dmitry Abbakumov
- ITEC, IMEC research group at KU Leuven, Kortrijk, BE
- Faculty of Psychology and Educational Sciences, KU Leuven, Leuven, BE
- eLearning Office, National Research University Higher School of Economics, Moscow, RU
| | - Piet Desmet
- ITEC, IMEC research group at KU Leuven, Kortrijk, BE
- Faculty of Arts, KU Leuven, Leuven, BE
| | - Wim Van den Noortgate
- ITEC, IMEC research group at KU Leuven, Kortrijk, BE
- Faculty of Psychology and Educational Sciences, KU Leuven, Leuven, BE
| |
Collapse
|
|
32
|
Kapp KM, Dore S, Fiene R. COGNITIVE MAPPING FOR ILOOKOUT FOR CHILD ABUSE: AN ONLINE TRAINING PROGRAM FOR EARLY CHILDHOOD PROFESSIONALS. THE ONLINE JOURNAL OF DISTANCE EDUCATION AND E-LEARNING : TOJDEL 2020; 8:80-89. [PMID: 32974058 PMCID: PMC7511090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
This article delineates the theory and framework for an innovative child abuse training program for mandated reporters called 'iLookOut'. iLookOut is an online learning delivery system that utilizes mastery learning and self-determination theory in the Core Training program, along with spaced retrieval and retrieval practice in a follow-up micro-learning program that reinforces learning from the Core Training. A cognitive mapping model provides the structure for documenting and organizing the learning content in both the Core training and the follow-up micro-learning program. The article provides a conceptual framework for designing and implementing effective and efficient online learning programs.
Collapse
Affiliation(s)
| | - Sarah Dore
- iLookOut for Child Abuse| Learning & Development Director, Department of Humanities, Penn State College of Medicine
| | - Richard Fiene
- Edna Bennett Pierce Prevention Research Center, Professor of HDFS & Psychology (retired), The Pennsylvania State University
| |
Collapse
|
|
33
|
Ünlü A, Schrepp M. Location-Scale Matching for Approximate Quasi-Order Sampling. Front Psychol 2019; 10:1163. [PMID: 31244703 PMCID: PMC6573793 DOI: 10.3389/fpsyg.2019.01163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2019] [Accepted: 05/02/2019] [Indexed: 12/03/2022] Open
Abstract
Quasi-orders are reflexive and transitive binary relations and have many applications. Examples are the dependencies of mastery among the problems of a psychological test, or methods such as item tree or Boolean analysis that mine for quasi-orders in empirical data. Data mining techniques are typically tested based on simulation studies with unbiased samples of randomly generated quasi-orders. In this paper, we develop techniques for the approximately representative sampling of quasi-orders. Polynomial regression curves are fitted for the mean and standard deviation of quasi-order size as a function of item number. The resulting regression graphs are seen to be quadratic and linear functions, respectively. The extrapolated values for the mean and standard deviation are used to propose two quasi-order sampling techniques. The discrete method matches these location and scale measures with a transformed discrete distribution directly obtained from the sample. The continuous method uses the normal density function with matched expectation and variance. The quasi-orders are constructed according to the biased randomized doubly inductive construction, however they are resampled to become approximately representative following the matched discrete and continuous distributions. In simulations, we investigate the usefulness of these methods. The location-scale matching approach can cope with very large item sets. Close to representative samples of random quasi-orders are constructed for item numbers up to n = 400.
Collapse
Affiliation(s)
- Ali Ünlü
- TUM School of Education, Technical University of Munich (TUM), Munich, Germany
| | | |
Collapse
|
|
34
|
Noventa S, Spoto A, Heller J, Kelava A. On a Generalization of Local Independence in Item Response Theory Based on Knowledge Space Theory. PSYCHOMETRIKA 2019; 84:395-421. [PMID: 30421075 DOI: 10.1007/s11336-018-9645-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/18/2017] [Revised: 10/10/2018] [Indexed: 06/09/2023]
Abstract
Knowledge space theory (KST) structures are introduced within item response theory (IRT) as a possible way to model local dependence between items. The aim of this paper is threefold: firstly, to generalize the usual characterization of local independence without introducing new parameters; secondly, to merge the information provided by the IRT and KST perspectives; and thirdly, to contribute to the literature that bridges continuous and discrete theories of assessment. In detail, connections are established between the KST simple learning model (SLM) and the IRT General Graded Response Model, and between the KST Basic Local Independence Model and IRT models in general. As a consequence, local independence is generalized to account for the existence of prerequisite relations between the items, IRT models become a subset of KST models, IRT likelihood functions can be generalized to broader families, and the issues of local dependence and dimensionality are partially disentangled. Models are discussed for both dichotomous and polytomous items and conclusions are drawn on their interpretation. Considerations on possible consequences in terms of model identifiability and estimation procedures are also provided.
Collapse
Affiliation(s)
| | - Andrea Spoto
- Department of General Psychology, University of Padova, Padua, Italy
| | - Jürgen Heller
- Faculty of Psychology, Universität Tübingen, Tübingen, Germany
| | | |
Collapse
|
|
35
|
Stefanutti L. On the assessment of procedural knowledge: From problem spaces to knowledge spaces. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2019; 72:185-218. [PMID: 30035297 DOI: 10.1111/bmsp.12139] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/27/2017] [Revised: 04/17/2018] [Indexed: 06/08/2023]
Abstract
By generalizing and completing the work initiated by Stefanutti and Albert (2003, Journal of Universal Computer Science, 9, 1455), this article provides the mathematical foundations of a theoretical approach whose primary goal is to construct a bridge between problem solving, as initially conceived by Newell and Simon (1972, Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.), and knowledge assessment (Doignon and Falmagne, 1985, International Journal of Man-Machine Studies, 23, 175; Doignon and Falmagne, 1999, Knowledge spaces. Berlin, Germany: Springer-Verlag.; Falmagne et al., 2013, Knowledge spaces: Applications in education. New York, NY: Springer-Verlag; Falmagne and Doignon, 2011, Learning spaces: Interdisciplinary applied mathematics. Berlin, Germany: Springer-Verlag.). It is shown that the collection of all possible knowledge states for a given problem space is a learning space. An algorithm for deriving a learning space from a problem space is illustrated. As an example, the algorithm is used to derive the learning space of a neuropsychological test whose problem space is well known: the Tower of London (TOL; Shallice, 1982, Philosophical Transactions of the Royal Society of London B: Biological Sciences, 298, 199). The derived learning space could then be used for adaptively assessing individual planning skills with the TOL.
Collapse
|
|
36
|
A Data-Based Simulation Study of Reliability for an Adaptive Assessment Based on Knowledge Space Theory. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2019. [DOI: 10.1007/s40593-019-00176-0] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
|
37
|
Spoto A, Serra F, Donadello I, Granziol U, Vidotto G. New Perspectives in the Adaptive Assessment of Depression: The ATS-PD Version of the QuEDS. Front Psychol 2018; 9:1101. [PMID: 30034352 PMCID: PMC6043690 DOI: 10.3389/fpsyg.2018.01101] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2018] [Accepted: 06/11/2018] [Indexed: 11/18/2022] Open
Abstract
Measurement is a crucial issue in psychological assessment. In this paper a contribution to this task is provided by means of the implementation of an adaptive algorithm for the assessment of depression. More specifically, the Adaptive Testing System for Psychological Disorders (ATS-PD) version of the Qualitative-Quantitative Evaluation of Depressive Symptomatology questionnaire (QuEDS) is introduced. Such implementation refers to the theoretical background of Formal Psychological Assessment (FPA) with respect to both its deterministic and probabilistic issues. Three models (one for each sub-scale of the QuEDS) are fitted on a sample of 383 individuals. The obtained estimates are then used to calibrate the adaptive procedure whose performance is tested in terms of both efficiency and accuracy by means of a simulation study. Results indicate that the ATS-PD version of the QuEDS allows for both obtaining an accurate description of the patient in terms of symptomatology, and reducing the number of items asked by 40%. Further developments of the adaptive procedure are then discussed.
Collapse
Affiliation(s)
- Andrea Spoto
- Quantitative Psychology Laboratory, Department of General Psychology, University of Padua, Padua, Italy
| | - Francesca Serra
- Quantitative Psychology Laboratory, Department of General Psychology, University of Padua, Padua, Italy
| | | | - Umberto Granziol
- Quantitative Psychology Laboratory, Department of General Psychology, University of Padua, Padua, Italy
| | - Giulio Vidotto
- Quantitative Psychology Laboratory, Department of General Psychology, University of Padua, Padua, Italy
| |
Collapse
|
|
38
|
Granziol U, Spoto A, Vidotto G. The assessment of nonverbal behavior in schizophrenia through the Formal Psychological Assessment. Int J Methods Psychiatr Res 2018; 27:e1595. [PMID: 29098743 PMCID: PMC6877208 DOI: 10.1002/mpr.1595] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/02/2017] [Revised: 09/27/2017] [Accepted: 09/29/2017] [Indexed: 12/16/2022] Open
Abstract
The nonverbal behavior (NVB) of people diagnosed with schizophrenia consistently interacts with their symptoms during the assessment. Previous studies frequently observed such an interaction when a prevalence of negative symptoms occurred. Nonetheless, a list of NVBs linked to negative symptoms needs to be defined. Furthermore, a list of items that can exhaustively assess such NVBs is still needed. The present study aims to introduce both lists by using the Formal Psychological Assessment. A deep analysis was performed on both the scientific literature and the DSM-5 for constructing the set of nonverbal behaviors; similarly, an initial list of 138 items investigating the behaviors was obtained from instruments used to assess schizophrenia. The Formal Psychological Assessment was then applied to reduce the preliminary list. A final list of 23 items necessary and sufficient to investigate the NVBs emerged. The list also allowed us to analyze specific relations among items. The present study shows how it is possible to deepen a patient's negative symptomatology, starting with the relations between items and the NVBs they investigate. Finally, this study examines the advantages and clinical implications of defining an assessment tool based on the found list of items.
Collapse
Affiliation(s)
- Umberto Granziol
- Department of General Psychology, University of Padua, Padova, Italy
| | - Andrea Spoto
- Department of General Psychology, University of Padua, Padova, Italy
| | - Giulio Vidotto
- Department of General Psychology, University of Padua, Padova, Italy
| |
Collapse
|
|
39
|
Abstract
If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.
Collapse
|
|
40
|
Anselmi P, Stefanutti L, de Chiusole D, Robusto E. The assessment of knowledge and learning in competence spaces: The gain-loss model for dependent skills. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2017; 70:457-479. [PMID: 28211048 DOI: 10.1111/bmsp.12095] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/07/2016] [Revised: 12/15/2016] [Indexed: 06/06/2023]
Abstract
The gain-loss model (GaLoM) is a formal model for assessing knowledge and learning. In its original formulation, the GaLoM assumes independence among the skills. Such an assumption is not reasonable in several domains, in which some preliminary knowledge is the foundation for other knowledge. This paper presents an extension of the GaLoM to the case in which the skills are not independent, and the dependence relation among them is described by a well-graded competence space. The probability of mastering skill s at the pretest is conditional on the presence of all skills on which s depends. The probabilities of gaining or losing skill s when moving from pretest to posttest are conditional on the mastery of s at the pretest, and on the presence at the posttest of all skills on which s depends. Two formulations of the model are presented, in which the learning path is allowed to change from pretest to posttest or not. A simulation study shows that models based on the true competence space obtain a better fit than models based on false competence spaces, and are also characterized by a higher assessment accuracy. An empirical application shows that models based on pedagogically sound assumptions about the dependencies among the skills obtain a better fit than models assuming independence among the skills.
Collapse
|
|
41
|
Serra F, Spoto A, Ghisi M, Vidotto G. Improving Major Depressive Episode Assessment: A New Tool Developed by Formal Psychological Assessment. Front Psychol 2017; 8:214. [PMID: 28261143 PMCID: PMC5313503 DOI: 10.3389/fpsyg.2017.00214] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2016] [Accepted: 02/03/2017] [Indexed: 12/13/2022] Open
Abstract
Aim: Major depressive episode (MDE) can manifest with different features. Discriminating between different types of MDEs is crucial for proper treatment. The aim of this study is to propose a new tool for MDE assessment in bipolar disorder (BD) or major depressive disorder (MDD) to overcome some limitations of current rating scales. The proposed tool investigates all of the clinical features of different MDEs and gives qualitative information, differentiating patients with the same score but different symptoms and psychopathology severity. To achieve this purpose authors used a new methodology called Formal Psychological Assessment (FPA). FPA allows creating relations between the items of an assessment tool, and the set of diagnostic criteria of a given clinical disorder. In the application at hand, given the capability to analyze all clinical features, FPA appears a useful way to highlight and differentiate between inhibited and agitated depressive symptoms. Method: The new tool contains 41 items constructed through 23 clinical criteria from the DSM-5 and literature symptoms. In line with FPA, starting from a set of items and a set of clinical criteria, a Boolean matrix was built assigning to each item its own set of clinical criteria. The participants include 265 in the control group and 38 patients with MDE (diagnosed with MDD or BD) who answered the QuEDS. After 1 month, 63 participants performed the test again and 113 took the Depression-Anxiety-Stress Scale to analyze convergent-divergent validity. Results: The scale showed adequate reliability and validity. A hierarchical confirmatory factor analysis highlighted the presence of three sub factors (affective, somatic, and cognitive) and one high-order factor (depression). Conclusions: The new tool is potentially able to inform clinicians about the patients' most likely diagnostic configuration. Indeed, the clinical state of a patient consists of the subset of items he/she answered affirmatively, along with his/her subset of specific symptoms. Qualitative information is fundamental from a clinical perspective, allowing for the analysis and treatment of each patient according to his/her symptoms in an effective way.
Collapse
Affiliation(s)
- Francesca Serra
- Department of General Psychology, University of Padua Padova, Italy
| | - Andrea Spoto
- Department of General Psychology, University of Padua Padova, Italy
| | - Marta Ghisi
- Department of General Psychology, University of Padua Padova, Italy
| | - Giulio Vidotto
- Department of General Psychology, University of Padua Padova, Italy
| |
Collapse
|
|
42
|
Ünlü A, Schrepp M. Toward a Principled Sampling Theory for Quasi-Orders. Front Psychol 2016; 7:1656. [PMID: 27965601 PMCID: PMC5126093 DOI: 10.3389/fpsyg.2016.01656] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2016] [Accepted: 10/10/2016] [Indexed: 11/13/2022] Open
Abstract
Quasi-orders, that is, reflexive and transitive binary relations, have numerous applications. In educational theories, the dependencies of mastery among the problems of a test can be modeled by quasi-orders. Methods such as item tree or Boolean analysis that mine for quasi-orders in empirical data are sensitive to the underlying quasi-order structure. These data mining techniques have to be compared based on extensive simulation studies, with unbiased samples of randomly generated quasi-orders at their basis. In this paper, we develop techniques that can provide the required quasi-order samples. We introduce a discrete doubly inductive procedure for incrementally constructing the set of all quasi-orders on a finite item set. A randomization of this deterministic procedure allows us to generate representative samples of random quasi-orders. With an outer level inductive algorithm, we consider the uniform random extensions of the trace quasi-orders to higher dimension. This is combined with an inner level inductive algorithm to correct the extensions that violate the transitivity property. The inner level correction step entails sampling biases. We propose three algorithms for bias correction and investigate them in simulation. It is evident that, on even up to 50 items, the new algorithms create close to representative quasi-order samples within acceptable computing time. Hence, the principled approach is a significant improvement to existing methods that are used to draw quasi-orders uniformly at random but cannot cope with reasonably large item sets.
Collapse
Affiliation(s)
- Ali Ünlü
- Centre for International Student Assessment, Technical University of MunichMunich, Germany
- *Correspondence: Ali Ünlü
| | | |
Collapse
|
|
43
|
A class of k-modes algorithms for extracting knowledge structures from data. Behav Res Methods 2016; 49:1212-1226. [DOI: 10.3758/s13428-016-0780-7] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
|
44
|
Anselmi P, Robusto E, Stefanutti L, de Chiusole D. An Upgrading Procedure for Adaptive Assessment of Knowledge. PSYCHOMETRIKA 2016; 81:461-482. [PMID: 27071952 DOI: 10.1007/s11336-016-9498-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/11/2014] [Indexed: 06/05/2023]
Abstract
In knowledge space theory, existing adaptive assessment procedures can only be applied when suitable estimates of their parameters are available. In this paper, an iterative procedure is proposed, which upgrades its parameters with the increasing number of assessments. The first assessments are run using parameter values that favor accuracy over efficiency. Subsequent assessments are run using new parameter values estimated on the incomplete response patterns from previous assessments. Parameter estimation is carried out through a new probabilistic model for missing-at-random data. Two simulation studies show that, with the increasing number of assessments, the performance of the proposed procedure approaches that of gold standards.
Collapse
Affiliation(s)
- Pasquale Anselmi
- Department FISPPA, University of Padua, Via Venezia 8, 35131 , Padua, Italy.
| | - Egidio Robusto
- Department FISPPA, University of Padua, Via Venezia 8, 35131 , Padua, Italy
| | - Luca Stefanutti
- Department FISPPA, University of Padua, Via Venezia 8, 35131 , Padua, Italy
| | - Debora de Chiusole
- Department FISPPA, University of Padua, Via Venezia 8, 35131 , Padua, Italy
| |
Collapse
|
|
45
|
Hung JC, Weng JD, Chen YH. A recommendation system based on mining human portfolio for museum navigation. EVOLVING SYSTEMS 2016. [DOI: 10.1007/s12530-016-9154-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
|
46
|
Schrepp M, Ünlü A. On the Creation of Representative Samples of Random Quasi-Orders. Front Psychol 2015; 6:1791. [PMID: 26640450 PMCID: PMC4661281 DOI: 10.3389/fpsyg.2015.01791] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2015] [Accepted: 11/06/2015] [Indexed: 11/13/2022] Open
Abstract
Dependencies between educational test items can be represented as quasi-orders on the item set of a knowledge domain and used for an efficient adaptive assessment of knowledge. One approach to uncovering such dependencies is by exploratory algorithms of item tree analysis (ITA). There are several methods of ITA available. The basic tool to compare such algorithms concerning their quality are large-scale simulation studies that are crucially set up on a large collection of quasi-orders. A serious problem is that all known ITA algorithms are sensitive to the structure of the underlying quasi-order. Thus, it is crucial to base any simulation study that tries to compare the algorithms upon samples of quasi-orders that are representative, meaning each quasi-order is included in a sample with the same probability. Up to now, no method to create representative quasi-orders on larger item sets is known. Non-optimal algorithms for quasi-order generation were used in previous studies, which caused misinterpretations and erroneous conclusions. In this paper, we present a method for creating representative random samples of quasi-orders. The basic idea is to consider random extensions of quasi-orders from lower to higher dimension and to discard extensions that do not satisfy the transitivity property.
Collapse
Affiliation(s)
| | - Ali Ünlü
- Technical University of Munich (TUM), Centre for International Student Assessment (ZIB) Munich, Germany
| |
Collapse
|
|
47
|
Heller J, Stefanutti L, Anselmi P, Robusto E. On the Link between Cognitive Diagnostic Models and Knowledge Space Theory. PSYCHOMETRIKA 2015; 80:995-1019. [PMID: 25838246 DOI: 10.1007/s11336-015-9457-x] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/15/2012] [Indexed: 06/04/2023]
Abstract
The present work explores the connections between cognitive diagnostic models (CDM) and knowledge space theory (KST) and shows that these two quite distinct approaches overlap. It is proved that in fact the Multiple Strategy DINA (Deterministic Input Noisy AND-gate) model and the CBLIM, a competence-based extension of the basic local independence model (BLIM), are equivalent. To demonstrate the benefits that arise from integrating the two theoretical perspectives, it is shown that a fairly complete picture on the identifiability of these models emerges by combining results from both camps. The impact of the results is illustrated by an empirical example, and topics for further research are pointed out.
Collapse
Affiliation(s)
- Jürgen Heller
- Department of Psychology, University of Tübingen, Schleichstr. 4, 72076 , Tübingen, Germany.
| | | | | | | |
Collapse
|
|
48
|
|
|
49
|
Naïve Tests of Basic Local Independence Model’s Invariance. SPANISH JOURNAL OF PSYCHOLOGY 2015; 18:E26. [DOI: 10.1017/sjp.2015.24] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
AbstractThe basic local independence model (BLIM) is a probabilistic model for knowledge structures, characterized by the property that lucky guess and careless error parameters of the items are independent of the knowledge states of the subjects. When fitting the BLIM to empirical data, a good fit can be obtained even when the invariance assumption is violated. Therefore, statistical tests are needed for detecting violations of this specific assumption. This work provides an extension to theoretical results obtained by de Chiusole, Stefanutti, Anselmi, and Robusto (2013), showing that statistical tests based on the partitioning of the empirical data set into two (or more) groups are not adequate for testing the BLIM’s invariance assumption. A simulation study confirms the theoretical results.
Collapse
|
|
50
|
Formal psychological assessment in evaluating depression: a new methodology to build exhaustive and irredundant adaptive questionnaires. PLoS One 2015; 10:e0122131. [PMID: 25875359 PMCID: PMC4398546 DOI: 10.1371/journal.pone.0122131] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2014] [Accepted: 02/10/2015] [Indexed: 12/11/2022] Open
Abstract
Psychological Assessment can be defined as a complex procedure of information collection, analysis and processing. Formal Psychological Assessment (FPA) tries to improve this procedure by providing a formal framework to build assessment tools. In this paper, FPA is applied to depression. Seven questionnaires widely used for the self-evaluation of depression were selected. Diagnostic criteria for major depressive disorder were derived from the DSM-5, literature and Seligman's and Beck's theories. A Boolean matrix was built, including 266 items from the questionnaires in the rows and 20 selected attributes, obtained through diagnostic criteria decomposition, in the columns. In the matrix, a 1 in a cell meant that the corresponding item investigated the specific attribute. It was thus possible to analyze the relationships between items and attributes and among items. While none of the considered questionnaires could alone cover all the criteria for the evaluation of depressive symptoms, we observed that a set of 30 items contained the same information that was obtained redundantly with 266 items. Another result highlighted by the matrix regards the relations among items. FPA allows in-depth analysis of currently used questionnaires based on the presence/absence of clinical elements. FPA allows for going beyond the mere score by differentiating the patients according to symptomatology. Furthermore, it allows for computerized-adaptive assessment.
Collapse
|