Measuring students' AI competence: Development and validation of a multidimensional scale integrating educational psychology perspectives

Abstract

The rapid integration of Artificial Intelligence (AI) into K-12 education has created a pressing need to evaluate students' capacity to engage with AI effectively and responsibly. As outlined in UNESCO's AI Competency Framework, AI competence is a multidimensional construct encompassing cognitive (knowledge and skills), affective (values and motivation), ethical (principles and responsibilities), and technical (practical and design-based) dimensions. Without systematic assessment, educators lack the evidence needed to ensure students possess not only technical proficiency but also the critical thinking, ethical awareness, and motivation required in AI-driven learning environments. To address this gap, the present study develops and validates the AI Competence Scale (AICS), a multidimensional instrument grounded in UNESCO's framework and educational psychology principles. The AICS conceptualizes AI competence across four interrelated dimensions-Human-Centred Mindset, Ethics of AI, AI Techniques and Applications, and AI System Design-capturing the interplay between students' knowledge, values, reasoning, and applied skills.Data were collected from 608 high school students in public secondary schools in Qatar, following a multi-stage process: expert reviews for content validity; exploratory and confirmatory factor analyses for construct validity; reliability testing via Cronbach's alpha and composite reliability; and convergent, discriminant, and nomological validity using AVE, HTMT ratios, and structural equation modeling (SEM). The results confirmed strong psychometric properties, with SEM revealing a significant positive association between AI competence and students' interest in AI-related careers. The validated AICS provides educators, researchers, and policymakers with a theoretically grounded and empirically robust tool for assessing students' readiness for ethical and informed engagement with AI, supporting curriculum design, targeted interventions, and evidence-based policy in K-12 AI education.