Overview

The CAL curriculum assessments are designed to measure computational thinking and coding ability in both children and teachers. The assessments are rooted in the observation of behaviors, the listening of children’s stories and reasoning, and the analysis of the work they produce. They are both formative and summative. Some are incorporated into the CAL curriculum, while others are conducted by the DevTech Research Group outside of the curriculum implementation.  The assessments include evaluations of ScratchJr Knowledge, through Check for Understanding, Show What You Know, Project Rubrics, and Coding Stages Assessment and evaluations of computational thinking, through TechCheck. Additionally, ScratchJr usage data and math and literacy standardized test scores are collected. 

Check for Understanding

Checks for Understanding are the primary formative assessment within the CAL curriculum. They are short, verbal check-ins that are incorporated 5-6 times throughout the 24 lessons. They are placed between the ScratchJr structure challenges, when new coding concepts are introduced, and the ScratchJr expressive explorations, when those concepts are put to practice. They are typically 3-5 questions long and take no longer than 5 minutes for a class to complete.

Show What You Know

The primary summative assessment within the CAL curriculum is called Show What You Know (SWYK). SWYK is a 10 questions long on-paper activity, and takes no longer than 20 minutes for children to complete. SWYK is designed to check children's comprehension of the most fundamental concepts introduced in the curriculum. The SWYK for Kindergarten, 1st grade, and 2nd grade, vary in the content in order to reflect the variations in curricula across grades.

Project Rubrics

The ScratchJr Project Rubric captures children's ability to transform their coding knowledge into creating purposeful and creative projects (Unahalekhaka & Govind, 2021). There are two main aspects in the rubric: Coding Concept and Project Design. Multiple subcategories in this rubric are highly related to the computational thinking concepts such as Sequencing, Events, Repeat, and Number Input. The Coding Concept evaluates coding blocks' complexity, efficiency, and functionality, while the Project Design focuses on aesthetic elaboration and customization. The ScratchJr Project Rubric can be administered after children complete their ScratchJr projects, typically taking around 3-5 minutes per project.

Coding Stages Assessment

The Coding Stages Assessment (CSA) is one of the primary measures of coding ability used in this study. The CSA assesses progress in learning the ScratchJr programming language (De Ruiter & Bers, 2021).  This assessment is conducted one-on-one by asking the child or teacher interactive and open-ended programming questions. The assessment probes the five Coding Stages (Emergent, Coding and Decoding, Fluency, New Knowledge, and Purposefulness) that children go through when engaging in the CAL curriculum. CSA is administered as a game before and after the CAL curriculum and takes under 45 minutes to complete.

TechCheck

TechCheck (Relkin, de Ruiter, Bers, 2020) is an “unplugged” assessment of Computational Thinking (CT) that presents children with challenges analogous to those that arise in the course of computer programming but does not require coding experience to complete. The assessment probes six domains of CT described by Bers (2018) as developmentally appropriate for young children (Algorithms, Modularity, Hardware/Software, Debugging, Representation, and Control Structures). The format is multiple choice and the assessment can be administered individually or to groups in 20 minutes or less. TechCheck is administered before and after the CAL curriculum.

ScratchJr Usage

ScratchJr usage data is collected through a specialized version of the ScratchJr app, designed specifically for the CAL study. This CAL ScratchJr app is only available for the research study. Google Analytics is the tool being used to collect the usage data, which includes: which parts of the interface the user navigates, types of blocks used, and length of sessions. All data is collected at the classroom level, and the privacy policy for this CAL version of the app can be viewed here

Standardized Assessments

In addition to assessment tools created by the DevTech Research Group, the study also measures children's performance through math and literacy standardized assessments. These are collected before and after curriculum implementation.

Overview

In addition to collecting student performance data via assessments, a significant part of the study design is to collect data from teachers about the fidelity of implementation. This ensures that the curriculum and trainings in place are the most effective they can be, and can be replicated in years to come.

Surveys

Throughout the study, teachers will fill out 4 surveys: Pre-Training, Post-Training, Mid-Implementation, Post-Implementation. These surveys are designed to give a general sense of the teacher's experience throughout the study. Question topics include classroom set up, comfort levels with teaching ScratchJr and computer science, and feedback on the curricula and provided resources.

Focus Groups

Focus groups are held before and after the implementation of the CAL curricula. Focus groups are designed to give teachers an opportunity to speak more deeply to their thoughts and feelings on the curricula, and give candid feedback.

Lesson Logs

Lesson Logs are 3 minute long surveys that teachers are asked to fill out after each lesson. The logs ask about preparation time, teaching time, and activity variation for each lesson. These are the primary way to measure the fidelity of implementation.

To become certified and gain access to our validated research instruments, visit our instrument website:

This work is funded by the US Dept of Education (grant #U411C190006) and by the Scratch Foundation