Understanding How Students Interact with Assessments
Educational assessments, such as the National Assessment of Educational Progress (NAEP), help educators, parents, and stakeholders understand what students know and can do in various subjects. Traditional assessment data, or collected test responses, are one way of understanding student achievement. NAEP has often been called the “gold standard” of assessments because it is developed using the best thinking of assessment and content specialists, education experts, and teachers from around the nation.
In 2017, the NAEP assessment transitioned to a digital format, opening up our ability to understand how students approach such assessments. The “electronic NAEP,” or eNAEP system, is able to collect information about all student interactions with the platform. Student actions are logged and coded with their associated timestamps. For instance, if a student uses a highlighter function, or zooms in on a particular resource, there is now a record of it. These data recordings are known as process data.
Juanita Hicks is a senior researcher at AIR and a contributor to the Center for Process Data, a research and analysis hub for large amounts of education assessment data, such as NAEP. In this Q&A, she explains what process data can reveal and how this relatively new source of information could potentially lead to more equitable ways of assessing educational achievement.
Q: Your current work focuses on analyzing NAEP digital assessment data. What information can be uncovered by examining these data?
Hicks: We’ve found through a search of relevant literature and our own work at AIR’s Center for Process Data that there are several areas where process data has been useful. To construct a better representation of the areas where process data can be used, we created what we call the “process data ecosystem,” to help contextualize our process data research. Using the ecosystem as a guide, we have conducted process data research to explore various topics, such as scoring practices, test assembly, item development, assessment tools, and student behavior.
For example, when it comes to student behavior, researchers might be concerned with the average time it takes for groups, such as students with disabilities, to complete their assessment. Using process data, we can compare those average times across groups. Such information may lead education program administrators to reconsider time constraints based on apparent disparities revealed using process data.
Q: What is the role of process data in the broader education policy research field?
Hicks: Process data is still an emerging area, especially in education and educational assessment. Four to five years from now, process data can potentially be used to validate and improve current testing practices that have been developed from paper-pencil tests, including, for example, how items are scored. We’re on the cusp of making significant improvements to assessments, as well as the test-taking experience for all students.
Q: Where do you see the field headed in the future?
Hicks: In the future, the process data research field will become even more interdisciplinary. My hope is that the field will attract not only educational researchers, but also start to attract data scientists and experts in machine learning and computer science, as well as cognitive psychologists and content experts. People with those skills are slowly making their way into the field of process data because they are intrigued by what the data can do and all the ways in which it can be used.
Ultimately, being a trained data scientist isn’t enough. We still need experts who have some educational research and assessment knowledge to go along with data science or computer science skills. This interdisciplinary approach will make the field grow more rapidly and create a larger pool of knowledge. I think the field will become more established and more standardized over the next few years because we are already working toward establishing and standardizing variables and operational definitions for very common process data features.
As the data sets available to researchers increase in size and scope—state assessments, other national and international assessments, even formative assessments, for example—the need to expand and diversify the field will become even more important.
