Due to monetary and time constraints, it is not always possible to visit a doctor for every cough or sniffle, and over-the-counter medicine provides help in the meantime.
While a doctor is not present to explain the medication’s purpose, ingredients, dosage instructions, and dangers, any remedy you buy in a store comes with a detailed label outlining these matters. Just as it would be disadvantageous to withhold basic medicine from people who are not receiving it directly from doctors’ hands, it would be equally ludicrous to rip the labels off over-the-counter medications, leaving people with no way to use them wisely.
So, how does this relate to schools? When it comes to data such as student achievement data, educational institutions are operating without the data-equivalent to over-the-counter medicine. When most educators are given student data to analyze, via an educational technology data system and/or its reports, that data is accompanied by no embedded usage guidance. These educators are essentially forced to open data bottles with no labels, and to consume the contents nonetheless.
How Did We Get Here?
Before the rise of technology and user-friendly data systems, school staff did not have site-wide access to data like they typically do today. Staff and students could not benefit from extensive, timely data such as classroom assessment data that could be used to closely monitor student progress and customize instruction accordingly. Perhaps staff received blocks of data by the government, a consultant, or a school/district-based trainer once or twice per year, but this could not help their students on an ongoing formative basis, and it only offered teachers and administrators isolated points in time during which students’ needs could be assessed and remedies determined. With this arrangement, educators were like patients with limited access to doctors and no access to over-the-counter medication.
Now most schools are using a data system with an intelligible interface that puts data at the fingertips of most teachers and administrators. However, the data reports contained therein are generally not accompanied by guidance on their purpose, how to properly read them, how to properly interpret the data, and analysis misconceptions to avoid. Essentially, educators are like patients using medication in unmarked – or marginally marked – containers. A data system might label a report (“Growth Report,” “Year-to-Year Comparison,” etc.), but relying on mere titles is akin to taking pills in a bottle marked only as “Cold” or “Flu.” Anyone using such data to inform decisions is essentially opening a pill bottle that reads “Flu” and taking the wrong number of pills, at the wrong time, for the wrong type of flu, and in direct opposition to dangers. Imagine the ramifications for students and staff affected by the decisions made in such a way.
Can Intelligence Be a Flaw?
A national study conducted in the U.S. found most teachers are likely to analyze data by themselves (such as with no other school staff there to assist), and their responses to hypothetical student data suggested they have difficulty with question posing, data comprehension, and data interpretation; in fact, even at districts with reputations for strong data use, only 48% of teachers’ data-based interpretations were accurate (U.S. Department of Education Office of Planning, Evaluation and Policy Development, 2009, 2011). Other research supports this trend worldwide: educators are using data to inform decisions, but they aren’t always using it correctly.
Let us consider what it takes to be an educator: intelligence, extensive education, and self-sufficiency, as teachers need to be able to run a classroom in relative isolation from colleagues. So, what does someone who is smart, well-educated, and self-sufficient do when encountering something like a “simple” bar chart? The educator often says, “I can handle this. I don’t need to bother anyone else for help.” The problem is educators can’t always handle data that looks simple but is actually more complex.
What Can We Do?
With educators making so many data analysis errors even at districts known for their data prowess, current data analysis supports are clearly not enough. Professional development (PD) is needed and effective, but we further its impact when we prescribe an over-the-counter data format for the data systems and reports that educators use.
- Label Include brief footers and/or annotations with vital information (warnings about common analysis mistakes, brief example of proper way to analyze data, etc.) directly on data reports. A recent quantitative study of 211 educators of varied roles found a report footer – just a few lines of text offering data analysis guidance at the bottom of each report – more than tripled the accuracy of educators’ data analysis (Rankin, 2013).
- Supplemental Documentation Accompany the report with a reference sheet and/or reference guide specific to the report. It should offer more details concerning the test, how to read the report, how to use it to answer essential questions, and where to get more information. A reference guide – just two or three pages of guidance – nearly tripled the accuracy of educators’ data analysis (Rankin, 2013).
- Help System Educators need access to an online Help system with lessons appropriate to the data system and data with which they work.
- Package/Display Data should be presented in ways that assist proper analysis, such as through appropriate graphing and usability conventions.
- Non-Expired Content Keep content and support (footers, guides, etc.) current with changing legislation and user needs.
Balance
It’s not advisable to annotate reports with every detail imaginable about the data (the purpose of an assessment and how scores are processed and the need to use multiple measures when making decisions, etc.). While there is research to support the inclusion of individual pieces of information, there is also much research calling for simplification and noting nothing gets read when supporting verbiage is vast and overwhelming. See the non-commercial research site www.overthecounterdata.com for a summary of research-based reporting guidelines from authorities so you can apply them to reports you generate, distribute, or use.
Each report’s data will have different needs, but the most vital information users might not know should always be included. Just as medicine labels are needed to help protect our well-being, the biggest incentive for over-the-counter data is the well-being of students.
References
Rankin, J. G. (2013). Over-the-counter data’s impact on educators’ data analysis accuracy. ProQuest Dissertations and Theses, 3575082. Retrieved from http://pqdtopen.proquest.com/doc/1459258514.html?FMT=ABS
U.S. Department of Education Office of Planning, Evaluation and Policy Development (2009). Implementing data-informed decision making in schools: Teacher access, supports and use. U.S. Department of Education (ERIC Document Reproduction Service No. ED504191)
U.S. Department of Education Office of Planning, Evaluation and Policy Development (2011). Teachers’ ability to use data to inform instruction: Challenges and supports. United States Department of Education (ERIC Document Reproduction Service No. ED516494)
