What data files do I need?
You may have started to think about some research goals, but now you need assistance understanding the data sources needed to run rapid cycle evaluations. We know that it can be a lot to figure out on your own, especially if you’ve never collected data specific to your particular research goal(s) before. This resource page will help you determine which data files you will need, how to find them, and what to do if data is unavailable.
Your LearnPlatform Research Team is here to support this work along the way, from planning with you, to supporting the discovery of data files mentioned in this document, as well as leading on any necessary data cleaning, running your RCEs, and providing any associated summary reports. Your team’s primary roles are to help us access data we need, and then use the results to make decisions!
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
Why do you need data?
Data that you're already collecting or planning to collect provides the information needed to help your team reach your goal(s). One piece of data may be sufficient (e.g., usage analysis) or more than one type of data may be needed (e.g., outcomes analysis). This is because different data answers different questions. We’ll help you determine which data you’ll need.
What data do you need?
Depending on your research goal(s), there are four main types of data that you will need to run a rapid cycle evaluation (RCE) with Rapid Cycle Evaluation RCE™:
- Usage data
- Student information system (SIS) data
- Outcome data
- Pricing data
The following sections provide additional information about each of these types of data.
For more information on preparing your data for Rapid Cycle Evaluation (RCE)™, please check our Community space for related articles .
✔ Usage data
An error occurred.
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
Usage data represents the extent to which a user (e.g., student) participated or engaged with the edtech tool. Examples of usage data (also sometimes referred to as usage metrics) include minutes on system, days of use, total number of logins, total lessons attempted, number of lessons completed, and number of lessons mastered.
Only one usage metric is needed to run a RCE.
How do I find usage data? Edtech vendors often allow administrators to export usage data. Depending on the edtech tool that you choose, you may need to contact your district administrator in charge of the intervention or the edtech company directly to get usage data. This data will need to be available at the user level. District or school-level data is not appropriate for use in rapid cycle evaluation.
Our team can help you determine whether a vendor's usage data is appropriate for RCE with Rapid Cycle Evaluation RCE™.
✔ Student Information Systems (SIS) Data
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
Student Information Systems (SIS) data consists of student identifiers and demographic data. Example data include student ID, grade level, school, race/ethnicity, gender, special education status, and free/reduced lunch status..
There are two ways that RCE work uses SIS data:
- To account for differences in the main analysis: Rapid Cycle Evaluation RCE uses SIS data as a covariate in the main analysis, because SIS differences across students, classrooms or schools may influence the study outcome. For example, the comparison group might have more students receiving special education services. By including an indicator for special education services, we can account for this variation between the groups so that it does not influence the results.
- To examine differences in results by subgroups: Rapid Cycle Evaluation RCE allows you to examine overall results and results for specific subgroups. If you include SIS data, you will be able to drill down into usage and performance within SIS subgroups (e.g., grade level, gender, school). For example, did using the edtech product result in better learning outcomes based on grade level?
How do I find SIS data? You may need to contact your district's SIS Administrator or Data Manager to give you access to that data, or to include them in the data discovery process.
While SIS data is optional when running a RCE regardless of your research goals, we encourage you to include as much SIS data as is appropriate to rule out the effects of other variables in order to gain confidence that effects can be attributed to the intervention. Your team can work with LearnPlatform to determine the optimal utilization of this data type.
✔ Outcome data
An error occurred.
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
Outcome data include measures of educational outcomes that the edtech product claims to impact. To run an outcomes analysis with LearnPlatform (i.e., an analysis that links edtech product use to student learning), our research team will ask for a standardized, reliable, and valid outcome measure. To examine how edtech impacts student learning, we recommend using an outcome measure to represent student achievement.
Examples of outcome data include standardized assessment scores, GPA, or attendance. There are two main types of outcome data:
- Pre-intervention outcome data refers to performance or scores prior to using the edtech product. We use this data as a covariate in RCEs to account for differences in student performance before the intervention (i.e., edtech product) was available to students.
- Post-intervention outcome data refers to performance or scores after using the edtech product. We use this data to examine student performance and impact after receiving the intervention.
There are a variety of research questions that an Outcomes Analysis can help your team answer. Two common types of Outcomes Analyses are described below:
In an Outcomes Analysis without a Comparison Group, you examine the relationship between edtech product usage and your post-intervention outcome.In an Outcomes Analysis with a Comparison Group, you examine if students who use the edtech product outperform other students, based on post-intervention outcome performance.
How do I find outcome data? Most assessment systems provide easy access to user-level reporting. You may need to contact your district's assessment director for access to these reports. Your RCE team member can also assist you by providing example data.
✔ Pricing data
An error occurred.
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
Pricing data includes the cost of the edtech product subscription or license, priced per student or site. Pricing data may also include other direct and indirect costs associated with owning the product (e.g., professional development, staff hours). Pricing data can be included in Usage and Outcomes Analyses, but is not required.
How do I find pricing data? Pricing is usually entered into your product library during technical implementation t and, as a result, can be found there. Alternatively, cost information can be submitted while setting up the RCE if it has not already been shared with LearnPlatform . If you are taking this approach, you may need to contact someone within your organization who has purchasing information for the edtech tool of interest. LearnPlatform can also help you determine the approximate cost per user if that is not clear from the provider’s invoicing.
Check out the RCE Data Collection Checklist for a helpful analysis of your files before submission.
What if the data you need is unavailable or inaccessible?
There are many reasons why appropriate data might not be available. Sometimes data is inaccessible because the vendor will not grant access or the data is not currently being collected by the vendor. Other times, the data is presented in an unusable way (for example, an aggregated district count of activities completed vs user-level data). ). If data required is not available, the research question may not be answerable, which is why it is important to check what data is available as you are formulating your research goal. Your RCE team can support you throughout this entire process, and also provide alternative options if necessary, to get you the most information possible.
