How do I request usage data for running a Rapid Cycle Evaluation (RCE)?

As a LearnPlatform administrator, you may need to request usage data from product vendors for a Rapid Cycle Evaluation (RCE). When requesting data, you can explain important details and use an email template as outlined in this guide. There are also several recommendations and tips from getting usage data from your own team.

It may also be useful to provide the format for the CSV usage file data, including some sample data files.

Notes:

Request Data from Providers

To request data from providers, explain the importance of understanding what's working and driving improvement, as well as how the data will be used. Use talking points such as the following:

"Our district works with LearnPlatform to manage and evaluate our edtech. They support us in making edtech evaluation part of our standard practices  to ensure we are implementing products well and getting the most value from our investments for our students and teachers. In order to do this, we need to understand how and to what extent our students and teachers are using your solution.

The usage data will be used to run rapid-cycle evaluations (RCEs) within LearnPlatform to better understand how products are working, for which students and teachers, and under what conditions. RCEs help us generate evidence needed to:

  • Build on our most successful implementations to advance best practices across our district.
  • Identify needs for additional teacher support and/or professional learning refinement with certain tools.
  • Understand where usage is not meeting expectations, so we can begin to explore why and uncover any barriers to use.
  • Discover the impact of program changes and interventions over time on specific groups.
  • Generate evidence needed to comply with local, state and federal regulatory and reporting requirements.

The data will be put through LearnPlatforms data-secure, Rapid Cycle Evaluation engine, alongside other data from our district (like demographic information and assessment scores). Evaluating program usage data alongside other data helps us determine how to improve implementation for better results.

Have questions? Please email support-LP@instructure.com."

Use Provider Email Template

To communicate with providers, you can use the following email template:

Hi [name],

As you know, our district wants to better understand how our students and teachers use your product, and how we can better support that use. In order to do that, we need to access our usage data  here are some details about how the data will be used. Below, you will find more details, including data formatting standards and privacy/compliance information.

Data Format: Data coming to LearnPlatform for use in rapid-cycle evaluations should meet the following basic criteria:

  • User-level information (vs class, school or averaged data).
  • All external identifiers, including district or third-party IDs included.
  • One row per user.
  • At least one categorical usage metric (time, lessons completed, etc.). Note: Multiple usage metrics are encouraged.
  • Include data for only the requested time period.
  • Data is shared in a .CSV formatted file.

Sharing Data: We use ExaVault for receiving files that include personally identifiable data (PII). This URL [link] can be used by your team to share files directly to LearnPlatform. There is no password required.

If you prefer to use your own internal systems to share data with LearnPlatform or if you would like to automate data sharing, please let us know, and we can determine the next best steps.

Data Use: Based on the parameters of this engagement with [organization name], we anticipate that the data you share with us will be used in the following ways: usage reports, outcomes reports, and on these general timelines [quarterly].

We are planning to support [district name] in data collection starting on [date]. Who can we work with on your team to discuss the requirements, answer questions and collect the data?

Thank you, and we look forward to partnering with you on this work!

Get Usage Data from Your Own Team

If you need support in getting usage data from team members inside of your own organization, you can use a key talking point such as the following:

"We are working with LearnPlatform to help us identify areas where our implementation of edtech programs is going well, and where there are opportunities to have better, more detailed conversations about whats working for your staff and students. We will only be asking you for the specific data that we need to conduct this work."

Recommendations and Tips

  • Do: Share often and share openly. It is easier to proactively discuss the value of RCE work with your team and other stakeholders.
  • Do: Consider using LearnPlatform resources before the implementation of RCE work begins, and consider which groups of people may have specific questions or concerns:
    • District leadership (cabinet level): Provide specific information about your plans, and how the results may be used. Encourage conversations and discussions earlier vs later, and make sure to share initial reports and results for feedback and to increase engagement.
    • Your academic teams: Provide them with information about how RCE work can help them to improve their teaching practices. Share that this work is not punitive, but rather a formative way to get data-driven information so that they can all have better conversations about whats working, and for whom. Invite them to be an active participant in this process as early as possible.
    • Your data and assessment teams: Reinforce that RCE work with LearnPlatform is there to increase capacity and effectiveness of the work they already do; and that no ones position or responsibilities will be threatened in any way. Be specific about what data files are going to be used for; and set the tone early on. Make sure your team understands that LearnPlatform does not make any decisions about how to interpret or act on RCE results.
  • Do not: Wait too long. The sooner you can speak with your teams and promote understanding of this work, the easier it will be to work with them now and in the future.

Provide Usage Data File Requirements to Solution Providers

Below are the content and format requirements for usage data files:

  • File Type: CSV
  • File Naming: The file name should include the name of the product and the evaluation time frame. E.g. SmartMathUsage_01-01-2020to12-30-2020
  • File Format: The data should be delivered in a CSV where each row is a unique record or case (almost always a student), and each column is a variable on which data are collected for each record.
  • Data Requested: User-level (most often student-level) program data including usage and/or performance metrics for a given timeframe.

File Format

Column Description
Product Identifier
 Optional: Unique identifier for the product; assigned by LearnPlatform.
NCES ID
 Unique identifier for the School or School District.
External Student/User Identifier
 Unique code, number or ID assigned to represent individuals by the district or a third party tool (e.g., district ID, Clever ID, state IDs).
Metric Type

Column name that identifies the type of usage metric being sent (minutes_on_system, logins, modules_complete, click). 

*multiple metrics can be included.
Metric Value
 Numeric value that corresponds to each metric type included.

*Primary Key = Product Identifier, NCES ID, Student/User Identifier, Period Type, Period Value, Metric Type

File Examples

The examples below illustrate the different ways data from various providers can look. Your solution is unique and LearnPlatform research tools are flexible. If you have questions about your program data and how it fits into this template, please contact support at lp-support@instructure.com.

Example A

In this example, usage metrics reported are minutes on the system and count of questions. The unique student identifier in use is District ID numbers.

QuizMaster_01-01-2020_to_06-30-2020.csv



Product Identifier
 NCES ID
 Student Identifier
 Minutes on system
 Questions completed
123123 456456
 789789
 244 4
123123 456456
 789212
 531 23

Example B

In this example, usage metrics reported are logins, count of modules completed and total session time. The unique student identifier in use is Student Name.

MathWhiz_05-01-2022_to_06-30-2022.csv




Product Identifier
 NCES ID
 Student Identifier
 Logins Modules Complete Total Session Time
123123 456456 George Washington 45 1 10:20:20
123123 456456 Thomas Jefferson 5 1 00:32:00
92342032342
 382425
 James Madison 23 0 05:50:08