The Instructure Community will enter a read-only state on November 22, 2025 as we prepare to migrate to our new Community platform in early December. Read our blog post for more info about this change.
1 Solution
Hi @bneporadny ,
TLDR; can you tell we're excited about this! :smileygrin:
I have been working on this solution for about 6 months now. I have designed it to be shared with the Canvas Community and it has gone through several iterations. I recently got over a few hurdles after working with Canvas on some details and then they shifted some resources to improving Live Events, and if you look at the Roadmap are working on releasing some of those improvements over the next few months. Some of those improvements are to handle an increased volume of schools deciding to use Live Events and how they can consume them (SQS or HTTPS), subscribe to them (in the UI) and changing which to use at the UI level (Caliper vs Canvas Raw). Currently Canvas has to have an engineer set it up for each instance. Because of these shifts and improvements, I told them I'd hold off on releasing the code to prevent an influx of requests before they had everything in place.
With that said... I was planning on a more formal release, blog post and explanation of the code early next year, but would be happy to share what I've been working on and why. So...
I was listening to Huddie Ledbetter aka Leadbelly's blues music one morning when I decided that was a fantastic name for my Live Events Daemon. LEDbelly, as I've come to call it, is a lightweight daemon that runs on a system (currently my Mac Mini while I build our new task server), connects to Amazon SQS, consumes messages for either Canvas Raw or Caliper format, and can push them into SQL.
Because I've used numerous community contributions over the years to help run our processes, including the many contributions of @James ... and when I was working on this and found few resources and mentions of Live Events here, I was inspired to write the code with the intention of making it available to anyone.
Some of my choices around Canvas Community were asking what databases are used so that I could support them. I can currently push events into MSSQL, MySQL, PostgreSQL and would like to support Oracle, but am waiting on my dev environment to be setup and anyone (using Oracle) in the community to answer a few questions.
My intention in making this available:
There are better ways to handle this data... (Splunk, AWS)
There are schools with money who can build nice data lakes and process this data on AWS...
we don't have that funding or the skills, yet. many of us don't, this is for us!
Not everyone is a data analyst, programmer, or understands Big Data
For the rest of us, SQL is the only way to handle data... unstructured or not
If we all write our own programs we're all reinventing the wheel instead of improving it
I'm hopeful there is interest from the community to collaborate and build something we can all use...
... if not, it will be available as is (as I add to it) for your use or as a starter kit. I will be contributing this to our Github account early 2019 when Canvas can handle the increased volume and turning on LE for Canvas is easier.
Since you asked about best practices and how to use it, I'll share some details.
I will definitely be writing another article on this soon but would like to give you some features now.
Ledbelly
When not running in the background I can watch events happening in real time. The date is old because I tried to post a 60 second video last month but Twitter hated the format.
Multi-threaded daemon for consuming SQS messages quickly... Multi threading is necessary because sometimes you'll see one or two events a second, or a minute... during peak times many many more. This is especially necessary since multiple events happen at the same time, I found that importing 1 message at a time in SQL would quickly create a backup and latency as I couldn't insert them fast enough.
Here's a screen grab from our internal ops dashboard smithb_ccsd developed to keep an eye on all our in-out processes.
Connection pooling, just like multi threading, the database connection needs to be able to handle increases and decreases in the amount of transactions happening in the database.
The rest of the features are pretty basic. I chose the Canvas Raw Format vs the Caliper format because its compatible with Canvas Data and is easier to consume, and it has more events. Both can be consumed at the same time. The code is currently set up to import the Caliper Format into SQL with underscore notation and dynamically generate tables in a test environment... I do this to track changes and help answer questions that pop up. Thoughts on that here... https://community.canvaslms.com/docs/DOC-15741-event-type-by-format#comment-132790
For the Canvas Raw format, I have mapped the event fields, setup data types and have generated DDLs for the 3 databases I mentioned above and written code to dynamically generate those DDLs when I needed an update or to create documentation. Live Events changes quite often, just this week they added client_ip to many of the events. I've been trying to help update the documentation, by creating Pull Request #1379 with an update to the API doc...
canvas-lms/live_events.md at live-events-documentation · robert-carroll/canvas-lms · GitHub
I've spent a decent amount of time trying to track the changes in the Canvas Github commits to keep the code up to date. On the off chance that I miss something in the code here are the exception handlers...
New Events... or unidentified events the code wasn't expecting. If a new event is sent to the SQS stream and the code isn't ready for it we'd either throw an error or (because I'm using a Case statement) would completely ignore it. When this happens I instead push that event to a secondary SQS queue (mine is called -moo, because cows have multiple bellies and I'm a nerd). This allows me to catch the event, evaluate it, and update the code. I can also log the payload to a local file.
Field Count... sometimes new fields appear in the event the code wasn't expecting, and I use this method (which is written to an error file) to evaluate the field, assess its data type and update the schema and the code. Otherwise the code would quietly continue working and never add that data to the database.
---
One of the things I've tried to do is prevent myself from making assumptions for the community where we'd probably benefit from collaborating on best practices. Currently, I push all events into their own table, which makes it easier to join data or update data with Canvas Data... but just yesterday I added a new method which grabs common fields in all events and pushes them to a live_stream table, which I use for user activity. This allows me to create faster views when trying to associate things in Canvas data for Users vs Records... joins are slow, so I currently feel like the speed benefit is worth the duplication. Also since Canvas Data will pick up most of this information within 48 hours... we probably don't need to store it for too long. But I haven't found how long I should hold on to it yet...
If you are already using Canvas Data, a lot of the events can be appended to reports to giving you the most updated submission timestamp or activity for a user, this helps extend Canvas Data since your last dump/import.
One thing we've been working on is displaying current number of users who are online and eliminating noise from the Requests Table which I... The Problem with the Requests Table. I can easily eliminate Amazon Data Centers (backend Canvas tasks) from current user activity. I have not quite figured out yet how to eliminate all noise from the table... but not entirely sure now if that's even necessary. I had originally asked for an IP address in the Login Event... but with IP in almost all events... the Requests Table starts becoming more useful for troubleshooting DevOps and random questions CD won't answer like this and this, not user activity...
User locations in the last 2 hours... (I append geo location data to this and view it in Tableau)
Last but not least, with Canvas Data... since we have over 350 schools. I can use the context_id from the event, get the course_id then the account_id and list the current users in Canvas at each School.
That's just who and how many data... Live Events is extremely useful in allowed applications and processes to act in real time instead of off old Canvas Data or static Account Reports.
If you already have Live Events, we can possibly start collaborating before I push the code publically.
Please feel free to ask any questions or DM me.
Hi @bneporadny ,
TLDR; can you tell we're excited about this! :smileygrin:
I have been working on this solution for about 6 months now. I have designed it to be shared with the Canvas Community and it has gone through several iterations. I recently got over a few hurdles after working with Canvas on some details and then they shifted some resources to improving Live Events, and if you look at the Roadmap are working on releasing some of those improvements over the next few months. Some of those improvements are to handle an increased volume of schools deciding to use Live Events and how they can consume them (SQS or HTTPS), subscribe to them (in the UI) and changing which to use at the UI level (Caliper vs Canvas Raw). Currently Canvas has to have an engineer set it up for each instance. Because of these shifts and improvements, I told them I'd hold off on releasing the code to prevent an influx of requests before they had everything in place.
With that said... I was planning on a more formal release, blog post and explanation of the code early next year, but would be happy to share what I've been working on and why. So...
I was listening to Huddie Ledbetter aka Leadbelly's blues music one morning when I decided that was a fantastic name for my Live Events Daemon. LEDbelly, as I've come to call it, is a lightweight daemon that runs on a system (currently my Mac Mini while I build our new task server), connects to Amazon SQS, consumes messages for either Canvas Raw or Caliper format, and can push them into SQL.
Because I've used numerous community contributions over the years to help run our processes, including the many contributions of @James ... and when I was working on this and found few resources and mentions of Live Events here, I was inspired to write the code with the intention of making it available to anyone.
Some of my choices around Canvas Community were asking what databases are used so that I could support them. I can currently push events into MSSQL, MySQL, PostgreSQL and would like to support Oracle, but am waiting on my dev environment to be setup and anyone (using Oracle) in the community to answer a few questions.
My intention in making this available:
There are better ways to handle this data... (Splunk, AWS)
There are schools with money who can build nice data lakes and process this data on AWS...
we don't have that funding or the skills, yet. many of us don't, this is for us!
Not everyone is a data analyst, programmer, or understands Big Data
For the rest of us, SQL is the only way to handle data... unstructured or not
If we all write our own programs we're all reinventing the wheel instead of improving it
I'm hopeful there is interest from the community to collaborate and build something we can all use...
... if not, it will be available as is (as I add to it) for your use or as a starter kit. I will be contributing this to our Github account early 2019 when Canvas can handle the increased volume and turning on LE for Canvas is easier.
Since you asked about best practices and how to use it, I'll share some details.
I will definitely be writing another article on this soon but would like to give you some features now.
Ledbelly
When not running in the background I can watch events happening in real time. The date is old because I tried to post a 60 second video last month but Twitter hated the format.
Multi-threaded daemon for consuming SQS messages quickly... Multi threading is necessary because sometimes you'll see one or two events a second, or a minute... during peak times many many more. This is especially necessary since multiple events happen at the same time, I found that importing 1 message at a time in SQL would quickly create a backup and latency as I couldn't insert them fast enough.
Here's a screen grab from our internal ops dashboard smithb_ccsd developed to keep an eye on all our in-out processes.
Connection pooling, just like multi threading, the database connection needs to be able to handle increases and decreases in the amount of transactions happening in the database.
The rest of the features are pretty basic. I chose the Canvas Raw Format vs the Caliper format because its compatible with Canvas Data and is easier to consume, and it has more events. Both can be consumed at the same time. The code is currently set up to import the Caliper Format into SQL with underscore notation and dynamically generate tables in a test environment... I do this to track changes and help answer questions that pop up. Thoughts on that here... https://community.canvaslms.com/docs/DOC-15741-event-type-by-format#comment-132790
For the Canvas Raw format, I have mapped the event fields, setup data types and have generated DDLs for the 3 databases I mentioned above and written code to dynamically generate those DDLs when I needed an update or to create documentation. Live Events changes quite often, just this week they added client_ip to many of the events. I've been trying to help update the documentation, by creating Pull Request #1379 with an update to the API doc...
canvas-lms/live_events.md at live-events-documentation · robert-carroll/canvas-lms · GitHub
I've spent a decent amount of time trying to track the changes in the Canvas Github commits to keep the code up to date. On the off chance that I miss something in the code here are the exception handlers...
New Events... or unidentified events the code wasn't expecting. If a new event is sent to the SQS stream and the code isn't ready for it we'd either throw an error or (because I'm using a Case statement) would completely ignore it. When this happens I instead push that event to a secondary SQS queue (mine is called -moo, because cows have multiple bellies and I'm a nerd). This allows me to catch the event, evaluate it, and update the code. I can also log the payload to a local file.
Field Count... sometimes new fields appear in the event the code wasn't expecting, and I use this method (which is written to an error file) to evaluate the field, assess its data type and update the schema and the code. Otherwise the code would quietly continue working and never add that data to the database.
---
One of the things I've tried to do is prevent myself from making assumptions for the community where we'd probably benefit from collaborating on best practices. Currently, I push all events into their own table, which makes it easier to join data or update data with Canvas Data... but just yesterday I added a new method which grabs common fields in all events and pushes them to a live_stream table, which I use for user activity. This allows me to create faster views when trying to associate things in Canvas data for Users vs Records... joins are slow, so I currently feel like the speed benefit is worth the duplication. Also since Canvas Data will pick up most of this information within 48 hours... we probably don't need to store it for too long. But I haven't found how long I should hold on to it yet...
If you are already using Canvas Data, a lot of the events can be appended to reports to giving you the most updated submission timestamp or activity for a user, this helps extend Canvas Data since your last dump/import.
One thing we've been working on is displaying current number of users who are online and eliminating noise from the Requests Table which I... The Problem with the Requests Table. I can easily eliminate Amazon Data Centers (backend Canvas tasks) from current user activity. I have not quite figured out yet how to eliminate all noise from the table... but not entirely sure now if that's even necessary. I had originally asked for an IP address in the Login Event... but with IP in almost all events... the Requests Table starts becoming more useful for troubleshooting DevOps and random questions CD won't answer like this and this, not user activity...
User locations in the last 2 hours... (I append geo location data to this and view it in Tableau)
Last but not least, with Canvas Data... since we have over 350 schools. I can use the context_id from the event, get the course_id then the account_id and list the current users in Canvas at each School.
That's just who and how many data... Live Events is extremely useful in allowed applications and processes to act in real time instead of off old Canvas Data or static Account Reports.
If you already have Live Events, we can possibly start collaborating before I push the code publically.
Please feel free to ask any questions or DM me.
Canvas Data definitely requires a beefier box. I'm lucky that ours is a VM and I can just ask for more resources when necessary. Spam now?! You wear so many hats!
@bneporadny , DM'd me Friday and we carried on a little about the difficulties of processing events and I shared some examples. He's currently getting the Caliper format, which I keep neglecting now that I have Canvas Raw. Anyway, I was working on the house this weekend (finally finishing the baby's room with casings, baseboard and crown molding) and rethinking this thread and decided I hate really_long_underscore_notation. Since, the easiest way for me to get Caliper into SQL so far was flattening the JSON and importing the key as the column, which left me with the following mess for a table structure. This morning, I've modified it a little and added _squish method.
CREATE TABLE dbo.ims_asset_accessed (
_context varchar(255) NULL,
id varchar(255) NULL,
[type] varchar(255) NULL,
actor_id varchar(255) NULL,
actor_type varchar(255) NULL,
actor_extensions_com_instructure_canvas_user_login varchar(255) NULL,
actor_extensions_com_instructure_canvas_root_account_id varchar(255) NULL,
actor_extensions_com_instructure_canvas_root_account_lti_guid varchar(255) NULL,
actor_extensions_com_instructure_canvas_root_account_uuid varchar(255) NULL,
actor_extensions_com_instructure_canvas_entity_id varchar(255) NULL,
[action] varchar(255) NULL,
object_id varchar(255) NULL,
object_type varchar(255) NULL,
object_name varchar(255) NULL,
object_extensions_com_instructure_canvas_asset_type varchar(255) NULL,
object_extensions_com_instructure_canvas_entity_id varchar(255) NULL,
eventtime varchar(255) NULL,
edapp_id varchar(255) NULL,
edapp_type varchar(255) NULL,
group_id varchar(255) NULL,
group_type varchar(255) NULL,
group_extensions_com_instructure_canvas_context_type varchar(255) NULL,
group_extensions_com_instructure_canvas_entity_id varchar(255) NULL,
membership_id varchar(255) NULL,
membership_type varchar(255) NULL,
membership_member_id varchar(255) NULL,
membership_member_type varchar(255) NULL,
membership_organization_id varchar(255) NULL,
membership_organization_type varchar(255) NULL,
membership_roles varchar(255) NULL,
session_id varchar(255) NULL,
session_type varchar(255) NULL,
extensions_com_instructure_canvas_hostname varchar(255) NULL,
extensions_com_instructure_canvas_request_id varchar(255) NULL,
extensions_com_instructure_canvas_user_agent varchar(255) NULL,
extensions_com_instructure_canvas_version varchar(255) NULL,
object_extensions_com_instructure_canvas_asset_subtype varchar(255) NULL,
actor_extensions_com_instructure_canvas_real_user_id varchar(255) NULL
);I think it needs a little improvement on structure. Sharing in case any Rubyist's can improve it.
def _flatten(hash, recursive_key = '')
hash.each_with_object({}) do |(k, v), ret|
key = recursive_key + k.to_s
key = key.gsub(/[^a-zA-Z]/, '_')
if v.is_a? Hash
ret.merge! _flatten(v, key + '_')
else
ret[key] = v
end
end
end
def _squish(hash)
hash = _flatten(hash)
hash.each_with_object({}) do |(k, v), ret|
k = k.gsub(/extensions|com|instructure|canvas/, '').gsub(/_+/, '_').gsub(/^_/, '').downcase
ret[k] = v
end
endWhich resulted in a much nicer DDL. Now, I can start writing code that isn't dynamic for Caliper, which would allow me to pass those to a production environment... and Ledbelly would be able to handle either or both formats simultaneously. Sometimes just saying something out loud makes you realize there's a better way.
CREATE TABLE dbo.ims_asset_accessed (
context varchar(255) NULL,
id varchar(255) NULL,
[type] varchar(255) NULL,
actor_id varchar(255) NULL,
actor_type varchar(255) NULL,
actor_user_login varchar(255) NULL,
actor_root_account_id varchar(255) NULL,
actor_root_account_lti_guid varchar(255) NULL,
actor_root_account_uuid varchar(255) NULL,
actor_entity_id varchar(255) NULL,
actor_real_user_id varchar(255) NULL,
[action] varchar(255) NULL,
object_id varchar(255) NULL,
object_type varchar(255) NULL,
object_name varchar(255) NULL,
object_asset_type varchar(255) NULL,
object_entity_id varchar(255) NULL,
eventtime varchar(255) NULL,
edapp_id varchar(255) NULL,
edapp_type varchar(255) NULL,
group_id varchar(255) NULL,
group_type varchar(255) NULL,
group_context_type varchar(255) NULL,
group_entity_id varchar(255) NULL,
membership_id varchar(255) NULL,
membership_type varchar(255) NULL,
membership_member_id varchar(255) NULL,
membership_member_type varchar(255) NULL,
membership_organization_id varchar(255) NULL,
membership_organization_type varchar(255) NULL,
membership_roles varchar(255) NULL,
session_id varchar(255) NULL,
session_type varchar(255) NULL,
hostname varchar(255) NULL,
request_id varchar(255) NULL,
user_agent varchar(255) NULL,
version varchar(255) NULL,
object_asset_subtype varchar(255) NULL
);
Hi folks,
The LTI uses a combination of Canvas Data and Live Events, filling in gaps where Canvas data has older records.
I'd like to organize a small group of users who are interested in collaborating and polishing the project (LEDBelly, not the LTI) for community use. We could test the code across organizations, database environments and figure out some common features and fix some bugs when we find them. If you are interested please DM me. Once I get a short list we'll work out a time for webex (or whoever can offer this space), and I'll post the code to a private Github repository while we work out the kinks. We can work out issues so that it is available when Canvas releases their new changes.
It would be helpful if those interested in this phase are:
Currently receiving either Canvas Raw or Caliper format events, or both.
Understand or can install and and use Ruby code and dependencies. The code is intentionally basic.
MSSQL, MySQL, PostgreSQL, or Oracle database environments.
I'd love to work with anyone who's especially talented at Indexing and Query Execution Plans. The Query used to display the information on the above screenshot utilizes a couple of views with Joins on CD/LE, and Stored Procedure with Temp Table use and additional Joins. With a WHERE clause in the temp table for course_id, the query takes about 3-8 seconds to load. I'd like to be able to utilize this data on larger sets and not have it take 60+ seconds... :smileygrin:
I'd also like to work on a Rakefile to automate the configuration setup. Not that the setup is hard, but we could probably programmatically automate and extend some features:
I duplicate all events (with common columns) into a 'live_stream' table, to prevent massive joins on dozens of tables. I use this table more than the individual tables. Others might want to configure this differently.
I have another program that's 1 file which collects and stores the GEO location data from the IP address... do we want to include this as a sub routine, or do people even want that?
Community help
To interact with Panda Bot, our automated chatbot, you need to sign up or log in:
Sign in
This discussion post is outdated and has been archived. Please use the Community question forums and official documentation for the most current and accurate information.