LEDbelly - Live Events Daemon for Canvas LMS - SQS to SQL

[robotcars]

This discussion is for the users and development of LEDbelly.

GitHub - ccsd/ledbelly: LEDbelly - Live Events Daemon for Canvas LMS - SQS to SQL

Let's Discuss

Here's a space for us to talk about features, changes, and contributions, what we'd like to change and what can be improved, as well as staying on top of future changes to Live Events that might affect LEDbelly users downstream. You can also ask for support or installation questions here, or direct message me. There are also Issues and Pull Requests, topics may be in either or both locations. I may try to curate topics here to Issues and PR's.

For a quick glance at how myself and CCSD use Canvas Live Events, you can read through the following posts.

Getting SQS message to SQL Database #comment-128692 

Live Events: Suffering with ID #comment-151924

The Problem with the Requests Table

If you are using LEDbelly, please bookmark, like, or follow in inbox this discussion so that you get updates when there are new comments.

[luo_dongchuan]

Hi Robert,

I have a question about the changes to the event, such as Canvas adds a new column to a event in future, then you will do some changes to your code, after you changed, how can the other LEDBelly user(such as me) apply those changes to our LEDBelly instance? Do we need to download a full or part of your code again and reinstall them or just run "rake create_tables" command, then run the SQL scripts in /ddl directory?

Thanks.

Luo 

[robotcars]

Hi Luo,

Great question. Are you deploying with git, or just downloading and using it?

Since there isn't really much in the way of a test environment for Live Events... I develop LED locally on my desktop, update the database, deploy to production, test, evaluate and then update the repository with new commits. I try to make those commits small and focused on single purpose.

From a secondary desktop, when I want to update code and pull changes down from the repo I run

git pull origin master

which fetches the newest source from the LEDbelly Github repo.

I'm a git novice at this point, so I am learning more to support this project, including branching and merging. If you've made local changes and need to pull new versions from the repo, I suspect you'll have to do some git merging, which I have not tried yet.

--

Since LED is basically a catch all and identify what isn't defined, the logging is setup to provide insight into what data is being lost, see the ledbelly/README.md logging section for more details.

I typically review the logs every few days and make adjustments and commit them regularly for others to get the updates. Commit Activity · ccsd/ledbelly · GitHub Without updates, we miss data. The issue is without additional contributors what's defined is based on what my K12 district sees and the scenarios based on our usage of Canvas. For instance, I see plagiarism_resubmit a few times a semester, so I can completely miss changes to that message stream until I see it...whereas higher ed might see this more often.

My workflow for updates is basically to review the log files, and adjust the DDL if I find lengths longer than what's defined, then I update the database tables and push the code to production. If I find new events I evaluate them, create the DDL and set the parser to handle it, generate the table and push to production and commit my changes to GitHub when it's working.

--

I have some rake tasks that I've been working on to improve this process and am pushing to get those released soon now that I know others are using LED. The rake tasks reduce the logs to identify changes and my hope is this will improve the effort necessary to update local code, test and hopefully make it easier for contributors push changes to be merged. These updates will come with changes to the logging formats. Essentially:

DDL Truncation, will be reduced to the new size of a column that needs to be updated.

DDL Undefined, new fields found in events will be reduced to what needs to be added.

Then a comparison of create_tables since last run will produce an alter table script.

I started finishing these up yesterday when I saw your post, my hope is to have them available next week.

I had written a rake task that was supposed to be a subscription updater, that loaded the schema files between commit versions to produce a diff, but it failed to account for local changes, and I had to eval() the hash from the repo which isn't safe.

I'm also working on moving some documentation off the README and into the Wiki· ccsd/ledbelly · GitHub, to help provide broader support.

[luo_dongchuan]

Hi Robert,

I used the "git clone" to get it. If you change some events and I run the "git merge" to get the new version, but I still need to run the sql scripts in /sql/ddl directory manually from my MySQL to reflect those changes into my database, right?  You updates will be end in those sql scripts. If this is true that means all data in database will be lost because those scripts will first drop off old table, then create a new one. I'm not sure whether my understanding is correct?

Thanks a lot.

[robotcars]

You're correct running the create_tables scripts would reset your database. After I make changes to the code, I have to run a series of ALTER TABLE statements, or manually edit the table in a DB IDE. Here's what I'm working on to make this easier.

edit:

Tasks · ccsd/ledbelly Wiki · GitHub 

[luo_dongchuan]

We hope we can keep the historical data for Events. In another word we don't want to delete all data in a table when we apply the update for the changes of a event. If I run the scripts in /sql/ddl directory it will drop all tables(including the tables without any changes)first and then re-create them. We don't want it. We hope we can keep the old data even this tables(event) has been changed(add a new column or change the definition of a column).  

[robotcars]

Completely understandable. Alter Tables is the way to go then, and I will release that soon, hopefully tomorrow. I've just tested the results of the rake task against my database.  I need to do a few more tests from a separate machine to ensure the workflow is nice.

[robotcars]

Coming soon, add live events for outcomes editing · instructure/canvas-lms@4b9fbc7 · GitHub 

[robotcars]

resolved, addition of body {user_login, user_sis_id} to user_created|updated · Issue #3 · ccsd/ledbelly · GitH... 

[robotcars]

Issue

Live Event learning_outcome_result_created.mastery passes wrong value · Issue #1526 · instructure/ca...

Affects

ledbelly/canvas.rb at master · ccsd/ledbelly · GitHub 

[robotcars]

The fix for this is in, and will be deployed to production 12/4. I will patch LED in advance.

[robotcars]

learning_outcome_result_*.mastery boolean, not int, canvas-lms issue#… · ccsd/ledbelly@c18e731 · Git...

ALTER TABLE live_learning_outcome_result_created ALTER COLUMN mastery VARCHAR(5);
ALTER TABLE live_learning_outcome_result_updated ALTER COLUMN mastery VARCHAR(5);

[robotcars]

I have been working on a rake task to generate documentation from the LED schemas. Essentially it will generate the descriptions from the LED schemas in an upcoming commit. I have been working to move descriptions and/or identify the field descriptions from other Canvas documentation and sampling our data and tests. The rake task will eventually generate local markdown of the schema with sample data of your local instance. Here is a sample with de-identified data... if anyone wants to help me describe some of the missing fields I need anything where 'description' is the only value under the Description column.

canvas live events alpha format schema with descriptions and sample values · GitHub 

An issue with this documentation is it is generated from our data... so I've eliminated SIS ids, and many of the Title and Name values, as they are institution specific, and the enumerables are collected from records we have stored since July, so some may be missing based on event dates or our own specific usage of Canvas.

I'm hoping we can eventually replace or update canvas-lms/live_events.md at master · instructure/canvas-lms · GitHub 

[robotcars]

Quizzes Events Deprecations · Issue #4 · ccsd/ledbelly · GitHub 

[robotcars]

After pgo586‌ informed me that I was missing some fields in the Caliper events, I finally added some logging for this format. As with the Canvas Alpha format, any message received that has fields sent that are not defined in the schema will be logged. These can be reduced to update the schema.

caliper support for missing message fields, logging and reducer · ccsd/ledbelly@cfd4200 · GitHub 

schema update, missing fields, see notes for sql · ccsd/ledbelly@54d04d2 · GitHub 

LEDbelly Caliper Schema Update · GitHub 

[robotcars]

The master branch for this repo has been renamed to main.