Inspiring by @robotcars 's solution to export SQS (Live Events) to SQL (LEDbelly), I'm working on a similar concept.
However, by using Fluentd (open-source), we can eliminate the need for Amazon SQS. Benefits of using Fluentd (What's is Fluentd?) are:
It's FREE and fast
Flexible and Extensible (Lots of plug-ins for input/output data - E.g. you can set up an email notification flow based on data received)
Self-host, thus you have total control over data being sent from Canvas
It can process a large amount of data with minimal system resources (E.g. 30-40MB of memory to process 13,000 events/second/core)
Ability to route live data from different Canvas's accounts or sub-accounts to different output
and maybe more...
Linux server (or Docker instance) that is accessible via the internet
SSL certificate (free via Let's Encrypt)
A. Set up Database and Fluentd
1. For ease of setup or transition and future updates, I aligned all database schema with LEDbelly's. You can follow the instruction on LEDbelly's GitHub to set up your database (https://github.com/ccsd/ledbelly/wiki/Getting-Started - you will only need to edit the database config file and follow up to step 5)
2. Next, we need to install Fluentd. I use Fluentd-UI to set up Fluentd as it's easier to view logs and config (Alternatively, you can set up Fluentd as a standalone). Enter the following commands to your terminal
You can now access Fluentd-UI via your web browser at http://your_server_ip:9292/ .The default account is username="admin" and password="changeme".
3. Once login, click "Install Fluentd". Fluentd and its config file will be located in /your_linux_user/.fluentd-ui/
4. Install plug-ins
#SQL input plugin for Fluentd event collector
$ fluent-gem install fluent-plugin-sql --no-document
$ fluent-gem install pg --no-document # for postgresql
#fluent plugin to rewrite tag filter
fluent-gem install fluent-plugin-rewrite-tag-filter
5. Open fluentd-ui folder and pull config files from my repo
git clone https://github.com/jerryngm/fluentd-canvas-live-events-to-sql .
(Please note the period "." at the end of git clone command, this will pull files to the base of fluentd-ui folder)
Then remove the default fluent.conf file
6. Edit fluent.conf.example and save as fluent.conf.
Look for this tag '#your_config_here' and the tag next to it to change the settings accordingly:
#http_port - endpoint port to receive JSON live data from Canvas (If you have a firewall on your server, you will need to open this port)
#ssl_cert - enter the path to your domain's SSL (SSL cert can be self-signed or obtain for free from Let's Encrypt - How's To)
#database_config - config your database here
7. Open Fluentd-UI again and press "Restart"
B. Set Up Canvas Data Service
1. Open "Data Services" from your account/or sub-account Admin page