UDOIT 3.0.0 "Howey-in-the-Hills"

[tr_jbates]

This is the first official open source release of the "Next Generation" UDOIT! It has been completely rewritten from the ground up to make it easier and faster to use. It's also more extensible, so we can more easily add new accessibility checks and other features. Cidi Labs put together a great overview video of UDOIT 3.0.0 that covers all the new features and user interface.

To read the full release notes and participate in the discussion, please visit the UDOIT 3.0.0 Release Page.

If you would like to become part of the user and developer community, please join the UCF Open Source Slack organization.

[bjork]

Hello,

I'm trying to install the non-dockerized version 3.0.0 of UDOIT on RHEL8 (LEMP stack) and have run into an issue with the symfony framework. All the pre-reqs for the app install have been met, but when trying to set up the database with...

symfony console doctrine:database:create --if-not-exists

 ...I get the following error:



It would appear that symfony does not pick up the environment variables set in .env.local. It's my first time working with the framework, so if this error is something trivial, I apologize. My redacted .env.local is below:

###> symfony/framework-bundle ###
APP_ENV=dev

###> doctrine/doctrine-bundle ###
DATABASE_URL=mysql://testudoituser:XXXXXXX@127.0.0.1:3306/testudoit3?serverVersion=mariadb-10.3.28

###> base url ###
# Base URL for client callbacks
BASE_URL="https://testudoit3.college.edu"
###> base url ###

APP_LMS=canvas

Thank you!

[bjork]

Hello again,

I figured out the error.  After running...

symfony console debug:container --env-vars

... I got a listing of variables available to the container, and they match up with what's in .env.local. In other words, the symfony variable injection worked just fine. The problem turned out to be that I forgot to escape special characters in the db password (as part of the DATABASE_URL param). Rookie mistake. Once that was taken care of, the doctrine script ran just fine.

[Landmark_Coll]

Looking for help installing UDOIT 3 on Ubuntu. I posted a plea for help at https://github.com/ucfopen/UDOIT/discussions/707 but it looks like my Github account is locked so I don't think anyone can see the post.

Here's my post:

Hello,
I'm hoping this is an appropriate place for this post. I'm running into difficulties upgrading to version 3 of UDOIT. I had been running an earlier version on the free Heroku platform. I attempted a new Heroku installation which seems to work on courses with fewer pages, but scans endlessly on larger courses. Even on smaller courses, about 30% of the attempted fixes return errors.

I figured that the free Heroku plan might not have the necessary horsepower to run 3.0. So I'm attempting to follow the documentation to install UDOIT locally on a Linux server. I'm finding INSTALL.md to be a little vague with regard to what's optional and what's required. For example, it says, "If you are setting up a local UDOIT instance through Docker..." But it doesn't say if there are alternatives to using Docker. In any event, I think I'm getting hung up at this command:

docker-compose up --build

It returns a series of errors saying "Connection aborted" and "Permission Denied."

If I run the command with sudo, I get a bit further but there are lots of warnings, and ultimately, the next command, docker exec -it udoit_php_1 /bin/bash, fails with more permission errors 'while trying to connect to the Docker daemon socket.'

Some guidance here would be tremendously appreciated. If this is an inappropriate post, kindly redirect me.
Thanks.

Michael Keen
Landmark College

[tr_jbates]

@Landmark_Coll  I tried this out on my own Ubuntu machine, and I found that I needed to use sudo with every docker call.  There are some warnings when I run

sudo docker-compose up --build

, but it ultimately works.  

I also had to use sudo when running the next command

`sudo docker exec -it udoit_php_1 /bin/bash`

[Landmark_Coll]

@tr_jbates Thank you for the quick reply. You got me a lot further along in the process by using sudo pretty much everywhere.

Unfortunately, the final test is still failing. The documentation say to navigate to 

<BASE_URL>/lti/config

This results in a 404/Not found from Apache. On the bright side, simply going to the BASE_URL gives my this...

which at the very least is coming from somewhere other than Apache.

If I look at the file system, there is no "lti" subdirectory under public, so I'm not too surprised I get a 404.

Also, I edited the .env.local. file accordingly, but the institution table in the MySQL database is empty. Does that seem right?

I feel like maybe I'm getting close. Any more ideas?

[tr_jbates]

@Landmark_Coll The /lti/config path is defined in the LTIController.php file.  The app uses Symphony, so paths like that are handled by Routes.  The Institution table will be empty by default.  You have to set it up by following the instructions in INSTALL_CANVAS.md.  However, if the /lti/config path is returning a 404, you won't be able to follow those instructions, so we need to figure that out first.  Are you using your own Apache install?  I don't have much experience with Apache, but you might try a ProxyPass to udoit_php_1:8000.  Basically, Apache needs to pass the whole request to the PHP container and let Symphony figure out the routing.

[Landmark_Coll]

@tr_jbates Thanks again for working this out with me.

So, I've added what I think are the necessary modules to Apache for this to work:

I found documentation on ProxyPass here.

I'm not understanding what you mean with: "you might try a ProxyPass to udoit_php_1:8000." The documentation appears to want the destination to be a full URL. I don't really get how to configure the conf file based on this.

[tr_jbates]

@Landmark_Coll You could try https://127.0.0.1:8000 .  Again, I don't have much experience with Apache, so we're already out of my depth. In NGINX, we use fastcgi_pass to redirect traffic to a Docker container.

[Landmark_Coll]

@tr_jbates I'm so close, I can almost taste it.

Using ProxyPass, /lti/config now gives me this:

{"title":"UDOIT 3","scopes":[],"extensions":[{"domain":"bullshark.landmark.edu\/udoit\/public","tool_id":"cidilabs_udoit3","platform":"canvas.instructure.com","settings":{"text":"UDOIT 3","platform":"canvas.instructure.com","placements":[{"text":"UDOIT 3","placement":"course_navigation","message_type":"LtiResourceLinkRequest","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/public\/dashboard","visibility":"admins","enabled":true,"default":"disabled"},{"text":"UDOIT 3 Admin","placement":"account_navigation","message_type":"LtiResourceLinkRequest","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/public\/admin","enabled":true}]},"privacy_level":"public"}],"public_jwk":[],"description":"User settings for UDOIT 3.x","public_jwk_url":"https:\/\/canvas.instructure.com\/api\/lti\/security\/jwks","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/public\/dashboard","oidc_initiation_url":"https:\/\/bullshark.landmark.edu\/udoit\/public\/lti\/authorize","custom_fields":{"lms_id":"canvas","lms_user_id":"$Canvas.user.id","lms_course_id":"$Canvas.course.id","lms_account_id":"$Canvas.account.id","lms_api_domain":"$Canvas.api.domain"}}

 Moving on to the Canvas install, I first notice a problem with the API logo:

The location should be https://bullshark.landmark.edu/udoit/build/static/udoit_logo.png. And then when actually trying to run the tool in a course, I get this:

Any idea where I can look next?

[tr_jbates]

@Landmark_Coll I think your LTI Key is pointing to the wrong URL.  Instead of https://bullshark.landmark.edu/udoit/public/dashboard, it should be https://bullshark.landmark.edu/udoit/dashboard.  The /lti/config file appears to be generating the various links incorrectly.  Did you add "/public" to the .env?  If so, try removing it to see if the /lti/config generates correctly.  You might have to manually fix all the URLs in your LTI Key in Canvas.

[Landmark_Coll]

@tr_jbates You were absolutely correct. I mistakenly had "public" in .env.local.  I've removed that and recreated the LTI key in Canvas.

Unfortunately, still no joy.

Here's the latest /lti/config:

{"title":"UDOIT 3","scopes":[],"extensions":[{"domain":"bullshark.landmark.edu\/udoit","tool_id":"cidilabs_udoit3","platform":"canvas.instructure.com","settings":{"text":"UDOIT 3","platform":"canvas.instructure.com","placements":[{"text":"UDOIT 3","placement":"course_navigation","message_type":"LtiResourceLinkRequest","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/dashboard","visibility":"admins","enabled":true,"default":"disabled"},{"text":"UDOIT 3 Admin","placement":"account_navigation","message_type":"LtiResourceLinkRequest","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/admin","enabled":true}]},"privacy_level":"public"}],"public_jwk":[],"description":"User settings for UDOIT 3.x","public_jwk_url":"https:\/\/canvas.instructure.com\/api\/lti\/security\/jwks","target_link_uri":"https:\/\/bullshark.landmark.edu\/udoit\/dashboard","oidc_initiation_url":"https:\/\/bullshark.landmark.edu\/udoit\/lti\/authorize","custom_fields":{"lms_id":"canvas","lms_user_id":"$Canvas.user.id","lms_course_id":"$Canvas.course.id","lms_account_id":"$Canvas.account.id","lms_api_domain":"$Canvas.api.domain"}}

This URL is still not finding the logo file:

https://bullshark.landmark.edu/udoit/build/static/udoit_logo.png

And I've upgraded the 404 error to a 500 error now. 😁

[tr_jbates]

@Landmark_Coll We're getting close!  Please make sure that you've run the database migrations and set up the Institutions table.  Most of our 500 errors are due to the database not being configured properly.

[bjork]

Hi @tr_jbates,

I've worked through the app install, but have run into some issues with the integration I was hoping you could assist with. When clicking on the "UDOIT 3 Admin" link in Canvas, I get the following error page:

The above error seems to be related to the LTI key id... What is a bit perplexing to me is that with earlier versions of UDOIT (<3.0.0) both the API and LTI keys/secrets were defined in the app configuration. In version 3.0.0+, only the API key/secret is defined, in the "institutions" table. Our institutions table contains values in all columns, matching up with the data (API key/secret, etc) in Canvas. The .env.local contains APP_ENV, DATABASE_URL, BASE_URL and APP_LMS variables only.

Any assistance would be greatly appreciated. Thanks!

[tr_jbates]

@bjork I've never encountered that error before.  The difference in configuration you noticed is because of the switch from LTI 1.1 to LTI 1.3.  In UDOIT <3.0.0, the Developer API Key was stored in the app configuration file, and a couple of made-up values (the consumer key and shared secret) were also placed there.  When you added UDOIT to a course, you also supplied the consumer key and shared secret you made up.

With UDOIT >=3.0.0, we still use a Developer API Key, but now we use a Developer LTI Key instead of the consumer key and shared secret.  The LTI Key is not manually placed anywhere in UDOIT because the configuration happens automatically when the LTI Key is created in Canvas.  Check over the process for creating an LTI Key for UDOIT again to make sure it's set up correctly.

[bjork]

@tr_jbates I was able to work through the error above by re-doing the Canvas integration. However, when clicking on "UDOIT 3 Admin" in Canvas I now get:

{"status":"bad_request","message":"Invalid redirect_uri"}

The redirect URIs set up for the keys are (with updated domain name): 
https://testudoit3.college.edu/authorize/check  (API key)
https://testudoit3.college.edu/lti/authorize/check (LTI key)

... and I checked that those are both valid app routes. Note that "testudoit3.college.edu" is not publicly available -- having the app domain public was not needed in version 2.8.x so I assume it is not needed for 3.x either.

The var/log/dev.log have the following corresponding entries (tweaked for security purposes):

[2021-11-30T17:34:24.231722+00:00] request.INFO: Matched route "lti_authorize". {"route":"lti_authorize","route_parameters":{"_route":"lti_authorize","_controller":"App\\Controller\\LtiController::ltiAuthorize"},"request_uri":"https://testudoit3.college.edu/lti/authorize","method":"POST"} []
[2021-11-30T17:34:24.232784+00:00] security.INFO: Populated the TokenStorage with an anonymous Token. [] []
[2021-11-30T17:34:24.239081+00:00] doctrine.DEBUG: "START TRANSACTION" [] []
[2021-11-30T17:34:24.239460+00:00] doctrine.DEBUG: INSERT INTO user_session (uuid, data, created) VALUES (?, ?, ?) {"1":"5efbee56-e89b-4feb-90d1-24 [...]","2":{"iss":"https://canvas.test.instru [...]","login_hint":"f84f8014f1a3A9c395cd2K2774 [...]","client_id":"23630000000000172","target_link_uri":"https://testudoit3.college [...]","lti_message_hint":"eyJ0eXQiOiJKt1QiLCJhbGciOi [...]","canvas_region":"us-east-1","lms_api_domain":"canvas.test.instructure.com"},"3":"2021-11-30T17:34:24+00:00"} []
[2021-11-30T17:34:24.239830+00:00] doctrine.DEBUG: "COMMIT" [] []​

Any ideas on what to check next? I feel like we're getting pretty close... Thanks in advance!

[tr_jbates]

@bjork Sorry for the delayed response; this week has been hectic.  Make sure "BASE_URL" in .env.local is set to the URL of your instance of UDOIT.  This is dependent on your web server configuration, but judging by your post it should be "https://testudoit3.college.edu".

[Landmark_Coll]

@tr_jbates Hi Jacob,

I figured I'd revisit and report back. You may recall I reverted to v2.8.3 on Heroku when we were unable to make v3.0 work on a local server which has an active production MySQL server running and v3.0 timed out on many of our courses when installed on Heroku.

I decided to try a fresh install of v3.0 on Heroku today. Unfortunately, it still cannot handle our courses with a large number of pages. Here's a snip from the log...

2022-03-21T17:57:14.009516+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/api/sync/3" host=udoit-landmark.herokuapp.com request_id=26a78e4b-ead5-400e-bd18-cdce4e90a0b7 fwd="198.54.211.2" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https

2022-03-21T17:57:31.345671+00:00 app[web.1]: [21-Mar-2022 17:57:31] WARNING: [pool www] child 113, script '/app/public/index.php' (request: "GET /index.php") execution timed out (30.831726 sec), terminating

---

Please let me know if this is likely to be resolved for Heroku. If not, I might try building a brand new local server from scratch.

Thanks.

[tr_jbates]

@Landmark_Coll Thanks for checking back in.  The fix for this issue requires extensive changes to the scanning process, and is covered in this issue:  Create worker for scanning

That feature will require extensive changes to the scanning workflow.  We haven't started work on it yet because we've been focusing on streamlining the server install process.  I will be releasing 3.2.0 on Friday of this week, and 3.3.0 should be coming at the end of April.  3.3.0 will contain many improvements to the server install process, so it should be easier for you to set up a brand new local server from scratch.  Creating a worker for scanning will be a high priority after 3.3.0 is out.

[Landmark_Coll]

Thanks so much for the quick response. I'll work with my IT Dept. to get a new server spun up soon.

[tr_jbates]

@Landmark_Coll We just released UDOIT 3.2.0