[ARCHIVED] "UDOIT FAILED TO SCAN THIS COURSE"

[kevinw]

Hi all, 

I posted this question elsewhere but thought perhaps it could have it's own thread. If I'm wrong go ahead and delete me. Anyways, I recently watched Canvas' accessibility video and was thrilled to learn about UDOIT. We have been using Sitemorse's snapshot which is unable to run in java so we were having to copy/paste HTML code and it was just a pain. 

However after going through and successfully deploying UDOIT to heroku I consistently get the same error: UDOIT failed to scan this course. 

Has anyone else experienced anything like this? I'm unsure of where to go from here. It's really disappointing because there's just nothing else like this in Canvas and it sees like it is an incredibly powerful tool. 

[rgibson1]

We're getting this error in one of the courses as well. I've narrowed it down to some combination of the Assignments and Discussions. If both are off, the course scans. If either are on, the scan fails.

[santhoshreddy]

Good Afternoon, Jacob.

I have installed UDOIT 2.3.5 on an ec2-instance and it is working perfectly fine. Ec2-instance is running beyond the load balancer, initially, we got trouble. When I changed the line (http to https) in the OAuth.php file, then it started working without any issues. 

I also configured the same setup inside the container(AWS-ECS), When I try to scan the course, Scanning is continuously spinning.  I also make sure that  "job_queue" table and the "results" field is of type "mediumtext".  Could you please help me with this issue? 

[tr_jbates]

I apologize for taking so long to reply.  I have been out on vacation and am just now catching up.

I have a few questions regarding the issue you're seeing:

• Are jobs getting created in the job_queue table?
• Are there any errors showing up in the PHP error logs?  If so, what are they?
• Does an API key get generated for your user?  If so, please test it out using cURL, Postman, or some other API testing tool.

[santhoshreddy]

Good Morning, Jacob,

Sorry for the delayed response. I am in travel I didn't get a chance to reply to you immediately. 

I had deployed the containerized UDOIT application into AWS-ECS. Initially, I got a scanning error, which I mentioned in the above email. Later I realized that there is a permission issue inside the UDOIT folder. When I corrected it, then it started working usually. 

Thank you, for your help!

[tr_jbates]

That's great to hear!  Which folder had the permission issue?

[gregvogl]

Hello Jacob,

UDOIT 2.3.4 has worked fine in many courses in our test instance of Canvas since I installed it on July 26. It takes about 10 seconds to scan a practice course.

But since I installed it in our production instance yesterday, it hasn't worked. It hangs during scans of this same practice course in production, a new blank course, and a few other courses I tried.
Scanning: module_urls, syllabus, pages, files, discussions, assignments, announcements... (7 left to scan)

If I scanned only one type of content, it still hung (regardless of which type I select):
Scanning: announcements... (1 left to scan)

When I reinstalled, it showed:
Scanning: files... (1 left to scan)

and showed 0 errors if I only scanned the other 6 content types, even though it had showed 72 errors, 8 suggestions and 1 unscannable Word file in Canvas test.

udoit.xml.php correctly displays XML in a browser.

I see no related errors in the Apache access logs; all response codes are 200 or 302. (I have no access to Apache error logs due to the shared server configuration.)
The users.canvas_url field in the database shows the production Canvas URL, with entries created today.
But the last entries in the job_queue and reports tables are dated November 19.

The changes I made to implement in production:

• Removed the LTI from Canvas test.
• Added a developer key to Canvas production.
• Copied the OAUTH2 ID and key to $oauth_id and $oauth_key in localConfig.php
  (When I first tried to install in production, I forgot to change the OAUTH ID, but this is fixed.)
• Added the LTI to Canvas production using the same consumer key and shared secret.

Other things I tried:

• Changing the UDOIT_ENV variable to ENV_PROD didn't help.
• Changing it to ENV_DEV gives an error "UDOIT failed to scan this course."
• Commenting out the GOOGLE_API_KEY and VIMEO_API_KEY didn't help.
• Deleting all rows in all database tables returned to hanging on "7 left to scan".
• Restoring the database and running update job_queue set status='expired' where status='running'; reverts the behavior to hanging on scanning files.
• I also tried installing version 2.3.5 on Canvas test, via both git clone and zip file, but in both cases the LTI can't be installed because udoit.xml.php gives an internal server error in the browser. (The logs show a 404 error, but the file exists and is readable in Linux, as is settings.php.)

Thanks in advance for any suggestions.

[tr_jbates]

I'll focus on the issues you're having with 2.3.4 first.  Have you tested the API key generated for your user to make sure it works properly?  You might also want to double-check your $oauth_id and $oauth_key to make sure they match the ones in Canvas (looking for spaces at the beginning and end, or an incomplete key).  Also check the $oauth_url to make sure it matches your Production instance of Canvas.

You might also want to verify your database connection info in localConfig.php, since it's not creating new entries in the job_queue.

If you can get access to the PHP error logs for the server, that would be a great help.

Let me know the results of the above tests, and we can work from there.

[gregvogl]

Jacob,

Thanks for your suggestions.

I reverted back to version 2.3.4 and uninstalled and reinstalled the Canvas LTI in both test and production a few more times.

• In Canvas test UDOIT it always works. My test course displays "0 of 0 have issues" in Files, 0 of 24 in Module URLs, 3 of 10 in pages, and 0 of at least 1 in the other content types.
• In Canvas production it never works. "0 of 1 have issues" is displayed for everything except Module URLs which displays "UDOIT failed to scan this course", and Files which displays "Scanning: files... (1 left to scan)".

Things I checked again:

• $oauth_id and $oauth_key and $oauth_uri in localConfig.php exactly match the developer key values from Canvas production.
• The Canvas production LTI uses the correct consumer key, shared secret and config URL.
• The database connection is correct, and I checked again that database entries are written to the job_queue table for both Canvas test and production.
• The last 300 Apache error logs in cPanel only contain warnings about udoit.xml.php beng group writeable, which I had fixed.
• The developer key is in the On state and shows: 
  Access Token Count: 42
  Created: Dec 3 at 4:31pm
  Last Used: Never

[tr_jbates]

The "Last Used: Never" is concerning.  I checked our own Dev Key for UDOIT, and it has a date in the "Last Used" field.  Did you get a chance to test out an API key it generates to make sure it works on your production Canvas instance?

Also, please make sure your $UDOIT_ENV variable is set to ENV_PROD.

Do the PHP error logs say anything?

[gregvogl]

Another difference between test and prod is that in production, I am more often asked to authorize when I click the UDOIT link to reload the UDOIT page, and a new row is created in the users table.

$UDOIT_ENV is set to ENV_PROD by default, but it didn't help to explicitly set it in localConfig.php.

There are no errors in the Apache logs.
For both test and production, there are lines like these, which I understand:
"POST /udoit_prod/public/ HTTP/1.1" 302 - "https://colostate.instructure.com/courses/680/external_tools/4153"
"GET /udoit_prod/public/oauth2response.php?code=7cb92580b... HTTP:/1.1" 302 - "https://colostate.instructure.com/login/oauth2/confirm"
"GET /udoit_prod/public/scanner.php HTTP/1.1" 200
"POST /udoit_prod/public/process.php HTTP/1.1" 200
"GET /udoit_prod/public/progress.php?job_group=job_5c09b29... HTTP/1.1" 200 (one progress check per second)
For test, when the report is completed, there is a line like this:
"GET /udoit/public/parseResults.php?report_id=108 HTTP/1.1" 200

(I have configured separate test and production instances in /udoit/ and /udoit_prod/ so I can test them both at once.)

I don't know the exact steps to test an API key to make sure it works on production. I am familiar with making API calls through both Postman and the Canvas Live API using an admin access token, but don't know how to generate an access token from the api_key field in the users table of the UDOIT database.

[tr_jbates]

To test one of the generated API keys, just authorize UDOIT with your own account, then find your user's line in the `users` database table.  Copy the data from the api_key field and use it exactly as you would a normal api key in Postman or Canvas Live API.  Make sure you take a key from a user that has just used UDOIT; the keys expire after an hour.  If it doesn't work, there's some issue with your Developer Key.

Other than that, I'm not sure what else to have you do.  It feels like there's something that got missed while translating the configuration from test to prod.

[gregvogl]

Jacob,

Sorry for the delayed response; I had to wait for further testing since we don't change production Canvas configuration from finals week until after grade submission.

I tried re-creating the developer key and adding the LTI, but still had the same error.

No entry was created in the user table. So I deleted all data from all database tables and now it works fine.

I was also able to use the api_key as an access token in a Postman GET request.

The upgrade to 2.3.5 also worked.

Thanks for all your help.

[tr_jbates]

I'm glad you got it working!

[santhoshreddy]

Good Evening, Jacob,

I have a small question. I deployed UDOIT in aws, where the default time zone is UTC. I have changed the timezone in the php.ini file to eastern(i.e America/New_York). When I do php.info, I can see the changes got reflected.  But after scanning the course, when I check the View Old Reports tab, the date and time are still displaying in UTC. 

Any suggestions?

In the below image, the time and date are supposed to show in Eastern time. 

I appreciate your help. 

[santhoshreddy]

Would you recommend me to modify the timezone in the database as well? 

[santhoshreddy]

Good Morning, Jacob, I changed the database time zone, then the View Old Reports tab is displaying results in EST.  I am wondering if there is any other way that I can change display time on the application level, rather than changing on database level? 

Thank you.

[tr_jbates]

I was just thinking you should look at the database timezone setting.  I think it's a good idea to set a display timezone for the app.  Maybe it could just work off of whatever PHP is set to rather than having a configuration value for the app?

[santhoshreddy]

After adding the default time zone in the settings.php file, the problem got solved. Now the View Old Reports tab is displaying date and time in EST. 

Thanks, Jacob.

[santhoshreddy]

Also, Developer made some changes to the saved_reports.php file.

[tr_jbates]

Awesome!  I was already looking into the changes to make to that file.  Would your developer be interested in contributing their changes to the project?

[rlzhang]

Sure Jacob. Santhosh passed your message to me.  I just twisted some change for our institution purpose on saved_resports.php to show local time zone, and would be happy to contribute the changes to the project if need.

Feel free to ping me the instruction to checked in the changes.   

Thanks,

Ruiling

[tr_jbates]

If your developer is familiar with Git and GitHub, they can fork our repository and submit a pull request.  Have them follow the contributor's guide on the UDOIT repository for more information.