Bulk or Batch Updating URLs included throughout Canvas

[rsauls]

I am looking to batch update 1000s of URLs throughout Canvas, but I am not finding a process for this type of update.  Any advice?  The URL's may have been added in Canvas in a variety of areas and in a variety of ways.  Needing to find and replace pattern within the URL- i.e wanting to change abc.com/example to xyz.com/example.

[robotcars · Accepted solution]

Hi,

I do not have any code to share on this at the moment, and each use case depends on what you have to target. I can provide some guidance for the most efficient way I've figured out how to do this. Since there is no 'content search' in Canvas, the best place to get a current state of key content and descriptions of content in Canvas is through Canvas Data.

If you have Wiki Pages to update, you can consume https://portal.inshosteddata.com/docs#wiki_page_dim into a Db. Do a query for your string, which would return you the course and the wiki page id.

In your code, you would grab that content, replace the necessary string/url, then update the page with the API and repeat for all items found. https://canvas.instructure.com/doc/api/pages.html#method.wiki_pages_api.update

This can be repeated for Quiz, Discussions, Assignments

[dgrobani · Accepted solution]

I've written a Python script to do this for Canvas pages. It gets each course in one or more subaccounts, gets each wiki page in the course, does a search and replace on the page content, and if the content changes, writes the page content back.

Unfortunately, you'd need to have experience running Python scripts against the Canvas API for it to help you.

[robotcars · Accepted solution]

Hi,

I do not have any code to share on this at the moment, and each use case depends on what you have to target. I can provide some guidance for the most efficient way I've figured out how to do this. Since there is no 'content search' in Canvas, the best place to get a current state of key content and descriptions of content in Canvas is through Canvas Data.

If you have Wiki Pages to update, you can consume https://portal.inshosteddata.com/docs#wiki_page_dim into a Db. Do a query for your string, which would return you the course and the wiki page id.

In your code, you would grab that content, replace the necessary string/url, then update the page with the API and repeat for all items found. https://canvas.instructure.com/doc/api/pages.html#method.wiki_pages_api.update

This can be repeated for Quiz, Discussions, Assignments

[dgrobani · Accepted solution]

I've written a Python script to do this for Canvas pages. It gets each course in one or more subaccounts, gets each wiki page in the course, does a search and replace on the page content, and if the content changes, writes the page content back.

Unfortunately, you'd need to have experience running Python scripts against the Canvas API for it to help you.

[rsauls]

Can you please share the script?

[dgrobani]

I have to do some work on it to make it shareable. I should be able to post it tomorrow.

[dgrobani · Accepted solution]

OK, I've gathered into one file all my helper functions scattered among multiple modules, expanded the options for specifying the desired accounts and terms, tested it a bunch to make sure it still works, and uploaded it to my GitHub repo: py3-canvaslms-api/page_text_replace.py at master · dgrobani/py3-canvaslms-api.

After the import lines, there are seven parameters to specify, hopefully self-explanatory.

A reminder that this searches only wiki pages, not discussions, etc.

Do let me know if you have any questions. And if you're able to use it, please let me know how it goes. In any event, good luck!

[Pete5484]

@dgrobani can I just extend a huge thankyou for your help and contribution to this topic, over 3 years ago!

About a month ago, we were informed that a site that our courses link to (extensively) would very shortly become unavailable. The scale/impact of this was enormous.

A CD2 query identified that there were more than 13,000 links spanning thousands of course pages (including still active course copies, not blueprints). Each of these links required updating.

We ended up with a team doing manual updates, while simultaneously working on a modified version of your python program. The latter ended up being part of a multi-stage process that involved using CD2 output to scrape the distinct links, it then searched for and generated replacement urls (which were subsequently scraped to confirm they were valid/correct).

We modified your program to read in the thousands of rows containing course_id, page_url, original_link, and replacement_link as a dataframe grouped by course_id and page_url. This allowed iterating over each row in the group to replace every link on the page before calling update_course_page_contents. The dataframe was then written back to a new file with extensive logging to record the status of each link and the overall page. There's been extensive testing, including testing rollback plans.

On production, the bulk link replacement completed in less than 13 minutes.

2024-09-14 02:29:02,249 - INFO - Finished: updated 7176 links in 744.24 seconds (0:12:24.238409)

While we still have work to do (e.g. legacy SCORM and item banks in New Quizzes that can only be updated in the UI sigh), your contribution provided us with a huge head start on bulk updating links in page content.

Thank you!

[wojcika]

@Pete5484 would you be willing to share information about how you queried CD2 for the data? We are in a similar situation where we need to report back all of the courses that contain a specific URL. I haven't found a great solution for this yet. 

[Pete5484]

@wojcika absolutely!

Note that we weren't using blueprints at the time. I've subsequently updated the 'pages' script below to highlight blueprint, child and regular courses (but I haven't updated the assignment/syllabus pages scripts). So, I would now only want blueprint and 'regular' courses to be updated, because trying to unravel blueprint child course changes would be... messy! But you'd still want to scan the child courses in case they've been 'unlinked' from blueprint updates in the child!

Postgres query below; [replace institution_name] with your instructure instance; the regex is matching anything that contains 'instructure' in the sample below. Note it's using DISTINCT for pages, so there could be more than one matching link on the page (but the search/replace process replaced all references, otherwise you could count number of instances).

1. Extract  regex pattern match from 'pages'

SELECT DISTINCT
  wp.context_id AS course_id,
  c.sis_source_id as course_code,
  w.title AS course_name,
  wp.title AS page,
  wp.wiki_id,
  wp.url,
  CONCAT('https://[institution_name].instructure.com/courses/', wp.context_id, '/pages/', wp.url) AS hyperlink,
  unnest(regexp_matches(wp.body, '(?i)(<a[^>]*href="https://[^"]*instructure[^"]*"[^>]*>[^<]*<\/a>)', 'g')) AS full_anchor_tag, 
  -- unnest(regexp_matches(wp.body, '(?i)(<a[^>]*href="([^"]*instructure[^"]*)"[^>]*>.*?<\\/a>)', 'g')) AS full_anchor_tag, --extracts tags
  -- unnest(regexp_matches(wp.body, 'href="([^"]*instructure[^"]*)"', 'g')) AS href,	
  length(wp.body) AS body_length,
  wp.workflow_state
, case when m.course_id is not null then 'blueprint' 
       when m2.child_course_id is not null then 'child' 
       else 'regular' 
  end as course_type
, m.course_id as master_course_id
, m2.child_course_id as child_course_id
FROM canvas.wiki_pages wp
JOIN canvas.wikis w ON wp.wiki_id = w.id
JOIN canvas.courses c ON c.wiki_id = wp.wiki_id
LEFT JOIN canvas.master_courses_master_templates m ON c.id = m.course_id
LEFT JOIN canvas.master_courses_child_subscriptions m2 ON c.id = m2.child_course_id  
WHERE 1=1
  AND wp.workflow_state <> 'deleted'
  AND c.workflow_state NOT IN ('deleted')

2. Get hyperlinks from Assignment pages (not updated to handle blueprints)

SELECT 
  a.context_id AS course_id,
  a.id as assignment_id,
  c.sis_source_id AS sis_id,
  a.workflow_state,
  c.name as course_name,
  a.title,
  unnest(regexp_matches(a.description, 'href="([^"]*instructure[^"]*)"', 'g')) AS href,
 length(a.description) as body_length,
 a.description as body
FROM canvas.assignments a
  JOIN canvas.courses c on a.context_id = c.id
WHERE a.workflow_state = 'published'
ORDER by course_id, assignment_id, description;

3. Syllabus page

SELECT DISTINCT
  c.id,
  c.name,
  c.course_code,
  c.workflow_state as course_workflow_state,
  CONCAT('https://taxinstitute.instructure.com/courses/', c.id, '/syllabus') AS hyperlink,
  unnest(regexp_matches(wp.body, 'href="([^"]*instructure[^"]*)"', 'g')) AS match,
-- unnest(regexp_matches(c.syllabus_body, '([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})', 'g')) AS match,
  length(c.syllabus_body) as syllabus_body_length
FROM canvas.courses c 
WHERE 1=1
   AND c.workflow_state not in ('deleted')
-- AND c.syllabus_body LIKE '%mailto:%'
ORDER by c.id;