Export Link Validation

[MarandaMiller]

I have over 150 links that were returned as broken links via the link validator.  I'd like to export the listing into a spreadsheet so that I can track what was fixed.

Is there an option in exporting the list rather than me copying/paste in a spreadsheet or document?

[ryan_corris]

Hey @MarandaMiller!

That is a great question!  I am not aware of a way to export the results and there is nothing mentioned in the Community Guide How do I validate links in a course? that states anything about exporting the results.  If anyone has some other thoughts regarding this, hopefully they will chime in.

[James · Accepted solution]

@MarandaMiller 

What I normally do is work my way through the list and then re-run the validator after I have fixed a bunch of stuff. Since you can click on the title, I normally go through and ctrl-click several of those to open in a new tab and then I close the tab once I've made the changes. Most people will find it easier to work this way than to work from a spreadsheet, especially since those 150 broken links could be across just 10 pages and you'll fix all the items on one page at a time. It goes faster than you might think when seeing the initial 150 warnings.

That said, if you really want a list in a spreadsheet, you can scrape the web page and extract the links.

I quickly threw something together that would generate a tab-separated list of all of the information from the link validator. You can then copy/paste it into a spreadsheet. It wouldn't take much to make a userscript out of this that actually creates a spreadsheet, but I don't have the time to fully debug or support it right now and I haven't seen a lot of demand for something like this.

The script gives you columns for the item (page, assignment, etc.) title, the link to the Canvas item, the type of item, the message, the link that is problematic, and the text of the link. It handles the cases where items have multiple types of messages (thanks go to one of my unnamed colleagues for having a page with 25 broken links).

These directions are for Chrome, but the script should work in all major browsers.

1. Navigate to your link validation page with the results you want to export.
2. Press F12 to open the browser's developer tools. You can also right-click on the page and choose Inspect.
3. Find the Console tab. There is a link at the top for Console, but there may be one at the bottom of other pages such as Elements.
4. Paste the code into the console and hit enter.
5. Select the output and copy/paste into a spreadsheet.

var info = [['title','url','type','message','link','linkUrl']];
document.querySelectorAll('#all-results div.result').forEach(r => {
  const headEl = r.querySelector('h2');
  const headLink = headEl.querySelector('a');
  const typeEl = headEl.parentNode.querySelector('span');
  const messagesEl = r.querySelector('ul');
  const title = headLink.textContent;
  const titleUrl = headLink.href;
  const type = typeEl.textContent;
  for (const m of messagesEl.children) {
    if (!m.nodeName === 'LI') {
      continue;
    }
    const messageFullText = m.textContent;
    const itemsEl = m.querySelector('ul');
    const partialText = itemsEl.textContent;
    const messageText = messageFullText.slice(0,messageFullText.length-partialText.length);
    itemsEl.querySelectorAll('li a').forEach(e => {
      console.log(e);
      const link = e.href;
      const linkText = e.textContent;
      const item = [title, titleUrl, type, messageText, link, linkText];
      info.push(item);
    });
  }
});
console.log(info.map(e => e.join('\t')).join('\n'));

[MarandaMiller]

Thank you all for this information and the code to scrape the web results.  

[James]

@MarandaMiller 

I found an easier way that doesn't require any scripting. There is a link validation API endpoint, but it's not documented so I didn't know about it.

1. Go to your link validator page. Let's assume the URL is like <canvas_instance>/courses/12345/link_validator
2. Modify the URL to put /api/v1 before the courses and change link_validator to link_validation at the end. The URL should look like <canvas_instance>/api/v1/courses/12345/link_validation
3. Hit enter to load the page.
4. You may want to check the box to use pretty-print.
5. Look for the section that says results and issues. After the issues will be a left square bracket.
6. Select the text starting with (and including) the left square bracket and going all the way through the bottom, stopping with (and including) the right square bracket near the bottom right before the "completed_at" line. Do not include the comma after the right square bracket.
7. Copy this information to your clipboard.
8. Use a JSON to CSV converter and export your results to Excel. Two I use are ConvertCSV and Data.page. The ConvertCSV site will give you one row per item with multiple columns when there are multiple issues but it does the conversion in the browser so it doesn't send any information to their site. Data.page will give you one row per issue, with the title and URL listed just once but it sends the information to their server for processing.

I think my script returns the results in a nicer format than either one of them, but you can get the results without coding or using the developer tools.