Relevant Elephants: Fixing Canvas Commons Search

[laurakgibbs]

This idea has been developed and deployed to Canvas

I'm not really sure how to write this as a feature idea because, as far as I am concerned, it is a bug that needs to be fixed. But my dialogue with the Helpdesk went nowhere (Case #03332722), so I am submitting it as a feature request per their advice. I am not proposing how to fix this problem; I am just going to document the problem. People who actually know something about repository search would need to be the ones to propose the best set of search features; I am not an expert in repository search and how to fix a defective search algorithm. But as a user, I can declare that this is a serious problem.

If you search Canvas Commons for elephants, you get 1084 results. Here are the "most relevant" results:

https://lor.instructure.com/search?sortBy=relevance&q=elephant 

There are some elephants, which is good... but a lot of other things that start with ele- ... which is not good. Apparently there is some component in the search algorithm which returns anything (ANYTHING) that matches the first three characters of the search string.

Election Unit Test.

Static Electricity Virtual Lab.

And so on. And so on. Over 1000 false positives. ele-NOT-elephants.

As near as I can tell, there might be a dozen elephants in Canvas Commons. I've found six for sure; there could be more... it's impossible to find them. Impossible because of the way that search works (or doesn't work). The vast -- VAST -- majority of results are for electricity, elections, elearning, elements, elementary education, electrons, and anything else that starts with ele.

You might hope that all the elephants are there at the start of the "most relevant" search results... but you would be wrong. There are 5 elephants up at the top, but then "Static Electricity Virtual Lab" and "Valence Electrons and Isotopes" etc. etc. are considered more relevant than Orwell's essay "Shooting an Elephant" (there's a quiz for that). I have yet to figure out why Static Electricity Virtual Lab is considered a more relevant search result for "elephant" than materials for George Orwell's Elephant essay which actually involves an elephant.

I found out about Orwell's Elephant this way: when I search for "Highest Rated," the top-rated elephant is Orwell's elephant. There are lots of other highest rated items at the top, though, which have nothing to do with elephants, and that is why you cannot see Orwell's elephant in my screenshot. It's below all these other items in the screenshot. But if you scroll on down, you will find Orwell's elephant essay. Eventually.

I found it using Control-F in my browser.

Here is the search URL:

https://lor.instructure.com/search?sortBy=rating&q=elephant 

Switch the view to "Latest" and all the elephants are missing here too. Really missing. Well, you'll get to them eventually I guess if you keep loading more and more... and more and more.. and more. But no one is going to scroll and load and scroll and load to find the elephants, right? 

Here's the search term: elephant. But the search results are for ele- like elementary mathematics, elementary algebra, "Abraham Lincoln Elementary's 5th Grade beginning of the year prompt," and " the elements involved in warehouse management," and so on.

https://lor.instructure.com/search?sortBy=date&q=elephant 

I hopefully tried putting quotation marks around the word "elephant" to see if that would help. It did not. 

The Helpdesk tells me that this is all on purpose in order to help people with spelling errors:

That is how our search engine was set up. To search as specific as it can get and then to gradually filter in less specific content. This is done so that if a word is misspelled the search is still able to locate it.

if I type "elphant," then Google search shows me results for "elephant." That sounds good. It corrected my typo. But Canvas Commons gives me no elephants if I type "elphant." Instead it gives me two things: an item submitted by someone named Elpidio, and something called "Tech PD and Educational Technology Standards" which involves the acronym ELP. So much for helping people with spelling errors. 

Electricity, elections, elements, elearning: these do not sound good. Those results are obstructing the search; they are not helping. There is nothing "gradual" about the filtering. Static electricity shows up as more relevant than George Orwell's elephant. Some kind of three-character search string is driving the algorithm to the exclusion of actual elephant matches.

If you assume that someone who typed ELEPHANT really meant to type ELECTRICITY or perhaps ELEARNING, well, that is worse than any autocorrect I have ever seen. And I have seen some really bad autocorrect.

This happens over and over again; it affects every search.

Want to search for microsopes? Get ready for a lot of Microsoft. These are supposedly the most relevant microscope search results, but the second item is Microsoft ... even though it doesn't seem to have anything to do with microscopes at all from what I can tell.

https://lor.instructure.com/search?sortBy=relevance&q=microscope 

Still, we're doing better than with the elephants here. There are a lot of microscopes in addition to Microsoft:

But look what happens if you want highest-rated microscopes. See the screenshot; there are no microscopes. It's Microsoft Microsoft Microsoft. But hey, there is also Of Mice and Men!

https://lor.instructure.com/search?sortBy=rating&q=microscope 

So, the search algorithm assumes that, while I typed "microscope" as my search term, I might really have meant to type "Of Mice and Men." Or Microsoft. Or the name Michael (a lot of content contributors are named Michael) (or Michelle).

I could go on. But I hope everybody gets the idea. If this is really a feature of Canvas Commons search and not a bug (???), I hope this three-character search string "feature" can be replaced with a better set of search features.

Although I still would call this three-character approach to search a bug, not a feature. Which is to say: I hope we don't have to wait a couple years for the (slow and uncertain) feature request process to warrant its reexamination.

Comments from Instructure

For more information, please read through the https://community.canvaslms.com/docs/DOC-15588-canvas-release-notes-2018-10-27 .

[ProfessorBeyrer]

Thank you  @James ! Yes, I think forcing spell-checking on the search field would be a good idea, especially (like Google does) giving users the option if they really want to search for that term that the spell-checker thinks they misspelled. This might mean adding a spell-checker to Canvas too, which I don't think is viewed as important to Instructure as long as browsers can handle it (sorry, spelling teachers). Having the spell-checker turned on automatically would help those with typos and those with misspellings that arise from ignorance.

Those are great questions about when to initiate the search, how to structure the query, etc. Personally I find it distracting that the search results start appearing as soon as I type. I would rather type my terms and then hit enter/return and wait for results. The timing of auto-appear results is inconsistent (network speed, etc.), and sometimes my brain jumps to "check out that result" but my fingers have already typed the next key in my search string and by the time I move the pointer to that result the page has changed and I end up selecting something I didn't want to select. The location of the search field on the page moves as soon as results start appearing, so I have a harder time viewing what I'm typing and looking for typos. I have to trust the muscle memory in my fingers, ignore the candy flashing before me, and take a breath before selecting a resource to explore. Requiring the user to initiate the search would allow for things like wild cards. Even auto-complete (Google again) would be okay. I know when I do a Google search I can select one of those auto-complete results at any time, or I can continue typing what I know I want -- either way, search results do not appear until after I take an initiating action.

To me it's all about making things easier for end-users, especially students (helped by the awesome resources their instructors find in the Canvas Commons) and instructors (who can easily find awesome resources thanks to improved search). I focus on the "hassle factor" whenever new educational technology becomes available. To the extent we can take hassle away from students and instructors and keep the hassle with developers and sysadmins, that's a good thing. Of course there are some instructors who enjoy tackling the hassle, and thanks to the open nature of Canvas they can. The Canvas Community's superstars, like you, are a wonderful resource to guide those of us sysadmins and instructors who view that word (hassle) with more capital letters.

[James]

Students use Canvas Commons?  In that case, Canvas definitely needs to do something.

It looks unfinished, like they threw something out there and then got distracted by something else. The search page even has a lot of console.log() statements for everything you do -- basically that's what you do when you're writing code and testing it, not for production use.

Anyway, I'm looking at some tweaks, not a complete rewrite. I've got some ideas, but I won't be altering when the text starts appearing or doing an autocomplete on things, although that does sound interesting.

[laurakgibbs]

It looks unfinished, like they threw something out there and then got distracted by something else.

That was basically my point about needing at OER Strategist. The Commons seriously needs some love. I am glad to hear (via Jared's comment on the other discussion) that Commons stuff will now be showing up in Studio.

I continue to ponder whether the three-character-wildcard searches have been there from the start, and what it might be about users that they don't notice how bad the search is.

Anyway, I'm hoping the three-character-wildcard will be something they can fix now, while revisiting all these other issues with Commons search later on. Surely all the work they did before giving up (for now) on Canvas search could help in terms of thinking through their strategies for Commons too.

Probably because of the odd log-in screen that is/isn't (???) required for Commons, it doesn't look like Google index the Commons thoroughly. Although some of it does seem to get indexed. Hmmm.

asia site:https://lor.instructure.com

[ProfessorBeyrer]

No, I don't think students can use Canvas Commons. I brought up students in the context of who among Canvas users should have the least amount of hassle. If instructors have an easier time finding effective learning artifacts in the Canvas Commons, students will benefit.

[laurakgibbs]

I'm not sure why students could not use it; at my school, students can create their own Canvas courses if they want, and some of them do for group projects, etc. If students are the owners of their courses (as they can be at my school when they create their own courses), then the content would import into their courses just like any other course, right?

And now you've got me curious: if I am logged in to Canvas, and I am browsing around the Commons, and I see something that I want to DOWNLOAD without importing, I would be able to do that even if I am logged in as a student, right? Not sure what I would do with it if I don't have a course where I can manage content, but I wonder how the Commons permissions work: it seems like after a Canvas user logs in and then goes to download something from the Commons, even students would be able to do that...?

But I'm not an admin; I don't know how permissions work outside of courses.

[ProfessorBeyrer]

I think that the Canvas Commons link as soon as a user is enrolled in at least one course as a TA or Instructor. So if students can create their own courses, then they can see that link. 

[laurakgibbs]

I will have to ask one of my curious students this semester to help me explore and see how that works from their perspective! 🙂

[Stef_retired]

Here's what I'm grateful for: this thoughtful conversation!

laurakgibbs‌, we have an idea already open for voting advocating for an enhancement to Commons search: https://community.canvaslms.com/ideas/10762-advanced-search-needed-for-canvas-commons   We believe the existing idea, in that it is broadly written, would if implemented encompass what you'd like to see implemented; would you add your use case description there?

Since our platform doesn't allow us to merge ideas/votes/comment streams, we plan to archive this one to prevent vote duplication.

Thanks!

[laurakgibbs]

So am I correct that this is not going to be treated as a bug fix, stefaniesanders‌?

The claim by the Helpdesk is that a three-character-wildcard search is meant to correct spelling errors, but it is NOT doing that: a three-character-wildcard search (which is not any kind of "best practice" in the world of search) is failing to help with spelling errors and, far worse, it is filling up search results with garbage results. It is not a feature; it is a bug, something that was never intended.

Unless someone at Instructure really thought it was a good idea to return three-character-wildcard matches ahead of real matches...? That is pretty hard to believe.

Find me any search anywhere on the web that returns results for a three-character-wildcard search ahead of real search matches, and I will retract my contention that this is a bug.

But otherwise: I will continue to insist this is a bug, and one that Instructure should be eager to fix. Until this bug is fixed, Commons is going to continue to generate garbage results for months or probably years to come. 

That is VERY different from asking for advanced search features at the Commons.

[James]

It is not a bug. It is poorly implemented and could be done better, but that doesn't make it a bug. It's like many of those "I want it sorted this way" feature ideas that have come through the Community. Those were all treated the same way.