RCE breaking LaTeX expression entered into a page if you view the page in HTML mode and other misbehavior

[maguire]

If one enters the equation that @James used in https://community.canvaslms.com/t5/Canvas-Question-Forum/What-is-latest-Update-on-LaTeX-accessibilit...

Directly into RCE with

\ (\small{(x+1)^{10}=x^{10}+20x^9+180x^8+960x^7+3360x^6+8064x^5+13440x^4+15360x^3+11520x^2+5120x+1024}\ )

rather than using the RCE equation editor. [The space after the \ is to avoid the expression being converted in this message.]

When you save the page - it will now display properly with MathJax.

However, if you edit the page and go to RCE's HTML view you will see

 <p>(x+1)10=x10+20x9+180x8+960x7+3360x6+8064x5+13440x4+15360x3+11520x2+5120x+1024\small{(x+1)^{10}=x^{10}+20x^9+180x^8+960x^7+3360x^6+8064x^5+13440x^4+15360x^3+11520x^2+5120x+1024}</p>

If you now go back to the regular view, you see the text and the LaTeX - but the \( and \) have been removed! Definitely not the expected behavior. 

This seems to be a problem due to RCE misbehavior.

Additionally, if you enter in HTML mode using RCE unicode strings such as the following:

<p><span>&#x1D49C;&#xFE00;</span> chancery style; # MATHEMATICAL SCRIPT CAPITAL A &#x1D49C;</p>
<p><span>&#x1D49C;&#xFE01;</span> roundhand style; # MATHEMATICAL SCRIPT CAPITAL A &#x1D49C;</p>
<p><span>&#x2131;&#xFE00;</span> chancery style; # MATHEMATICAL SCRIPT CAPITAL F &#x2131;</p>
<p><span>&#x2131;&#xFE01;</span> roundhand style; # MATHEMATICAL SCRIPT CAPITAL F &#x2131;</p>

When you exit HTML mode, RCE removes the variation selectors and just puts the letter in. However, you now do not have the variant of the character that you expected. If you now go to HTML mode, you see the character and not the &#x1D49C; or &#x2131; and the variation selectors are all gone.

Being unable to specify the variation leads to very peculiar behaviors where the same character will be rendered in different ways in different places in the content! This is because the browser is trying to guess which variant to display. In LaTeX notation, the character you would expect from \mathcal{L} can become the character generated by \mathscr{L].

Has anyone else seen these (mis-)behaviors?

[chriscas]

Hi @maguire,

I'm not an expert in this area at all, but I'm going to tag @James, who I know teacher math and is quite the wizard (in my opinion) with mathematical display options in Canvas.  I can't guarantee that he'll be able to reply, since we're all just here contributing things to the community in our free time, but if he does see this and has time available, I'm sure he'll provide you with a great answer!

-Chris

[maguire]

Locally, the campus Canvas administrators added a note to their page about putting equations into Canvas pages, saying:

Note! LaTeX added directly in the text can be ruined if you switch to the HTML view. If you have to switch between normal text editing and HTML editing, we recommend you use the equation editor instead.

They had also put a request into Instructure about the problem, but as far as I know, have not heard that the problem is being addressed.

That going into an out of editing in HTML mode leads to irreversible damage would seem to be a design flaw.

With respect to the variants: My suspicion is that there is an error in the class: class Sanitize; module Transformers; class CleanElement

in the file canvas_sanitize.rb where REGEX_DATA_ATTR is removing things that it should not; hence, it is losing the variant information. 

Personally, I would prefer to have the editor invoke an external editor - such as emacs and not do the sanitizing that RCE tries to do. I understand that it is trying to enforce accessibility and other constraints - but I find that it often gets in the way of instructors' efforts to present information as they wish and need to for their students. I also understand that the local administrators do not want to have to install custom javascript, stylesheets, etc. - as they may encounter problems and this adds to the difficulty of supporting these custom changes. However, having a system where the symbols shown change depending on the browser's guess of the context does not make sense. The inability to ensure that the symbols shown are the expected symbols is quite important.

Chip

[ErikNunnally]

Just want to confirm that this is still an issue. It seems to me that the problem is only with the Pretty HTML editor. The Raw HTML editor has not affected text in my time using it, but the Pretty editor still has this problem where any LaTeX code (i.e., text within LaTeX delimiters such as backslash parentheses \( \), backslash brackets \[ \], double dollar signs $$ $$, or \begin{} \end{} environments) shown within the window still gets parsed and changed to an output as described above, e.g., \(x^2=-1\) in the Pretty editor changes to x2=−1x^2=-1. (Notice the LaTeX delimiters are no longer there, so there is no equation displayed in the math font, as described above.)

My guess is the Pretty HTML editor puts the LaTeX code in the HTML through MathJax accidentally to display the math, places whatever symbols and numbers it can from that display, then places the LaTeX code after it. I'm also guessing here that the Raw editor is not having this problem because it does not have to parse through the HTML to make any indents or change color depending on syntax (hence the name Raw Editor), so MathJax might not be involved with the Raw editor.

Also, I typically use the Raw editor because it has some benefits over the Pretty editor, and the Raw editor usually displays without me having to switch the view from Pretty to Raw. However, since the HTML editor displays the Pretty HTML editor by default and the Raw editor is not as easy to use without syntax highlighting, switching to the Raw editor is not a solid solution. In fact, as of about a week ago, it has started displaying the Pretty editor even if I didn't switch to it in my last edit session. As far as I know, the editor I selected last (Pretty or Raw) should be the one that gets displayed when I use the HTML editor, but now it seems that sometimes the Pretty editor still gets displayed. I haven't cleared my cache or anything like that, so I'm not sure what's going on there. Maybe it's because I was editing a page in another course, but it doesn't seem to me like intended behavior. Switching to another source like MathML or symbols in plain text is not an option either for my work environment.