Conditions where non-prod systems wont work for LTI testing

[nardell]

Greetings : At our institution, I am an advocate for testing new LTI integrations with tool providers against our non-production Canvas systems. I know there are some cases where using *-test.instructure.com or *-beta.instructure.com are not applicable:

• When a third-party tool relies upon a key or token for API calls, and the same token/key does not exist on production and test systems.
• When a third-party tool only authorizes connections from the production URL.
• When there is something fundamentally different between production and test (configuration difference, code upgrade etc.) 

I wanted to ask are there any other cases where using non-production Canvas for LTI testing will fail. My bottom line is that I do not see this approach to testing as final or conclusive. But I prefer to start in a non-production system, to make sure there are no unintended side-effects of the integration, before moving into production. And then, only in production in a sand-boxed Sub-Account. Would like others input and recommendations on this concern.

Mike

[ag3811]

We just came across an interesting case yesterday:  Safari refused to set session cookies for us because the window was from instructure.com while the iframe was from columbia.edu.  It worked fine on production, but we had inaccurate cookies in dev and test!

[nardell]

Angus, thanks that is helpful. Must have been "fun" to diagnose that.. 

[ag3811]

It was quite the safari!

[stuart_ryan]

Hi  @nardell ,

I will admit it has been a while since we ran into this, so I will cover it off as best I can remember (and provide the solution that worked best for us).

Given I have worked in an enterprise environment my entire life, I too share this same concern. I like to test things on a test system, in isolation, just to make sure things are OK before letting it near my production environment. I will also admit that LTI is a generally safer and more limited solution, however my larger concerns come when third party tools need an API key, and therefore, have unfettered access to the system.

The biggest concerns in my mind were:

• For things like Turnitin, once the systems are cloned, all systems connect to the production environment of Turnitin (just as an example)
• Personally, I think an API key that works in production should never work in a test system or vice versa. Once you have had a test system connect to a production environment once in your life that is more than enough
• Not all third party LTI providers have test systems, sometimes there is one and one only (really no way around this, but I call it out for completeness)
• I know some organisations have a 'no production data in test systems' policy (which is a good thing, dummy data is always far preferable), alas this is not possible in the default setup

So, how did we get around this! We found a little golden nugget of information from another institution that Instructure now offers. There is a one of setup cost (but believe me, it is extremely reasonable and very easy to justify to management). I of course would point you to your sales rep for the cost, but will outline the 'what' here for you.

The short of it is that Instructure can provide you with a completely separate 'prod' instance (and hence an additional 'test' and 'beta' as well). There is no data copy between the two, it sits as a completely separate environment and works perfectly for testing. I outlined this in detail on a previous post here https://community.canvaslms.com/message/86014-prod-vs-test-and-best-practices-for-testing-changes#co...  so I won't go *too* much into detail on it but would recommend a read there.

The benefit from an LTI perspective is that you have a completely separate system with no live student data (unless you deliberately load it into there) which means rapidly spinning up LTIs is far safer. It means that your 'Dev Prod' (or 'Test Prod' (whichever you wish to call it)) doesn't clone over itself which means you can have separate API keys to connect to those systems (and giving those out for LTI testing has a far more preferable risk profile).

It does represent a one-off up-front cost, however, I can say that the business case VS the outlay should be a pretty easy sell (if you want some tips on this I am more than happy to give you some guidance). For us, it provided the safest option, with the most amount of flexibility (despite the caveats of having to occasionally clone a class over when we needed some test data).

If you go down this path, I also recommend a look at Consortiums on Commons as that is an easy way to copy things between environments when you need it.

Now, back to a couple of your original points:

• When a third-party tool relies upon a key or token for API calls, and the same token/key does not exist on production and test systems.

Generally the token or key would be copied over during the weekly/three-weekly clone (unless I am mistaken, it has been a while since we look at that)?

• When a third-party tool only authorizes connections from the production URL.

One way around this would be to see if they permit subdomains, with an extra instance you could have test.canvas.yourinstitution.com (and that *may* work, but would need to be looked at on a case-by-case basis). In this case I would be championing with the third-party vendor to provide a proper means to hook in a test system.

• When there is something fundamentally different between production and test (configuration difference, code upgrade etc.) 

Agreed, that is where the solution I mention above was incredibly appealing to us, our Dev-Prod environment is always the same (code-base) as production. We have a very specific process in place to test new features and settings changes so these are always tested in Dev-Prod then promoted to UAT-Prod before going to actual prod.

I hope that gives you some good insights as to how we got around this and helps you out too!

Cheers,
Stuart

[mvandevelde]

Agree with  @stuart_ryan  on this, a second production environment for testing is well worth it. We found a lot of issues with testing not just LTI’s but other features on the test.production environment.  Also it’s no fun losing your work after the test environment refreshes every few weeks.

Mark.