Canvas Release Changes Update: Engineering Edition

This blog from the Instructure Product Team is no longer considered current. While the resource still provides value to the product development timeline, it is available only as a historical reference.

erinhmcmillan
Instructure Alumni
Instructure Alumni
18
5374

Last November we posted a proposal for Canvas release changes in 2019. Our proposal intends to improve the quality of Canvas releases, resolve bugs more quickly, and create more predictable releases with a more manageable pace. Community feedback proved highly supportive, and our engineering teams have continued to make progress with necessary adjustments. Thanks for your patience as we’ve been working out the specifics to share with you.

 

 

Release and Deploy Schedule

Since our initial proposal summary, our engineering teams have explored necessary adjustments for the feature release schedule, non-feature code deploy schedule, and the treatment of Canvas feature options.

 

To review terminology:

  • deploy means making code available in a specific environment; code may or may not be visible, but it is code that does not change customer workflows.
  • A release represents a scheduled event when customer-affecting code is intended to be visible.

 

Schedule Adjustments

The original proposal suggested that releases be moved from every three weeks to every month. The proposal also suggested that deploys would take place every two weeks.

 

After thorough research among all engineering teams, releases and deploys has been adjusted to the following dates:

  • Feature releases in the production environment take place on the third Saturday of every month (originally noted as the first Saturday of every month).
  • Deploys of the main Canvas codebase to the production environment take place on Wednesdays, and beta deploys take place on Thursdays (originally shown in the calendar example with production on Saturdays and beta on Mondays).
  • Deploys of supporting Canvas services are and will continue to be deployed frequently to both the beta and production environment. This behavior is not a change from current practice.

Timeline Implementation

The current date timeline for implementing these changes is as follows:

  • July 13: Last regular Canvas production release with deploy/release on the same day
  • July 15: New feature changes released to beta environment as August preview
  • July 22: August release notes published
  • July 31: First weekday production deploy (recurring every two weeks following)
  • August 1: First weekday beta deploy (recurring every two weeks following)
  • August 17: First monthly production release with new process (recurring every third Saturday of the month)
  • August 17: New feature changes released to beta environment as September preview (recurring monthly on the same day as the production release)
  • August 26: September release notes published
  • September 21: September production release date

 

The process would then continue according to the established cadence.

 

The image representation below displays a visual of the deploys and releases for August and September.

 

Caveats:

  • The beta environment data refresh and test environment data refresh will continue to take place every Saturday and with every production release, respectively, although not represented in the image.
  • The Upcoming Canvas Changes page may display an enforcement date that does not align with a feature release.

Release Strategies

Production Releases

Feature releases will be managed by a feature flag enabled by engineers. The feature flag contains all previously deployed customer-affecting code that is not made visible in the production environment until the feature flag is enabled. This functionality is like a feature option, except that all customer-affecting changes are made available to all customers at the same time.

 

Feature flags also protect the production environment as the feature flag can quickly be disabled by Canvas engineers if needed.

 

Beta Releases

On the same day as the current month’s production release, a separate feature flag will also be enabled within the beta environment containing all customer-affecting code that will be released in the following month. This change ensures that you have a full month to experience all upcoming changes in the beta environment and will include all Canvas services, such as Quizzes.Next.

 

The beta environment data refresh will still take place refreshed every Saturday, which means the updates available in the beta preview will not be available until the refresh is complete.

 

Beta Release Terminology

To better define the intention of a beta release, terminology is being renamed as a monthly preview.

 

This change will also align with adjustments to Feature Option terminology, which is being renamed as Feature Previews. More information about Feature Previews will be made available shortly in a separate blog post.

 

Release Notes

As with our current process, the first week of the beta release is still considered unstable and subject to change as engineers continue to verify the newly available code within the beta environment. Therefore, feature release notes will not be available until the following week. As our new processes become more reliable, release notes may also be able to adjust appropriately in the future.

 

Deploy Strategies

Non-feature deploys contains code changes that are intended to fix bugs, improve performance, and prepare for new features. We know through experience that smaller, more frequent deploys leads to a higher quality product. And when there are problems, our teams can fix them faster.

Production Deploys

Our overall deploy process has matured enough to move the primary Canvas deploys to a weekday. Deploys of supporting Canvas services are and will continue to be deployed even more frequently. To ensure that deploys are non-disruptive to customers, our teams have been planning and are already implementing several new safety precautions, including tightening our processes on how we characterize a change in Canvas (e.g. feature vs bug).

 

Canary Deploys

In the current release process, deploys and releases take place at the same time. Code is incrementally deployed around the world by region, and the first regions have always been the first to experience and report unintended behaviors.

 

The new process will slightly modify this behavior. Deploying code to a small group of end users is called a canary—inspired from historical reference of placing canaries in coal mines as a warning system. Canary deployments slowly distribute the changes to this small subset of users before they are made available to the entire infrastructure, and its impact is usually minimal. Code can be reversed or corrected before resuming deployment.

 

The scheduled production deploy takes place every other Wednesday, but the Canary deployment begins a day earlier on Tuesday under close monitoring. Deployment only proceeds if no problems are detected.

 

All customer accounts will go through a rotation as the canary for the release. Being part of the canary deployment should occur infrequently. Customers will not be notified when they are placed in the Canary deployment. Any unintended behaviors noted by an institution should be treated as part of that institution’s support protocol.

 

Bug Fixes

Only changes characterized as a bug fix can be deployed to the production environment at any time. Our tightened processes ensure that bug fixes are only restoring intended behavior. If a bug needs to be addressed by adjusting functionality or the user interface, the bug will be changed to a feature-type categorization and placed behind the monthly release feature flag for future visibility.

 

Beta Deploys

Deploys of the main Canvas codebase to the beta environment take place the day after a production deploy. Deploying to the beta environment allows our engineers to confirm the newly deployed code is not causing unintended behaviors and ensures the production deploy will be as accurate as possible. Engineers may continue adjusting code in the beta environment up to the production release.

 

Deploy Notes

Because engineering is creating smaller teams with quicker releases, up-to-date deploy notes would become difficult for our teams to maintain—as well as customers to reference. Notable entries for the main Canvas deploy will be available as deploy notes on the same day as the production deploys. Additionally, customers with a greater interest in the full deploy can access our open-sourced canvas-lms Github repository deploy list.

 

When the new production deploy process goes into effect beginning July 31, deploy notes will still be created manually and added as part of the Canvas Release Notes page in the Canvas Community. However, our goal is to make deploy notes more automated while ensuring that notable bug fixes, for instance, are properly highlighted. Automating the notes may affect the future location of the deploy notes. Discussions about automation are still ongoing, and more information will be posted as we have additional updates.

 

Previous Comment Concerns

We had several topics weaving through the comments we received in our first blog post. This second post expands on many of the received questions, but we want to ensure all underlying questions have been addressed. Presented in a question-and-answer-type format, the remaining content summarize the topics that concerned you:

 

Will this change place more focus on fixing issues discovered in the beta environment?

Our teams are redefining the process for how features are being deployed; some that are designed as feature options will undergo beta testing with a small group of customers before being available to all customers.

 

Will this change place more focus on fixing bugs?

Quicker deploys will allow our teams to be able to fix bugs more timely. Bugs are prioritized as part of each planning meeting depending on severity. We obviously do not want to release features or changes prematurely and we anticipate that this new process will also help us improve our quality in our feature deliveries.

 

Will some features still deploy straight to production?

All features will eventually be available in the beta environment, although their use may include some caveats.

 

Other services are currently planned and will be announced at a later date.

 

Will this change increase development of Community feature ideas?

Engineering's goal with this release change is to improve product quality and be more efficient, which may come back as a benefit to allow them to work on more ideas! The feature process itself won't change, and our product team always considers ideas against existing priorities and timelines. Stay engaged in the feature idea process so our product team is aware of the workflows and functionality you’d like to see incorporated into Canvas.

 

Can same-day deploy notes be posted in advance?

As noted, we’re tightening our process on how we characterize a change in Canvas so that deploy notes are not a concern. Official deploy notes will most commonly include notable fixed bugs, considering the majority of Canvas code is deployed for future functionality and infrastructure improvements. If there is any question as to whether a change will affect existing functionality, that change will be changed to a feature-type categorization to be available as part of an upcoming monthly release. And if a code change slips through the cracks, our teams always review the prior release to identify what needs to be improved to prevent unintended future situations from occurring.

 

_____

 

If you’ll be joining us at InstructureCon, we’ll be hosting a session that will review these upcoming changes as well.

 

As always, thank you for your interest in upcoming Canvas changes. Let us know if you have further questions that we can address.

Tags (2)

This blog from the Instructure Product Team is no longer considered current. While the resource still provides value to the product development timeline, it is available only as a historical reference.

18 Comments
KristinL
Community Team
Community Team

313623_new_features.png

Every time the Release Notes are published, there is a CanvasLIVE‌ event which encourages Canvas Community members to walk through the notes, ask questions, and to collaborate. New participants are always welcome!

Upcoming Dates:

Release Notes (2019-06-01) Collaborative Chat

Release Notes (2019-06-22) Collaborative Chat

Release Notes (2019-07-13) Collaborative Chat

Release Notes (2019-08-17) Collaborative Chat

cdoherty
Community Participant

Maybe instead of farming out quality assurance duties to paying customers (canary deploys), and having the nerve to suggest those paying customers make it part of their support protocol, Instructure should take responsibility for its own QA and invest the necessary resources to do so. Does Instructure understand that 'canary in a coal mine' is NOT a good metaphor to use with regard to their customers?

themidiman
Community Champion

I'm not sure exactly how this is going to change our institutional support protocol, but agile cloud software development requires a bit of "in the field" development on a limited use so that any use cases not encountered during internal QA testing.

Here's a great infographic that I 'Googled' up that might provide a frame of reference and the ratio it shows is 5% in the 'canary' version vs 95% of everyone else that ends up using the non-canary version.

https://whatis.techtarget.com/definition/canary-canary-testing

It's a voluntary group so if you're not in that 5%, you should at least be aware that it's an industry standard practice. The modern web browser you are likely using to gripe about this means you depend on the canary process and therefore rely on people willing to field test new features so that everyone else can have a useful product. 

cdoherty
Community Participant

According to this document "All customer accounts will go through a rotation as the canary for the release." Please explain how it's voluntary to be part of the canary group. A commercial company should be 100% responsible for making a useful product that it sells to customers, not force customers to be beta testers. Common practice doesn't mean it's good practice.

themidiman
Community Champion

Good question...I wonder if there's an opt-out in the rotation if one is really afraid of something bad happening. Even without this canary process, Instructure has historically been good to its cloud customers by deploying hot-fixes when something unintended happened...it's been extremely rare, but when it did, the impact was minimal even then.

erinhmcmillan
Instructure Alumni
Instructure Alumni

Hi, Christine,

Thanks for the question. To clarify, one of the primary purposes of this change is to expose fewer users to new changes and potential bugs than in our old process. These changes will help our teams improve their internal processes as well, and to monitor and respond to any issues even faster and more effectively, before and after the deploy. We do anticipate that being part of the canary will take place infrequently because of the number of Canvas accounts located around the world.

Erin

beth_mossop
Community Novice

Thanks Erin, I'm still digesting this- there's not much mention of the test instance (in fact it is only mentioned with regards to the data refresh continuing to take place alongside the Production release).  When will the test instance get the code deploys?  Will it be the same day as the Production deploy?  Will features be released to test on the same day as production?  Also, with the fortnightly code deploys, will there be any system downtime (in any of the instances)?  Thank you!

scottdennis
Instructure
Instructure

Hi Beth,

I can answer your second question; no instances will be taken offline for updates.

erinhmcmillan
Instructure Alumni
Instructure Alumni

Hi, Beth,

Test will continue to follow production as it does now. No changes there except for the release date.

Thanks!

Erin

beth_mossop
Community Novice

Hi erinhallmark‌ and  @CanvasDocTeam ‌, 

Do you know when the Canvas Updates Calendar will be updated with the new schedule and dates?  It currently still reflects the 3 week cycle, showing a production release on Saturday 3rd August...also, are you planning for the calendar to reflect the code deploys now that those are separate from the releases?

Thank you!

jsowalsk
Community Champion

erinhallmark‌ I am interested in learning about this as well.

erinhmcmillan
Instructure Alumni
Instructure Alumni

Hi, Beth,

Are you referring to the Google calendar? The calendar has been updated for over a month—will you refresh your browser and see if the previous version may be cached?

Thanks,

Erin

jsowalsk
Community Champion

What should it look like? Can you please share a screenshot?

beth_mossop
Community Novice

Thanks Erin, yes I meant the Google Calendar (https://community.canvaslms.com/docs/DOC-14787-what-is-the-canvas-release-schedule-for-beta-producti...).  I had to delete the calendar from my Outlook and re-add it to get it to come through correctly!

Is there any plan to add the code deploys to the Calendar or is it strictly for feature releases now?

jsowalsk
Community Champion

Beth -- do you mind sending a screenshot of what it looks like? How did you change out the calendar to add the right one?

beth_mossop
Community Novice

Hi Jessica,

First I deleted the calendar from my Outlook, then, from the link shared above (screenshot below), I clicked on the link "iCAL" and when I opened it, it opened in my Outlook calendar...

Does that help?

jsowalsk
Community Champion

Thank you.

erinhmcmillan
Instructure Alumni
Instructure Alumni

Thanks, Beth! Glad you got it figured out! 

We won't be adding deploy dates since the calendar is designed for releases. Smiley Happy 

Erin