A Peek at Quality Assurance and Release Practices in Canvas

[shaun_moon]

We always strive to release defect-free, robust software. I wanted to share some practices and principles that we strive for as we design, develop, and deliver Canvas with an eye on quality.

Following are some of the key aspects of our efforts to release quality software.

• Manual testing, where people on the product team work through workflows and tasks in a candidate release. There can be different areas of focus for manual testing, such as exploratory/ad hoc testing and accessibility testing.
• Automated testing approaches and tools:
• Unit tests (individual blocks/sub-processes of code),
• Integration tests (holistic testing across code components and infrastructure),
• User interface tests (verifying that a person’s experience of using the product is as expected).
• Performance testing around scalability and speed. This is often done through a combination of human-in-the-loop testing approaches and automation. Load testing is a facet of performance testing.

Another practice that helps maintain quality is to release smaller changes. Bigger changes are more likely to create problems for two reasons: (1) they tend to be more complex in and of themselves and (2) they tend to affect or interface with more parts of the system. Both of these factors making testing more and more difficult. Working iteratively and releasing things as we go can significantly reduce the introduction of new issues as well as the effort required to increase confidence in quality. 

There is another benefit to frequent, smaller releases, which is that individuals and teams get better at things that they do frequently. With good quality assurance principles, processes and tools, teams that release more often will get better at pre-release testing and will release better quality code.

Even with these frameworks and practices in place, we don’t catch every defect. When that happens, we make very quick decisions about whether to rollback a release or try to quickly address those issues. Making that decision depends most crucially on (1) the level of impact to customers and (2) our projected times for addressing issues.

We are committed to improving your experience with Canvas, including the release process. We know that pre-release quality is a huge factor in this experience. We are going to continue to work on ways to make this happen, including refining and maturing current pre-release and post-release practices.