Table of Contents
What is canary testing?
It is a type of deployment based testing where a new feature or migrated code is tested in production. The essence of the technique is that new code is pushed to certain chosen or a certain small percentage of users and these users are not aware of the change. QA team too may act as a user group to test the new features. They can do the testing in the same environment as end users to find bugs occurring in production that may not have been encountered or identified in staging.
Why canary testing?
This type of deployment based testing ensures that the blast radius of the change in software initially is small and rollbacks and fixes are manageable. It enables the deployment team to evaluate the acceptance of changes or possibility of bugs fairly quick.
When to use canary testing technique ?
- Canary testing is a very suitable approach for rolling out incremental changes to code. Remember incremental features and not parallel features
- The userbase of your product/code is huge
- When setting up of a beta testing program for your product/software is required, Canary testing is the bulls eye.
When to avoid using canary technique?
- Canary testing becomes increasingly challenging if the deployment is done via standard mobile application installers like google play store
- If too many features are released parallelly, this would increase deployment cost as more machines will be required
How to implement canary testing?
Let’s see a diagram to understand the methodology
As you clearly see that there is a routing mechanism brought in and the users are divided. Let’s see how it is done as a step by step process
- Selection of testers among the users is done by devs. This selection must be done such that they could determine a meaningful fault detection, patterns and trends. These users are not aware of the fact that they are a part of the testing group. A success criteria is also defined.
- A testing environment is setup that consists of the updated software and the load balancer/routing mechanism setup in such a way that the testers are directed to the test environment and rest users continue using the old environment
- Monitoring of testers is done for a set time period to ensure that the new version meets the expectations.
- Once the results are realized, then there could be either of the 2 cases:
- The new version meets the success criteria : In this case the new software is released to all users
- Users encounter bugs in the new version : The bugs are then fixed and then the software is released to the entire userbase.
How do I know when to stop canary testing?
Evaluation criteria are generally defined in step 1 and that help in determining when we can stop canary testing. Some other useful parameters to think about in this construct are:
- Performance benchmark: Benchmark of the new software needs to be compared with the old software including user perceptions.
- Time to run beta testing : While estimating, developers must take into account the time taken for the complete lifecycle of the product
- User headcount : How many users would be diverse enough to cover almost all possibilities in the software flow needs to be taken into account.