Rollback to last working prod deployment if post deployment health checks fail

Question

Is there a way to configure jenkins, or any other CI/CD tools, to automatically rollback to the last working deployment, if there are any issues post deployment?

E.g, say I expect a minimum number of page views every hour. If, after a deployment, the page views drop to zero or a very low amount, can the CI/CD tool be notified (or monitor it) and automatically rollback to the previous working version (and notify the developers)?

The idea is to catch issues that might not be covered in current tests (or got missed)

Answer 1

We support this via Reliza Hub (new tool I'm working on). Currently we have working integrations with GitHub Actions and GitLab CI. CircleCI will come next, then Azure DevOps; Jenkins will come too but later.

Reliza Hub has a concept of approval matrix, so that your build only goes to specific environment if you assign to it specific set of approvals. If you remove approvals, it will essentially mean that previously approve build is now latest per environment. This does not replicate existing CI/CD functionality and instead provides a routing mechanism - so Reliza Hub keep metadata about which build goes where, serves as a point of truth for CI/CD tools and notifies such tools when changes occur.

In example, here is how our GitHub Actions / GitLab CI integration may look:

1. When builds happen, you poll Reliza for version details and then notify Reliza Hub about new builds to store metadata there: sample - https://github.com/relizaio/reliza-hub-integrations/tree/master/GitLab
2. As discussed above, Approval Matrix in Reliza Hub is configured in such a way to only allow releases into production which obtained specific approvals:
3. You configure project in Reliza Hub to trigger CI builds on specific actions, or on all release create, release update and approval / disapproval events:
4. During CD process, Reliza's mechanism to get latest release metadata is used - either directly or via templating functionality. Sample using templating - https://gitlab.com/taleodor/sample-helm-cd, larger write-up: https://itnext.io/building-kubernetes-cicd-pipeline-with-github-actions-argocd-and-reliza-hub-e7120b9be870 Note: You may have different triggers on different events or a combination of triggers.
5. You can configure programmatic approvals or disapprovals via Reliza Go Client - https://github.com/relizaio/relizaGoClient#8-use-case-programmatic-approvals-of-releases-on-reliza-hub

Essentially, for your use-case - on failing production tests you would programmatically send disapproval event to Reliza Hub, which in turn would trigger corresponding CI build and provide previous stable release details to that CI system - which would result in expected roll-back to last stable state.

If all the above sounds reasonable and you're ok to work with either GitHub Actions or GitLab CI or (a little later) CircleCI as other CIs come later, feel free to contact me at https://twitter.com/taleodor - I will help you configure this.