RollbackStrategies
The plan for undoing a bad change — automated (instant cutover, previous image redeploy) or manual (exactly what happened in the scenario: re-running the upgrade tool pointed at the older golden image).
Why we need this / what value this brings
Without a rollback plan, a failed verification just becomes an open incident with no defined next step — exactly what happened here, where the fix had to be manually improvised.
When to use this
Define and test this before the first production rollout, not while a real upgrade is already in a failed state and people are improvising.
How to use or implement this
Automate the connection explicitly: PIV failure → trigger redeploy of the last-known-good golden image, without requiring a human to notice the failure and manually remember which version to go back to; log the rollback itself so there's a record of what happened.
Research questions
- This is the actual gap the scenario exposes: the upgrade tool and the PIV tool aren't connected — PIV failing doesn't automatically trigger anything. What would it take to wire 'PIV fails' → 'automatically redeploy the last-known-good golden image'?
- Manual rollback-via-reupgrade has its own risk: it's just another upgrade operation, which means it can also fail PIV, or fail for a different reason — does the process have a stop condition if the rollback itself doesn't succeed?
- Does the previous golden image's version/hash get recorded automatically at upgrade time (see RecordsAudit/AuditLogging), or does someone have to know/remember which version to roll back to?
Empty folder — drop notes, links, and findings here as you research.