Skip to content

Staged rollouts by group

A rollout can go out in stages, and a stage can be a group of hosts. The usual shape is two groups, staging and production: an update goes to staging first, soaks there for a while, and only goes to production when somebody says so.

On the Groups page in the portal, create a group called staging and one called production. Admins and owners can create groups. Put hosts in each:

  • by hand, by ticking them under Hosts, or
  • by label, with a selector such as env=staging. Every host carrying those labels is in the group, including hosts that enrol later.

Policies refer to groups by name, so create the groups before the policy. A policy that names a group your organization doesn’t have is refused when you save it, with the place it names it, for example rollouts.staged.stages[0].hosts.groups: there is no group called "stagin".

name: staging-then-production
priority: 150
selector:
groups: [staging, production]
rules:
- match: { kind: security }
action: auto_merge
rollout: staged
- match: { kind: [patch, kernel] }
action: propose
approval: { required: true }
rollout: staged
rollouts:
staged:
stages:
- name: staging
hosts: { groups: [staging] }
soak: 24h
- name: production
hosts: { groups: [production] }
approval: required
halt_on:
failed_jobs: 1
failed_health_checks: 1

The portal has this as the Staging, then production example on the new-policy page.

  • rollouts.staged is the strategy, and a rule uses it with rollout: staged.
  • Each stage takes the hosts its hosts selector matches. A stage takes only hosts that no earlier stage took, so a host in both groups goes out with staging.
  • soak: 24h waits a day after the last staging host finishes before the rollout moves on.
  • approval: required on the production stage means it does not start on its own. It waits for somebody to promote it.
  • halt_on stops the whole rollout at the first failed job or failed health check, so a fix that breaks staging never reaches production.
  1. The proposal is approved, or merges itself under an auto_merge rule, and the staging hosts get their jobs.
  2. When every staging host is done, staging soaks for 24 hours.
  3. When the soak ends, the rollout shows awaiting promotion. Nothing is sent to the production hosts yet. A rollout.awaiting_promotion notification goes to any channel whose notification rules include it, and the rollout is listed first on the Rollouts page.
  4. An operator, admin or owner opens the proposal and presses Promote production on its Rollout card. Production starts.

While a stage is soaking, Promote now on the Rollout card ends the soak early and starts the next stage straight away. If that stage needs approval, the person pressing it counts as the approval, so the portal asks first.

Halting, resuming and cancelling work as for any rollout. A rollout halted while it waits for promotion goes back to waiting when it’s resumed. It doesn’t start production.

Nobody is left behind. Whatever the last stage selects, any host the proposal covers that no earlier stage took goes out with the last stage. Without the selector above, a host in neither group would be updated together with production. The selector: { groups: [staging, production] } keeps the policy to the hosts in those two groups, so hosts in neither are left to your other policies.

A stage whose group has no hosts is kept, empty, rather than dropped. If staging is empty, production still waits for somebody to promote it.

  • approval: required can’t go on the first stage. Approving the proposal is what starts the first stage, so a gate there would ask twice.
  • Promoting needs the same role as halting and resuming a rollout: operator, admin or owner.

Because policies name groups, renaming or deleting a group that a policy names is refused, with a list of the policies that name it. Change those policies first, then rename or delete the group.