Initial state
Under Nexus-driven update, the initial state of the system has zone and host phase 2 image sources set to Artifact.
root@oxz_switch1:~# omdb db inventory collections show latest
root@oxz_switch1:~# omdb nexus blueprints show target
Performing a MUPdate
When a MUPdate is performed via Wicket, it:
updates Hubris artifacts (SP/RoT etc)
writes out the host phase 1 and phase 2 images
writes out control plane zones to the install dataset
As part of MUPdate/update coordination, it also writes out to the install dataset:
a zone manifest: a list of all the zones in the install dataset, along with their hashes
a mupdate override file: an indicator that a MUPdate happened recently.
After a MUPdate, the install dataset looks like this:
The zone manifest is a JSON file that contains zone names, file sizes, and SHA-256 hashes:
The mupdate override file contains, most importantly, a mupdate override ID (a UUID):
The mupdate override ID uniquely identifies a MUPdate on a particular sled. (When batch-MUPdating multiple sleds, each sled gets its own mupdate override ID.)
Why use a UUID rather than a simple marker file? If multiple MUPdates happen to a single sled, we’d like to treat them as separate events: if a sled is halfway to recovery and another MUPdate happens, we’d like to reset to the beginning.
Detecting the MUPdate
Sled Agent reads the zone manifest at startup and validates all the zones, reporting them up to Nexus during the next inventory collection:
root@oxz_switch1:~# omdb db inventory collections show latest
Also, if Sled Agent finds a mupdate override file, it does two things:
Report this fact in the inventory:
Redirect zone images to start from the install dataset, honoring the fact that an operator did a MUPdate:
BRM42220011 # looker -f $(svcs -L sled-agent) -C | less -FRX
Honoring the MUPdate
The Reconfigurator planner, on seeing a mupdate override file in the inventory, generates a new blueprint that acknowledges the MUPdate. This consists of four changes:
root@oxz_switch1:~# omdb nexus blueprints diff latestThe "will remove mupdate override" field is set for that sled:
All zone image sources are set to the install dataset, reflecting the redirection that Sled Agent did:
Any pending host or Hubris artifact updates are cleared.
The global target release minimum generation field is set to the current target release generation plus one (in this example, the target release generation is currently 2):
The idea behind the last two changes is to bring any Nexus-driven updates to a screeching halt, awaiting operator instruction before proceeding.
Removing the mupdate override file
When the new blueprint is set as the target, the Reconfigurator executor will send an updated configuration to Sled Agent with instructions to:
remove the mupdate override file if the override ID matches
in its internal config, update zone image sources to the install dataset
Sled Agent will not restart a zone when the actual image source (after mupdate overrides are considered) is the same.
Once the new configuration has been applied, it will show up in inventory with zone image sources set to the install dataset:
Also, the Sled Agent config reconciler will report the fact that the mupdate override was removed:
The next planning run with this inventory will generate a new blueprint where the "will remove mupdate override" field is cleared:
root@oxz_switch1:~# omdb nexus blueprints diff latest
Fully recovering
Nexus will not add or update zones until it is sure it can take over updates again. The operator instructs Nexus by:
uploading the MUPdated-to TUF repository to Nexus
then setting it as the target release
rain@castle $ unzip -p repo.zip repo/metadata/1.root.json | oxide api -X POST /v1/system/update/trust-roots --input -
rain@castle $ oxide system update repo upload --path repo.zip
rain@castle $ oxide experimental system update target-release update --system-version 16.0.0-0.ci+gitf64a195ff05
{
"generation": 3,
"release_source": {
"type": "system_version",
"version": "16.0.0-0.ci+gitf64a195ff05"
},
"time_requested": "2025-07-31T22:30:41.614644Z"
}The Reconfigurator planner will generate a new blueprint with zone and host phase 2 image sources set back to Artifact, as long as the hashes of each image in the target release TUF repo match the ones reported by the zone manifest (which should always be the case if the MUPdated-to TUF repo is uploaded):
This noop conversion is always performed whenever zone image hashes match, as long as a mupdate override file isn’t detected on the sled. (On receiving a new configuration, Sled Agent will not restart zones if the hashes match.)
When all zone and host phase 2 images across all sleds have their image sources set to Artifact, the Reconfigurator planner considers the system to have fully recovered from the MUPdate. From this point onwards, zone adds and updates are allowed.
Zone adds (but not updates) are also allowed when no target release has ever been set for the rack (i.e. the current target release generation is the initial generation, 1).
Recovering from a single-sled MUPdate
MUPdating a single sled is only supported in a very restricted set of conditions. One of these must be true:
The sled is not yet part of the control plane (e.g., it’s being mupdated to match the current release prior to being added during a sled replacement).
The control plane is not performing a live update (i.e., all components are on the same release), and a sled is mupdated to that same release.
In particular, mupdating single sleds (a) during a live update or (b) to a version that doesn’t match the rest of the control plane is extremely dangerous and may lead to arbitrary control plane corruption.
When recovering from a supported single-sled mupdate, the flow differs from the description in Fully recovering in that the operator does not need to upload the TUF repo again, and Nexus will perform no-op conversion on its own once it detects the mupdate. This will typically look like two blueprints separated by a small window of time, where the first detects the MUPdate and bumps the blueprint’s minimum target release generation and the second performs no-op conversion of the sled(s); e.g.,
root@oxz_switch0:~# omdb reconfigurator history --limit 2
VERSN TIME BLUEPRINT
... (earlier history omitted)
11 2026-07-27T17:40:34.286Z 98265cf7-69ac-4963-9e71-527c7795790b enabled: updated target release minimum generation from 2 to 3
12 2026-07-27T17:40:58.984Z cddae393-7e2f-4ae3-bb27-676ee6d4e5fd enabled: sled f8bd4a7b-ce72-4439-87a9-abdccacbbf8d: performed all noop measurements updates, sled f8bd4a7b-ce72-4439-87a9-abdccacbbf8d: performed 16 noop zone image source updates, sled f8bd4a7b-ce72-4439-87a9-abdccacbbf8d: noop updated host phase 2 to Artifact: both slot A and slot BThe operator still needs to set the target release to recover, but in this case the version they specify will be the same version already running. With additional work, Nexus could detect this case and fully recover on its own with no operator intervention; this is omicron#10918.
Appendix: Implementation rationale
This section contains detailed reasoning for implementation decisions, as an alternative to excessively long code comments.
Sled Agent reconciler error handling
While reconciling the mupdate override field within Sled Agent, errors during reconciliation for this field can be handled independently of the rest of the system. The argument for this is somewhat non-trivial. Here’s an outline:
If removing the mupdate override succeeds but zone shutdown or startup fails:
(This is the most common case, and also the most interesting.)
Before ledgering a config, we check that if the
remove_mupdate_overridefield is set, all zones' image sources are set toInstallDataset.As a result, while reconciling against a configuration that initially removes the mupdate override, the reconciler will attempt to start zones exclusively from the install dataset.
Once removing the mupdate override succeeds, the next inventory collection for the sled will contain this fact.
Based on this, the planner will clear the
remove_mupdate_overridefield in the blueprint. (In the future, clearing theremove_mupdate_overridefield will be gated on success doing no-op image source conversions, as mentioned in the next step below.)For sleds that don’t have a
remove_mupdate_overridefield set in the blueprint, including this one, the planner will perform no-op image source updates from InstallDataset to Artifact, if the hash of the zone manifest matches the one in the target release.As a result, the Sled Agent reconciler will receive a new configuration with zone image sources set to
Artifactwith that hash.Even though the update is logically a no-op from the blueprint’s perspective, the reconciler will switch to starting zones from the artifact store rather than the install dataset.
But, notably, assuming the zone was correctly written out, its hash is the same as that of the install dataset! So in effect, there’s no difference between starting from the install dataset versus the artifact store.
The planner is responsible for ensuring that no new zones are started until zone image sources are up-to-date, and all existing zones are successfully running.
If removing the mupdate override fails due to an ID mismatch:
(The control plane is expected to handle this case gracefully.)
The config reconciler continues to honor the mupdate override.
Inventory will report the latest mupdate override ID.
The Reconfigurator planner will update the blueprint’s mupdate override field with the new value.
The next configuration will have the new value, and reconcilation should hopefully succeed at that point.
If removing the mupdate override fails due to an error deleting the override file from disk:
(The control plane should handle this case reasonably as well.)
The config reconciler continues to honor the mupdate override.
Inventory will continue to report the current mupdate override ID.
The Reconfigurator planner will not update the blueprint’s mupdate override field.
The next configuration will have the new value, and reconcilation should hopefully succeed at that point.
If removing the mupdate override fails due to an error reading the override:
(The control plane is not designed to handle this case gracefully.)
Omicron zones will fail to start because the zone image resolver will continue to produce an error.
Inventory will report an error in place of the mupdate override ID.
The Reconfigurator planner will not make any updates to the blueprint’s mupdate override field until the error is resolved.
This will be a support incident — a corrupt mupdate override file is not a case the system is currently designed to handle.