High availability and its limits

Cluster mode runs more than one active instance over one operational store and one work share. What it buys is the three things the feature was asked for: a job is never processed by two instances at once, an instance dying does not leave work permanently stuck, and the pipeline keeps signing while one host is gone.

This page is the other half of that sentence — what it does not buy, stated up front rather than discovered in a change window. Everything here applies only where Cluster:Enabled is true. Off, which is every deployment that has not deliberately turned it on, none of it is in effect and the product is the single-instance one it always was.

The deployment walkthrough is Azure App Service (cluster mode); the day-2 operator view is Operations; the failure catalogue is Troubleshooting.

---

One supported topology

An Azure Web App, Linux container, the existing image, scaled out on one App Service Plan. That is the whole of it.

The mechanisms do not know that. Everything coordinates through the operational store and the work share, and nothing is App-Service-specific but the identity derivation and this documentation — so two on-premises VMs against one SQL Server would run the same code. They are undocumented, untested and unsupported, and three sub-designs that exist only for that shape were deliberately not built: configuration fingerprints, operator-assigned instance names, and per-host certificate name-indirection.

One fact about App Service does a great deal of the work and is why the list is one item long: app settings are per app, not per instance, so every instance is identical by construction. The cross-instance config-divergence hazards — an encryption password differing between hosts, a signing profile one instance never heard of — cannot occur here. On a topology where they can, they are unaddressed.

Upgrades are stop-the-world

Stop the app, deploy, start. There is no rolling restart, no zero-downtime path, and no deployment slots.

A staging slot carrying production's connection string is not a staging environment — it is a second set of instances joining the cluster on a different application version, sharing the job queue, the heartbeat table and the work share with the version you are still running. The swap does not introduce the condition; the slot's first boot does.

The heartbeat's version stamp is the tripwire, not the guard: a booting instance that finds live heartbeats from a different version logs a Critical and carries on. That is deliberate. A hard refusal would block instances from coming up for as long as a dead old-version heartbeat took to go stale — which is exactly the moment an operator needs them up, because it is the moment after a failed deploy.

So mixed versions are detected and reported, never prevented. Treat the Critical as the alarm it is.

An in-place image change is survivable, but it is not the recipe. Since 2.5.0, changing the image on a running app (az webapp config container set) no longer costs a refused start. App Service starts the new container beside the old one under the same instance id and keeps the old one serving until the new one is warm; the new container displaces the old, which stands down — it finishes what it holds and claims nothing new — and says so in its log, as an InstanceStoodDown event and on its System page. See Operations. An instance that shuts down cleanly retires its heartbeat row (since 2.4.3), so after stop, set, start there is nothing to displace at all, which is why that remains the tidier deploy.

A failed deploy takes the site down, and setting the previous tag back is the recovery. If the new container fails its warm-up after displacing the old one, App Service does not keep routing to the old container: it stops the whole site, the stood-down container included, and keeps restarting it with the new image. Nothing the product does can prevent that. Point the app back at the previous image tag with az webapp config container set; it is ready again within about two minutes.

Session affinity is required

The dashboard is Blazor Server, and a circuit is a stateful SignalR connection that must keep landing on the instance holding it. App Service ships ARR affinity on by default and it must stay on. This is documented as a requirement rather than engineered around.

What affinity is not doing is keeping people signed in. Both session cookies are Data Protection payloads, and in cluster mode the key ring moves into the operational store precisely so a cookie minted by one instance validates on every other. Affinity is for the circuit; the shared ring is for the cookie. Turning the ring off would produce intermittent sign-outs that no affinity setting fixes.

Rate-limit budgets are per instance, so the effective limit is ×N

Every rate-limit policy in the product is a per-process limiter. Two instances mean twice the permits; N instances mean N times.

This is documented rather than fixed. A distributed limiter would be this on-premises product's first shared-infrastructure runtime dependency, for a control that is coarse by design, and the arithmetic that matters most — the compensating-control argument for the anonymous approval route — was re-read with the ×N factor and survives at the small N this topology runs. If you scale past a handful of instances, re-read it yourself rather than assuming it still holds.

Sizing a budget for a cluster means dividing by the instance count you actually run — and remembering that the instance count changes when you scale.

Pipeline:MaxConcurrency multiplies the same way, and that one is a feature rather than a limitation: it is per instance, so a fleet of two at concurrency four signs up to eight files at once. Size the certificate source for that number, not for the configured one.

Metrics scraping reaches an arbitrary instance

/api/metrics is per-process, and App Service's front door cannot target an instance. A Prometheus scrape therefore lands on whichever instance the load balancer picked, and the series it collects jumps between instances between scrapes. Scrape continuity breaks, and no configuration recovers it.

The recommended cluster observability path is the Application Insights distro — opt-in, instance-aware natively. See Telemetry.

If you keep Prometheus anyway, two gauges have meanings decided rather than inferred, and reading them wrong under-reports:

• bulksigner_jobs_awaiting_signer counts the rows this instance polls. sum() across the fleet is the cluster-wide total with nothing double-counted, a job having exactly one owner. Reading a single instance's series as the total is the mistake to expect.
• The same shape applies to every per-instance counter on the page. One instance's numbers are one instance's.

GET /api/folders carries an instance field for the same reason, so a machine client can at least tell "the folder changed" from "the answer came from somewhere else".

Logs are ephemeral unless you make them durable

A Linux container's disk vanishes on recycle, and rolled log files with it. Storage:Root's logs/ directory is inside the container.

Cluster mode warns and starts when Logging:AzureTable:Enabled is false: a Critical at startup naming the loss. It does not refuse, on the product's own severity gradient — an unreachable work share and an unreachable store both warn and start too, and a boot refusal over a diagnostic stream inverts that. The never-only-sink rule is untouched, so the file sink stays on either way, on ephemeral disk, where App Service log streaming reads it live.

Turning the table sink on has its own cost, and it is a decision to make before enabling it

Nothing prunes that table and no Azure mechanism can. See Retention and schedule the pruning script.

A presumed death is a wager

Liveness is heartbeats in the operational store. An instance that is alive but cannot write them — partitioned from the store, or stalled past Cluster:StaleAfterSeconds — may have its work taken over while it is still doing it. The losing case is named rather than hidden.

What bounds the damage is unchanged from single-instance operation, and all three mechanisms predate cluster mode:

• The staged bytes are re-hashed immediately before any signature exists.
• A promote onto an occupied destination is refused — a duplicate completion becomes one Completed job and one Failed one, never two delivered artifacts.
• An input is compared against its staging fingerprint before it is deleted.

Raise Cluster:StaleAfterSeconds where a deployment meets this routinely. The floor is three cadences, refused at boot below that, because a threshold that short presumes death on one or two missed beats and a beat goes missing for reasons that are not death.

The mirror image is also stated: an instance that goes stale to its siblings keeps signing. It is not stopped, because this product's standing rule is that an in-flight job runs to natural completion. It never takes over its own jobs, whatever the table says.

The same window paces the question about a displaced instance. After a redeploy overlap, the new container's sweep asks the store, once the window has passed, for jobs the displaced incarnation left behind. Once that question has come back empty it is asked again once per Cluster:StaleAfterSeconds, not once per poll (since 2.8.0) — on App Service every roll overlaps, so the displacement record never clears. The cost is that a displaced process which never stood down and claims a job late has it taken over within one window and one poll rather than within one poll, which is the wager above, not a new one.

Rows nobody owns are reconciled by nobody

A job row carrying no owner — left by a build older than the ownership column, or by a run with the mode off — is a row cluster mode will never sweep. Boot recovery takes only this instance's own identity, a sibling's takes theirs, and takeover follows an owner's heartbeat, of which there is none.

It is reported rather than adopted. Adopting would mean a filter matching null, which matches in every instance simultaneously — the defect the feature removes, arriving through the code that removes it.

The remedy is real, and is named on every surface that meets one

Boot once with Cluster:Enabled = false, which sweeps every in-progress row whoever owns it, then turn the mode back on. Do this once at the upgrade, before the first cluster boot, and it stops being a concern — the owner is recorded on every claim whether or not the mode is on, and only read under it.

Two specifics are worth having in writing because they are worse than they look:

• A job dispatched to Lacuna Signer with no owner is polled by nobody and no longer times out. Signer:TimeoutHours is enforced while a row is being polled, so a row nothing polls is a row nothing bounds. Scoping the poll to owned rows removed the last terminal path such a job had. The poll worker says so once per process and names the count.
• A row owned by a named instance that has no heartbeat row at all is stranded the same way and takes the same remedy. Absence of a heartbeat is not evidence of death — an owner with no row is reported once and left, rather than read as a licence to fail somebody's live work.

There is no per-instance drain

You cannot ask instance B to finish what it has and stop taking new work. POST /api/pipeline/pause holds every instance's worker — the pause flag lives in the one row every worker reads each poll iteration, so cluster-wide is what the existing control comes to mean, and it is what an operator pausing "the pipeline" intends.

The answer to "patch instance B" is to stop it and let takeover do its job. Per-instance drain was deliberately not built.

One consequence of pause being cluster-wide is worth knowing: takeover sits behind the pause gate. An operator pausing a cluster to investigate a store that has gone slow is exactly the person who must not have every instance declare every sibling dead. The approval-expiry sweep sits in front of the gate instead, because a wait budget is a wall-clock deadline that pausing does not extend.

Cross-instance latency is the poll interval

The wake signal is process-local. An enqueue on instance A does not wake instance B's worker; B picks the job up on its next poll. Pipeline:PollIntervalSeconds is therefore the cluster's cross-instance latency bound — accepted and documented, not engineered around.

A signing profile edit rides the same bound, by design. Signing profiles live in the operational store and are edited from the dashboard; a profile write bumps a counter in the store inside its own transaction, and every instance reads that counter on the poll it already runs — so a change made on instance A reaches instance B within one poll interval, with no message path between them to configure, secure or debug. What propagates is behaviour: format, verification, encryption, CNAB240 checking and the whole approval rule. A certificate change does not, on any instance: each host opens its certificate once at startup and keeps the key handle, so changing one is a restart — on this topology, the stop-the-world restart described above.

Two consequences worth expecting:

• Between a certificate edit and the restart, two instances can honestly report different sources for the same profile — each naming the key it would sign with, which is also the source it stamps on the jobs it signs. A scrape or a dashboard request lands on an arbitrary instance, so read the answer as "what would sign here", not "what the row says".
• The pending-restart marker a certificate edit raises is per instance, and truthful. An instance that has restarted shows nothing and one that has not shows the marker — both correct about themselves — which is what makes a partial recycle readable. On the stop-the-world restart every instance loses the marker together.

Two certificate sources cannot be chosen at all on this topology. A PKCS#11 token and the host's own Windows certificate store live on one machine, and these instances are created and destroyed by scale operations — so creating a profile on either source, or re-pointing one onto them, is refused in the form while Cluster:Enabled is on. Configuration that states one is still a boot refusal. A profile saved before the switch was turned on is the one case neither catches: nothing re-validates a stored profile, so it is a startup warning instead, naming the profile and its source. Expect it to look asymmetric across the fleet: the instance that has the token signs and warns, while every instance without it reports the profile as degraded and fails the jobs routed to it.

The same locality is what makes re-ingestion keep working for free: under all-watch, the instance that finishes a job always watches the folder it came from, so the process-local signal still reaches a watcher that owns the path.

The work-share gate is narrower than the catastrophe it is named for

The marker binds a work share to one operational store, and a booting instance whose store does not match refuses to start naming both. That catches the shape it is for: a second deployment meeting a share an established cluster has already marked.

What it does not catch is any moment the marker is unreadable, because it refuses on evidence and never on the absence of it. Two such moments exist — a share carrying no marker yet, and the instant of a naming write on Azure Files, where the file is briefly full-length zeros. Both are narrowed by one extra look when a lease is held, not closed. A check that runs once at boot also cannot see a rival cluster that arrives afterwards.

And the gate is not what stops two instances signing one file. The per-file lease and the database claim are. The marker is for the one catastrophe no database can see: two stores, one share.

Database backup is unreachable here

Backup:Enabled = true under Database:Provider = SqlServer is a boot refusal naming both keys — and cluster mode requires SqlServer. The combination is therefore unreachable by construction, which is a pleasant consequence rather than a gap: the process-wide backup gate needs no distributed replacement.

Backing up the operational store on this topology is your DBMS regime's job. Azure SQL's own point-in-time restore is the answer, not this product's feature. See Retention.

The session key ring is plaintext in the store

In cluster mode the Data Protection ring is rows in SessionProtectionKeys, in plaintext, guarded by the database's own access control — consistent with a security model in which the connection string is the credential and keys/ read access is already documented as "a session as anyone".

Two things follow, and the second is the one that is easy to miss:

• The Windows DPAPI encryptor is dropped under the switch. Machine-scoped DPAPI is precisely the property that makes a copy of keys/ useless on another host — and precisely the property that makes a ring unreadable by a sibling, so keeping it would be keeping the defect. On Windows this is weaker at rest. It costs nothing on the supported topology, whose Linux container has no encryption at rest for the on-disk ring either, and it is the one place turning the mode on trades a control away rather than adding one.
• An unreachable store fails the request, with no fallback. A host that could not reach the store and quietly minted sessions from a per-instance ring would issue cookies its siblings reject — the intermittent sign-out the shared ring removes, arriving through the code that removes it. What that failure looks like is the provider's own exception on the request path, not a diagnosed refusal naming the ring. The condition is reported where it is diagnosed: the boot check and /api/ready's per-instance database row.

A first-boot observation, stated so it is not read as a fault: instances that start together all meet an empty ring, so several can mint an element before any has read another's, and the table can carry more elements than there were key rolls. Nothing is wrong with that.

Contention has a cost, and it is a small one

Two instances claiming from one queue conflict routinely, and the batch claim falls back to one-at-a-time with a logged lost race. In cluster mode that line is demoted to the expected-outcome level, under its own event id — "a sibling got there first" and "something else on this instance did" are different facts for a reader.

Related and deliberately not a failure: a lease conflict on an input, and an AlreadyActive enqueue outcome. Every instance watches every folder, so losing a race is routine — neither counts toward a folder's consecutive-failure breaker, and an already-active outcome resets the counter exactly as a successful enqueue does. See Operations.

What cluster mode does not change

Listed because each is a rule somebody reasonably expects a clustering feature to have relaxed, and none of them is:

• No automatic retry of signatures. A signature is never re-attempted without a human deciding it. The takeover policy walks that edge deliberately: a job that never reached the sign call is re-enqueued because nothing was attempted; a job past it fails. Failed is an honest terminal outcome, not "stuck", and the operator's manual retry remains the retry. A job signing a Lacuna Signer document past that point is first asked about at Signer, and completed or canceled when Signer's answer settles it; it fails with signer-folder.interrupted only when it does not (see Lacuna Signer integration).
• Every instance sweeps the bound Lacuna Signer folders. With a SignerFolder profile, each instance lists the folders every Signer:FolderSweepIntervalSeconds; a document two instances find is taken once, because the store records each (document, flow action) it has taken and refuses the second. The signer-notifications and signer-folder-sweep readiness rows are per instance.
• In-flight jobs are sacred. POST /api/jobs/{id}/cancel is valid only for jobs no worker is running — Queued, AwaitingSigner and AwaitingApproval — on every instance.
• Watched folders stay in configuration; signing profiles no longer do. Moving profiles into the database was considered as a prerequisite here and dropped when the topology settled, because App Service provides cross-instance consistency by construction. It has since landed on its own terms: profiles are rows in the operational store, edited from the dashboard, which moves approver-pool edits from host-file access to dashboard authorization. What that adds here is the propagation bound above and nothing else — no new topology and no coordination between instances. Watched folders are still configuration, and every instance still watches every folder.
• The approval gate is unchanged. The rule is still frozen onto the job at the park, a rejection is still a veto, and the staged bytes are still re-hashed before any signature exists.
• Clear Jobs takes every job, a sibling's included — by decision, not by omission. From 2.0.0 to 2.8.x it was terminal-only for everyone, because deleting the row under a running sibling's job was the sharpest operator-action hazard in the inventory. Since 2.9.0 it deletes every job whatever its status: an operator wiping the system from the danger zone has wiped a system with a job in it, on purpose, and the dialog says so. The abandoned job's worker fails it, finds no row, and logs both — and the per-file lease, not this action, is still what stops two instances signing one file. See Clear Jobs.

What is not a limitation, despite looking like one

• Pause is cluster-wide. One call holds every instance, which is what an operator pausing "the pipeline" means.
• Statistics are cluster-wide. They moved into the operational store and are computed on read, so the panel describes the fleet rather than whichever instance answered. What was excluded before is still excluded: the awaiting-signer and awaiting-approval waits, and QueuedAt rather than CreatedAt as the queue-wait anchor. See Job statistics.
• Approver links, second factors and sessions cross instances. The verification window lives in the operational store keyed by an identifier carried inside the cookie, which predates clustering entirely — so a window opened via one instance is honoured via another with nothing added.

---

Related: Azure App Service (cluster mode) · Installation · Configuration · Operations · Troubleshooting