Operations

Day-2 operations for a deployed TrustBridge: what to monitor, what to read in the logs, how a restart and an upgrade behave, how to keep the fleet on a release the Server accepts, and the recurring maintenance a deployment owes.

Health and monitoring

Probe	What it proves	Use it for
GET /healthz	One database round trip. Healthy (200) or Unhealthy (503), and nothing else in the body. Anonymous, unaudited.	Uptime monitoring and alerting. A 503 is a container that is up and a database that is not.
GET /	The container is alive. Answers 200 with the database gone.	A restart trigger, where the platform has one (App Service Health Check).
Sign in at /admin	The path a real request takes, through the database.	The check after every deployment and upgrade.

/healthz does not report the fleet: a Server with no workstations connected is Healthy. The fleet screen is where a machine's presence belongs.

The Server starts and stays up with its database unreachable after start, so that a database blip is not an outage a human has to end. While the database is away, the front door answers 200, the contract fails, and the vault is closed. Alert on /healthz, not on the container.

Since the Server applies its own schema at start, a container that is running has proved it reached its database once; a database unreachable at start is a refused start after about ninety seconds.

What the admin console tells you

• The Dashboard (/admin/dashboard), where every sign-in lands, counts the documents and digests signed today, this month and over the last 30 days, settled a few minutes behind, and lists beside them what needs you: Signing Activity behind, the last 24 hours' failures, machines refused until they update, admitted machines not connected, readers left failed, locked tokens and certificates expiring, each linking to the screen that settles it. It is the first page to read; see Dashboard.
• The status strip at the top of every admin page counts the workstations online, the machines whose protocol the Server no longer accepts (N update required), active sessions, documents and digests signed in the last 24 settled hours, failures in the last 24 hours, and certificates expiring within 30 days. Each chip opens the screen it summarizes. N failures (24 h) opens the Audit Log filtered to failures since the moment the count started; it is the quickest way from "something is wrong" to the rows that say what. A Signing Activity stopped at … chip means the Dashboard's count has stopped advancing: read the Server log (see Warnings worth an alert).
• A banner on every admin page means one of three things: the Server's PKI licence was refused, so no Vault Certificate signs and no workstation will once its Service restarts (renew it); audit writes are falling back to the Server log; or the Lacuna test PKI is trusted.
• To prove the deployment signs end to end, make a Test Signature from the Test Signature screen (/admin/test-signature): a CPF, its PIN and a Client Application's API key give back a signed one-page PDF, or the step that failed. On Docker it needs a loopback listener first; see The Test Signature.
• Why a workstation disconnected is on its workstation.disconnect row, and beside each disconnect in the machine's panel on the fleet screen: timed-out (the Server stopped hearing from it, see Workstations:ClientTimeout), closed, revoked, removed, returned-to-queue, server-stopping or error, with the transport it was on. A closed on LongPolling may also be a machine that went quiet: long polling is not timed out, and a proxy that does not carry WebSockets is the usual reason a machine is on it.
• What is wrong at a machine without being at it: a machine on Workstation 1.8.0 or later reports its own condition, shown in its row's Report chip, its detail panel and the Workstation Reports screen, and from Workstation 1.10.0 who is at its console. Its refusals in the last 24 hours are counted on its row. See Workstation Reports and Finding a machine's problems.

Warnings worth an alert

Server log line, at Warning	What it means
Workstation … holds a candidate certificate for this session but is connected at protocol 5, below 6: it cannot hold a Signing Authorization …	A machine on Workstation 1.5.0 was passed over for a session whose Client Application has Per-Transaction PIN on. It counts as offline for such sessions until it runs 1.6.0. See Rollout order.
Workstation … has given N PIN Proof(s) for CPF … that did not check …	A machine spent its failed-proof budget for one holder (Sessions:MaxPinFailures inside Sessions:PinFailureWindow). It is asked nothing for that CPF until the failures age out. Audited as pin.proof-budget-spent.
The Certificate Vault has tried N password(s) for CPF … that opened nothing …	The same budget, spent by the vault. Also audited as pin.proof-budget-spent, with Certificate Vault as the holder name.
Passwords are switched off on this Server's admin console …	Logged at every start while Admin:CertificateSignIn:Required is on. It counts the enabled accounts with no Bound CPF and how many of them are Administrators. When that number reaches the number of enabled Administrators, nobody can administer the platform.
Forwarded headers: honoured from whoever connects …, and this Server is not on App Service	A Docker deployment still using ASPNETCORE_FORWARDEDHEADERS_ENABLED. Name the proxy in ForwardedHeaders:TrustedProxies and unset the variable.
… pings every 00:00:15, but Workstations:ClientTimeout is …	A machine pings less often than the timeout allows for, so it is hung up on and reconnects over and over.
One Warning at start naming TestSignature: keys that are no longer read	A deployment upgraded from Server 1.14.x still sets TestSignature:Cpfs, ApiKey, FallbackEmail, Turnstile:* or Timeout. Remove them, and revoke the key ApiKey held. See The Test Signature.
Sessions:SignatureAllowance is set and is no longer read (ADR-0050) …	A deployment upgraded from before Server 1.17.0 still sets the deployment-wide allowance. Its sessions are unbounded until an Administrator sets a Signature Allowance on each Client Application that needs one; then remove the key. See Upgrading to Server 1.17.0 or later.
Audit retention purge is held at …, where Signing Activity is folded to, rather than …	The Dashboard's Signing Activity is behind, so the purge keeps audit rows past Audit:RetentionDays until they are counted. Logged at Error once it is a day or more behind. The Dashboard and the status strip say where it stopped.
Signing Activity fold skipped a pass: the database answered its lock with a fault …	One pass of the Dashboard's count did not run. Recurring every minute, it is a count that never advances.
Signing Activity is held at … by a Test Signature that began then and has not ended …	A Test Signature holds the Dashboard's count back while it runs; this one has run longer than any can. Nothing after it is counted until it ends or the Server restarts.

A spent budget is not the holder's PIN lockout: a wrong PIN costs the holder's own machine one failed proof per certificate of theirs it holds, so a machine can be passed over before the lockout trips. Both clear as their failures age out of the window.

The vault latch

On a Server with Vault:Enabled, the vault's start-time checks read the database. Where the Server comes up before its database answers, the vault latches shut rather than the host refusing:

warn: TrustBridge.Server.Common.Vault.VaultReadiness[0]
      The Certificate Vault is on, but its start-time checks could not read the database, so the
      vault is closed until they have run. Nothing will be signed with a certificate this Server
      holds and nothing new will be imported until then; the checks are retried at the first
      attempt to use it (ADR-0024 §8).

A closed vault is a healthy-looking Server that refuses every signature with a server-held certificate and every import, while the workstation tier works normally. The checks are retried at the first attempt to use the vault, and a database that has come back settles them open. What a closed vault must never do is mint a fresh key, which would orphan everything stored: that is the reason the latch exists.

Reading the logs

The console is the record. Ship it somewhere with a retention you chose: the container's logging driver on Docker, Application Insights or a Log Analytics workspace on App Service. The rolling file under /tmp/trustbridge-logs feeds the admin console's Server log screen and is a convenience, not a copy to keep.

These categories carry the lines an operator should recognise, all visible at the default level:

Line	Category	When
Which migrations were applied, or why the schema was refused	…Persistence.SchemaUpgrade	Every start
Which Key Ring Protector is sealing, its thumbprint, and how many retired ones are readable	…KeyRings.KeyRingPosture	Every start
Whether the vault is off, open, or latched shut	…Vault.VaultReadiness	Every start, and when the latch opens
Which reverse proxies are trusted for the caller's address, in one line beginning Forwarded headers:	…Http.TrustedProxies	Every start
How many accounts hold no Bound CPF while passwords are off	…Admin.CertificateSignInPosture, at Warning	Every start with Admin:CertificateSignIn:Required on
An audit row could not reach the database, carrying the row	…Audit.AuditLog, at Critical	Whenever the database refuses a write
The PKI SDK licence was refused: expired, garbled or not a licence	TrustBridge.Pki.Validation.LacunaPkiRuntime, at Critical	Every start, vault or not, and when a licence lapses while the Server runs
The Administrator overrides could not be re-read after the schema step	…Persistence.SchemaGate	A start where the settings table did not answer
A machine passed over at session creation: on protocol 4, or its failed-proof budget spent	…Sessions.CreateSession.Handler, at Warning	Each time it happens
How Signing Activity was back-filled, and whether digests are counted since the installation or from the oldest surviving audit row	…Dashboard.SigningActivityFold	The first start of Server 1.17.0 or later
What the audit purge deleted, or that it is waiting for Signing Activity	…Audit.AuditRetentionPurge	At start, then daily
A document of an application switched to REST PKI Core that could not be signed before any key was used: a key REST PKI Core refuses, a key without the Worker role, a security context it does not have, or no RestPkiCore section any more	…Signatures.RestPkiCoreAdmission, at Error	Each signing call it refuses

Every HTTP response carries an X-Correlation-Id, and every log line written while serving it carries the same id, as do the Audit Log rows it produced. A client's complaint, a log line and an audit row meet on that id.

Audit fallback belongs in your log retention, not your backups. An audit row the database would not take is written to the Server log at Critical and nowhere else, and the admin console raises an alert. The trail has a hole exactly where the database was down; the Critical lines are what fill it.

The Audit Log

Retention is Audit:RetentionDays (365), overridable on the Settings screen. A daily purge deletes older rows. Match it to the deployment's dispute window. The purge over a large AuditEvents table is also what makes the occasional migration long, which is what WEBSITES_CONTAINER_START_TIME_LIMIT=1800 on App Service and the thirty-minute per-command budget are for.

The purge waits for the Dashboard's Signing Activity, which is counted off the Audit Log. On the first start of Server 1.17.0 or later, Signing Activity is back-filled from the whole Audit Log a chunk at a time, and the purge deletes nothing until that is done. After that it never deletes past the point Signing Activity is settled to: while the count is behind, audit rows are kept past the retention, and the purge logs a Warning saying so, an Error once it is a day or more behind. Signing Activity itself holds no personal data and is never purged, so the Dashboard's totals since the installation can be larger than what the Audit Log still holds.

Reports and queries written against the database should know what changed in the rows they read:

• Since Server 1.24.0, an api.call row carries errorCode, the Error Code the call was answered with, where it was answered with an error envelope. Older rows have no such field.
• Since Server 1.23.0, SigningOperations.FailureCause records what the Workstation answered when a run failed there (PIN_NOT_CACHED, PIN_REJECTED, TOKEN_LOCKED, CERTIFICATE_UNAVAILABLE, NOT_AUTHORIZED, FAILED, or UNANSWERED where no answer came back), beside FailureCode. It is null for every other failure, and for a run that failed before the upgrade. Workstations gains seven nullable columns holding the console its last report carried (ConsoleAccountSid, ConsoleAccountLabel, ConsoleLocked, ConsoleTrayAttached, ConsoleTrayVersion, ConsoleOtherTraySessions, ConsoleReportedAt). A pin.prompt-answered row answered Undeliverable carries detail, in the shape below.
• Since Server 1.22.0, a signing.authorization-refused row whose reason is NO_TRAY carries detail: the machine's reason, its console session, the sessions a Tray App was attached in, and the Tray's build. It is null on every other reason and from a machine before Workstation 1.9.0.
• Since Server 1.21.0, SigningSessions.WorkstationId and SigningOperations.WorkstationId are no longer foreign keys. They record the id of the machine a session or operation was bound to, and for a removed machine that id matches no row in Workstations: join with a left join. The removal's admin.workstation-removed audit row names the machine for as long as it is kept.
• Since Server 1.20.0, certificate thumbprints are stored as upper-case hex in the certificate registry, the vault and the Trusted CAs, and every moment is stored in UTC. The Audit Log is never rewritten, so its older rows keep the spelling they were written with.
• Since Server 1.18.0, every row a document's signing run writes carries documentSigner, and session.created carries pin ("proven" or "none").
• Since Server 1.17.0, the signing outcome rows carry their timing (timing, and machineReported where the machine gave its own account), whatever Diagnostics:ServerTiming says.

And two changes since Server 1.11.0:

• certificate.encoding-dropped and certificate.announcement-refused write one row per snapshot, not one per certificate. Their details are { count, entries: [...] }. Rows written before 1.11.0 keep their flat fields, so anything parsing these details must read both shapes. The row's cpf is set only when every entry names one holder; a row covering several is found by the machine, not by the CPF.
• Workstations.EnrolledAt is now Workstations.FirstSeenAt. Only the column name changed; the admin console already said "First seen".

Restarts and upgrades

One instance, so every restart is visible downtime of a minute or two. What survives:

	Survives a Server restart
Workstation connections	Machines reconnect by themselves with backoff
Workstation-bound Signing Sessions	Yes, verbatim: the PIN they rest on is cached at the machine
Vault-bound Signing Sessions	No. They end as Stranded; clients meet SESSION_EXPIRED and create new ones with the same PIN. A vault-bound Per-Transaction PIN session (transactionPin: "client") keeps no secret between calls and survives
Administrator sign-ins	Yes: the web-session key ring is in the database
In-flight signing requests	No; they fail retryably. A Signing Operation left Pending is ended by a sweep as Failed with RUN_STRANDED
Administrator settings overrides	Yes

On App Service the platform restarts containers for its own maintenance, so a restart cadence that is not yours is a visible one for vault signing. Leave DOCKER_ENABLE_CI off and deploy deliberately.

Upgrading the Server is a new image tag: docker compose pull && docker compose up -d, or a new image on the App Service, for example trustbridge-server:1.24.0. The new image applies its own migrations at start and names them in the log. With Database:MigrateOnStart=false, apply them first or the new image refuses to start.

Back up the database before every upgrade: it is the only way back. Most Server releases from 1.9.0 on migrate the schema (1.14.1, 1.16.0, 1.19.0, 1.22.0 and 1.24.0 do not; the Changelog says which), and an older image refuses to start against a database a newer one has migrated. Rolling back is therefore restoring the pre-upgrade backup and deploying the previous image, never just swapping the tag. The one exception is a release that adds no migration over the one before it: from 1.24.0 back to 1.23.0, for example, is an image swap. A fleet that has already moved to a newer workstation protocol cannot connect to the older Server either. On PostgreSQL the backup is a pg_dump; see Server installation.

Upgrading from Server 1.6.x: set Certificates:Validation:License first. Every Server since 1.7.0 needs the PKI licence, vault or not, and refuses to start without it.

Upgrading to Server 1.17.0 or later

Three things to know on the first upgrade past 1.16.x:

• Sessions:SignatureAllowance is no longer read, so a deployment that sets it has unbounded sessions after the upgrade. No Client Application has a Signature Allowance after the migration, and nothing carries the key's value over. Straight after upgrading, set the figure on the Client Applications screen for each application whose sessions it bounded (1 for an integrator that asked for single-use sessions), then remove the key. A Server that still finds it set starts, with one Warning naming it, whatever its value.
• Signing Activity is back-filled from the whole Audit Log, a chunk at a time, while the Server runs. The Dashboard says nothing is settled yet until it is done, and the audit purge deletes nothing until then (The Audit Log).
• Dashboard:TimeZone (default UTC) decides where the Dashboard's days and months start. Set it to the deployment's IANA zone, such as America/Sao_Paulo, if UTC midnight is not the day the deployment means; a zone the Server does not know, or a Windows zone id, refuses the start.

From Server 1.18.0, Signing:StrandedAfter must be greater than twice Signing:RequestBudget plus Signing:SignerCancelBudget. The defaults clear it; a deployment that set StrandedAfter between the old floor and the new one refuses to start, naming the key.

Upgrading to Server 1.23.0 or later

Server 1.23.0 applies three migrations at start, on both engines: ReportWorkstationConsole adds seven nullable columns to Workstations, RecordTheRunsFailureCause adds the nullable FailureCause column to SigningOperations, and IndexTheAuditLogByWorkstationAndTime indexes the Audit Log on machine and time, for the Events tab, the Audit Log's Workstation filter and the refusal count on the Workstations screen. The first two are quick. The index is built over the whole AuditEvents table, so its length grows with the Audit Log:

• Go straight to Server 1.24.0. Server 1.23.0 built the index offline, which holds a lock that blocks writes to the Audit Log for most of the build, and every signing call writes one. Server 1.24.0 builds the same index online at low priority where SQL Server can: Azure SQL Database, Azure SQL Managed Instance, and Enterprise or Developer edition (2022 or later at low priority, older versions plainly online). A database that already applied the migration under 1.23.0 needs nothing.
• SQL Server Standard, Express and Web, and PostgreSQL, still build it while holding writes back. On a large Audit Log, upgrade at a quiet moment, with no other Server instance still serving the same database.

Nothing on the wire changes in Servers 1.22.0 to 1.24.0, and none needs a Workstation upgrade.

Rollout order: Server first, then the fleet

The Server accepts two workstation protocol versions, its own and the one before it. A machine on an older protocol is refused with update required; one on a newer protocol is refused with update the Server first. Either refusal is recorded on the machine's fleet row and in the Audit Log as a failed workstation.connect, and a refused machine signs nothing.

Server	Accepts protocols	Workstation releases it accepts
1.6.x	1–2	1.0.x
1.7.0 – 1.8.0	2–3	1.0.x, 1.1.0 – 1.3.0
1.9.0 – 1.12.0	3–4	1.1.0 – 1.3.0, 1.4.x
1.13.0 – 1.14.1	4–5	1.4.x (connects, serves no new sessions), 1.5.0
1.15.0 – 1.24.0	5–6	1.5.0 (serves every session but a per-transaction one), 1.6.0 – 1.10.0

Workstation 1.0.x speaks protocol 2, 1.1.0 to 1.3.0 speak protocol 3, 1.4.0 to 1.4.2 speak protocol 4, 1.5.0 speaks protocol 5, and 1.6.0 to 1.10.0 speak protocol 6.

Workstation 1.8.0 and later go with Server 1.20.0 or later, although the protocol is the same. They send the Server a Workstation Report, which only Server 1.20.0 and later take, with messages up to 1 MB on the workstation channel: an earlier Server closes the connection of a report past 32 KB, which a machine on 1.8.0 with many tokens, middleware or errors can send. From Workstation 1.8.1 a machine stays connected to an earlier Server: until the Server has taken a report on a connection, it keeps its reports under 32 KB. Workstation 1.7.0 needs no Server change.

What a newer Workstation adds shows only on a newer Server, and an older Server ignores it, so either side may move first within these releases. Workstation 1.9.0's reason for a NO_TRAY is recorded from Server 1.22.0. Workstation 1.10.0's console, and its reason for a session-start PIN prompt it could not show, are kept and shown from Server 1.23.0, where a machine on an earlier release reads Needs Workstation 1.10.0 in the Console Account column. Servers 1.22.0 to 1.24.0 go with any Workstation in their window.

Within a protocol window the rule is: upgrade the Server first, then the workstations, within one release. A machine ahead of its Server cannot connect at all: the older Server refuses it with update the Server first, hands it no PKI licence, and every certificate on that machine reads UNTRUSTED in its Tray App until the Server catches up. A Server ahead of a machine that is still in its window is supported.

Protocol 5 was the exception: Server 1.13.0 and Workstation 1.5.0 moved together. On protocol 5 the Server no longer sends the client application's PIN to any workstation: it challenges each machine holding a candidate certificate, and checks the PIN Proof the machine answers with. A protocol 4 machine (Workstation 1.4.x) still connects to Server 1.13.0, is licensed, pulls the Trusted CAs, announces, signs for sessions already bound to it, and can be offered a release. It is never sent a PIN challenge or a prompt, so for session creation it counts as offline. A holder whose machines are all still on 1.4.x is answered 422 WORKSTATION_OFFLINE until one of them updates, and the Server logs a Warning each time it passes such a machine over. The upgrade is therefore an outage for every holder whose machines update late. A machine on 1.3.0 or older is refused outright.

Protocol 6 is Server first again. Server 1.15.0 refuses Workstation 1.4.x, and Workstation 1.6.0 cannot connect to a Server before 1.15.0. A machine on 1.5.0 connects to 1.15.0 and serves every session as before; only a session whose Client Application has Per-Transaction PIN on passes it over. A rollout to Server 1.15.0 or later, in order:

1. Back up the database.
2. Check the Workstations screen for any admitted machine still on 1.4.x, and move it to Workstation 1.5.0 first (offer the release, or install it by hand or by Group Policy). It will be refused at the handshake once the Server moves.
3. Upgrade the Server.
4. Upload the Workstation release on the Workstation Releases screen, 1.6.0 or later (1.8.0, 1.9.0 or 1.10.0 only once the Server is on 1.20.0 or later, and 1.10.0 with Server 1.23.0 or later to see the console), and offer it with Deliver to all not yet updated…. Machines whose Group Policy turns Server-delivered updates off take it by hand or by Group Policy.
5. Turn Per-Transaction PIN on for a Client Application only once its holders' machines run 1.6.0 or later, watching the Warning lines naming protocol 5 machines passed over.

Integrators whose Client Application does not use Per-Transaction PIN see nothing new, except that a pin sent on a signing call, which earlier Servers ignored, is now refused 400 INVALID_REQUEST.

A Server before 1.13.0 with a fleet on 1.4.x cannot go straight to 1.15.0 or later: the new Server would refuse every 1.4.x machine, and Workstation 1.5.0 cannot connect to a Server before 1.13.0. Go through Server 1.14.1 and Workstation 1.5.0 first, in this order, then follow the steps above:

1. Back up the database.
2. Plan for the machines the Server cannot update: those on Workstation 1.3.0 or older, and those whose Group Policy turns Server-delivered updates off. They need Workstation 1.5.0 by hand or by Group Policy, in the same maintenance window, because a 1.5.0 machine cannot connect to a Server before 1.13.0.
3. Upgrade the Server to 1.14.1.
4. Upload TrustBridge-Workstation-1.5.0-x64.msi on the Workstation Releases screen and offer it with Deliver to all not yet updated… on the Workstations screen. The upload has to wait for the upgrade: a Server refuses a release that speaks a newer protocol than its own.
5. Watch the fleet screen's Service version column and the Warning lines naming protocol 4 machines until every machine runs 1.5.0. Then, if you want vanished machines to read offline sooner, lower Workstations:ClientTimeout to 00:00:15: every machine that can connect to 1.13.0 pings often enough for it.

On that step nothing changes for client applications: the PIN still arrives in POST /v1/sessions, and from 1.13.0 it goes no further than the Server.

Upgrading workstations by hand is a newer MSI installed over the older one, with no properties. See Workstation installation. From Workstation 1.4.0 on, a machine can also take a release from its Server: see Updating the fleet from the Server.

Updating the fleet from the Server

Since Server 1.9.0, an Administrator can deliver a Workstation release to the fleet from the admin console, and a machine running Workstation 1.4.0 or later installs it by itself.

1. Upload the release on the Workstation Releases screen: the signed MSI, up to 256 MB. The Server reads its version, the protocol it speaks and its signer out of the file and stores it in the database. It refuses, by name and with nothing stored, a file that is not an MSI, an unsigned package, one signed by a certificate its Publisher Pin does not list, a version that is not three numbers, a package that declares no protocol (every release before Workstation 1.3.0), a protocol outside the Server's window, and a version already stored. Each refusal is a failed admin.workstation-release-uploaded row in the Audit Log. On App Service the upload is one request and must finish within the front end's four-minute timeout.
2. Offer it on the Workstations screen: Offer release… for the selected machines, or Deliver to all not yet updated…, which lists every machine the release can reach for you to untick before confirming. An offer reaches only an Admitted machine on protocol 4 or later, below the release, whose Publisher Pin lists the release's signer and whose Group Policy leaves updates on. The rest are named with the reason and keep what they held.
3. Each machine installs it on its own: after a random wait of up to ten minutes it downloads the package from the Server, verifies it, waits until no signature is in flight and no PIN prompt is open, and installs it. It reports the outcome on its next handshake, as installed, failed (with msiexec's exit status and the end of its log) or refused (naming the check that failed). Each outcome is one workstation.update-installed, -failed or -refused row in the Audit Log.

A failed or refused release is not retried until it is offered again. Fix the cause, then offer the same release to the machine again, or use Deliver to all not yet updated…, which offers again every machine it selects. A machine that was offline when it was offered again tries at its next handshake. An install that Windows Installer was too busy to start (exit 1618) is retried by the machine about fifteen minutes later.

Withdrawing stops every machine that has not started the install yet: Withdraw offer for the selected machines, Withdraw from all on the Workstation Releases screen for one release. A release offered to any machine cannot be deleted until its offers are withdrawn. A machine that already installed a release stays on it.

Who the Server cannot reach is named on the Workstations screen: Updates disabled by policy for a machine whose Take updates from the TrustBridge Server policy is off, and Needs a manual update for one whose version cannot take updates from the Server. Workstation 1.3.0 and older can never take a release this way. Those machines get the release by hand or by Group Policy.

Downloads have their own cap, WorkstationReleases:MaxConcurrentDownloads (20); a machine over it is told when to come back. They do not count against the client applications' throttle, so a rollout does not make the contract answer 429.

Rotating the Key Ring Protector

Rotation is additive: a new protector seals new keys and never re-wraps what is already stored.

1. Move the outgoing protector's base64 into KeyRingProtector__RetiredCertificates__0__Certificate (with …__0__CertificatePassword beside it if it has one). A second retired protector takes index 1, and so on.
2. Put the new one in KeyRingProtector__Certificate.
3. Restart. The KeyRingPosture line should now name the new thumbprint and 1 retired protector(s).
4. Keep the old PKCS#12 in your backups, permanently. There is no moment at which rotation "completes".

Never template an empty retired entry: an entry binding to an empty string is a refused start.

Renewing the PKI SDK licence

The licence lives on the Server alone, and the Server hands it to the whole fleet. Every Admitted workstation receives the Server's licence when it connects and holds it in memory only. There is nothing to set on the machines: no MSI property and no Group Policy value carries a licence.

The Lacuna PKI SDK licence expires on a date, and nothing warns before it does. Once it lapses, the Server logs a Critical line, every admin page carries a banner, and nothing in the fleet or the vault can sign: certificates report UNTRUSTED, and session creation is refused with CERT_NOT_ACTIVE. Read the expiry out of the file and put it on whatever calendar tracks certificate renewals:

base64 -d license.base64 | grep -o '<Expiration>[^<]*'

To renew, replace Certificates__Validation__License with the new base64 and restart the Server: on Docker, replace PKI_LICENSE in .env and recreate the container; on App Service with a Key Vault reference, add a new secret version and restart. The restart is what reaches the fleet: every connected machine reconnects and applies the new licence, and a machine that was offline picks it up when it next connects. A machine on Workstation 1.8.0 or later shows the licence it holds, valid, expiring, or expired or refused, in the Workstation Report on its detail panel.

Rotating the Installer Key

Only for a deployment that minted an Installer Key of its own. A Server accepts every key it lists, so a change is a rollout rather than a flag day: add the new public key beside the old one under Admission:InstallerKeys, restart the Server, re-point the machines' INSTALLER_KEY by a silent repair at their own pace, then drop the old entry and restart again.

Rotating API keys

Issue a new key for the Client Application on the Client Applications screen, deploy it to the client, then revoke the old one. Every accepted key of an application is equivalent, so the switch is invisible to sessions in flight.

Workstation Reports

A machine on Workstation 1.8.0 or later tells its Server about its own condition, unasked: after every handshake, and again a few seconds after anything in it changes. The Server keeps each machine's last report, so support can see a machine's trouble without being at it, and an Administrator can see the same trouble across the fleet:

• Fleet screen: a Report chip per machine, the worst of Reader failed, Token locked, N kinds of error (24h) and Not reported, with filters for the first two. The Dashboard lists readers left failed and locked tokens under What needs you.
• Detail panel: the readers left failed, each token and how close its PIN is to locking, each middleware and whether it loaded, the PKI licence as the machine holds it, its switches, and the errors the Workstation Service and the Tray App logged, grouped by template.
• Workstation Reports screen (/admin/workstation-reports): errors by template, tokens by model and middleware by version across the Admitted machines, each row opening the machines it counts.

From Workstation 1.10.0 the report carries the console: the Console Account signed in at the machine's physical console (its SID, and its DOMAIN\user name where it resolves) or nobody, whether that session is locked, whether a Tray App runs there and the version it announced, and how many other sessions have a Tray App, without their accounts. The Workstation Service reads the account and the lock itself, so they are reported while no Tray runs. The report is sent again within a few seconds of a sign-in, a sign-out, a lock or an unlock, a switch of user at the console, and a Tray App starting or ending, and not while the console is unchanged. Server 1.23.0 and later keep it and show it in the Workstations screen's Console Account column and the panel's At the console section (Admin console); an earlier Server ignores it. The Server keeps it for an offline and a revoked machine, and a report that could not read the console that time leaves the last one in place.

A report is not the Audit Log and writes nothing to it; the machine's own log still holds every line. Error groups no report has added to for WorkstationReports:ErrorRetentionDays (30 by default, counted on the Server's clock) are deleted by a daily sweep, and a machine's report goes with it when it is removed. A machine can be told not to report, by its MSI or by Group Policy (Workstation configuration); it then reads Not reported, as does every machine on a release before 1.8.0. Not reported is never a sign of a healthy machine. With reporting off, no account or other part of the console leaves the machine either, and the Server forgets the console it held.

Finding a machine's problems

Two things on the Workstations screen tell an operator which machines will fail their holders, before the holders call:

• Console Account, no Tray (Server 1.23.0 with Workstation 1.10.0) lists the online machines where somebody is signed in at the console and no Tray App runs there. Every per-transaction signing call and every session-start PIN prompt sent to such a machine is refused until a Tray runs. Ask the person at the machine to open TrustBridge from the Start menu (Workstation 1.9.0 and later), or to sign out and in again on an earlier release. Nobody at the console with 1 other session with a Tray beneath it is a holder on Remote Desktop, where no PIN prompt reaches.
• With refusals (24h) (Server 1.23.0) lists the machines with a Workstation Refusal in the last 24 hours that was a problem with the machine. Open the machine's panel: its Events tab opens on Refusals only and says each one in words, with what to ask the holder where the holder can fix it. A refusal from a machine before Workstation 1.9.0 says no Tray App could show the prompt and not why. See everything in the Audit log opens the machine's whole trail.

A support call usually starts from a person, not a machine: the Workstations screen's search finds machines by the caller's Windows login, their name or their CPF. A holder whose session will not start is under Refused creations (24h) on the Signing Sessions screen, with every machine the request prompted and what each answered; a failed signature is on the Signing Operations screen, in words. See Admin console, and Troubleshooting for each reason a machine gives.

Managing the fleet

• A machine that never appears in the queue is almost always a URL or a TLS chain problem; read its Application event log first. See Workstation installation.
• A lost, stolen or compromised machine is revoked on the fleet screen, whether or not it was uninstalled. Revocation does not need the machine's cooperation, and its key is refused for as long as its row stands.
• A machine that was retired, reimaged or renamed can be removed on the fleet screen: its row, its certificates and its Workstation Report go, while its signing history stays. If it is still installed and asks again, it arrives in the queue as a new machine. Removing a revoked machine lifts the refusal its key is held to, so leave a lost or compromised machine revoked. See Removing a machine.
• A reinstalled machine is a new machine with a new fingerprint and arrives in the queue again; remove the old row once nothing needs it. An upgrade or a repair is the same machine.
• A machine sent back to the queue and abandoned never lapses: remove it from the admission queue.
• A machine refused with update required runs a release older than the Server's window: on Server 1.15.0 and later, Workstation 1.4.x or older. Install the current release on it by hand or by Group Policy.
• A machine reporting Behind on the Trusted CAs column reconnects and re-pulls on its own.
• Clock drift. A machine whose clock is more than Workstations:ClockSkewTolerance (60 s) out cannot connect, and the Server's log names it at Warning: A workstation assertion was refused: Expired or NotYetValid, with the machine's key thumbprint and its offset, positive when the machine is behind the Server and negative when it is ahead. Time synchronisation is the fix.

The temporary directories

The Server writes three directories under /tmp inside the container and nothing else:

Path	Holds	If lost
/tmp/trustbridge-pki	The PKI SDK's cache of fetched issuers and CRLs, and the licence handed to the SDK as a file	Nothing. Validation re-downloads what it needs, at about 1.15 s per pass instead of 140 ms
/tmp/trustbridge-logs	The rolling log file the Server log screen reads	Nothing. Every line was written to the console first
/tmp/trustbridge-releases	A copy of each stored Workstation release, which the fleet's downloads are served from, and an upload while the Server reads it	Nothing. Each copy is rebuilt from the release's database row at the next download

A read-only filesystem is tolerated, with two exceptions. The SDK will only read a licence from a file, so a container with no writable temporary directory at all starts a Server whose licence the SDK never read: nothing can sign, and the admin console says so. And an upload on the Workstation Releases screen is written to trustbridge-releases before it is read, so where that directory cannot be written, or is full, every upload is refused by name and the Server log names the directory. Downloads still work without it, read from the database instead.

Mount a tmpfs over /tmp for the licence. The releases directory grows by about 100 MB per stored release and a tmpfs counts against the container's memory limit, so where releases are kept, mount a disk-backed volume over /tmp/trustbridge-releases itself. To move all three, set TMPDIR.

Delete the log directory only with the Server stopped: the sink holds the day's file open, and a file removed from under it keeps taking lines nobody can read until the next roll.

Backups, in one table

What	Where	Frequency
The database	Your SQL Server backups, or pg_dump on PostgreSQL (Server installation)	Continuous or daily, to your policy, and before every upgrade
The Key Ring Protector .pfx and every retired one	Off the platform, where irreplaceable things go	Once each; they never change
The TLS certificate (Docker)	Wherever your certificates live	Per issuance
The Server log stream	Your log pipeline's retention	Continuous

Restore the database and the protector together. Nothing else needs restoring; workstations reconnect, and their state lives on the machines. The stored Workstation releases, the Dashboard's Signing Activity and the machines' last Workstation Reports are in the database too.