Developer guide

Emergency recovery

When a release misbehaves in production, emergency policies switch a named feature off, make it read-only, show a maintenance message or ask for an update, in the apps already on your users' phones and on your own server, without shipping a build. You decide in the dashboard; the SDK fetches a signed policy document, verifies it against a key compiled into your app, caches it and evaluates it on the device.

What it is, and what it is not

• Feature guards, not code push. A policy only says whether a feature may run and what to show instead. Guards never download or run code. Signed patches are a separate early access feature with their own runtime. A bug in native code is fixed only by a new store release. A Flutter app built with Shorebird can take a Dart patch through Shorebird itself (ship the fix).
• Client guards improve what users see. A modified or old app can skip them. Anything that moves money or data must also be refused by your server, with the server guard.
• Not instant. Apps pick up a change at startup, when they come back to the foreground, and every refresh interval (60 seconds by default). Apps that are offline or in the background see it later.
• Replay. Capsules record the guard decisions the user saw, and a replay returns them, with source: replay, so a feature that was blocked during capture is blocked in the replay too. Nothing is fetched. A feature the capsule has no decision for is allowed, and so is every feature in a capsule from an older SDK. Decisions need an SDK release that supports them.

Before an incident

1. In the dashboard, open Emergency Recovery → Settings and create the signing key. Copy the trust anchor (bcek.v1…).
2. Add the trust anchor to your app's emergency configuration (below) and ship that build. Apps without it can't receive policies.
3. Choose feature keys for the flows you may need to stop, such as checkout.confirm_order or wallet.transfer. Keys are lower-case dotted names; a policy on checkout.* covers every key below checkout.
4. Mark keys that move money FAIL_CLOSED, in the app and in the server guard (failure modes).
5. Run a drill in a staging environment.

Add the guard to your app

await BugCapsule.emergency.initialize(const EmergencyConfig(
  trustAnchor: 'bcek.v1.…', // from the dashboard; compile it into the app
  environment: 'production',
  failureModes: {'checkout.*': EmergencyFailureMode.failClosed},
));

final decision = BugCapsule.emergency.evaluate('checkout.confirm_order');
if (!decision.allowed) {
  showUnavailable(decision.publicUserMessage); // decision.action: BLOCK, READ_ONLY, …
  return;
}

BugCapsule.emergency.initialize(applicationContext, EmergencyConfig.builder()
    .trustAnchor("bcek.v1.…")
    .environment("production")
    .failureMode("checkout.*", EmergencyFailureMode.FAIL_CLOSED)
    .build())

val decision = BugCapsule.emergency.evaluate("checkout.confirm_order")
if (!decision.allowed) {
    showUnavailable(decision.publicUserMessage)
    return
}

BugCapsule.emergency.initialize(EmergencyConfiguration(
    trustAnchor: "bcek.v1.…",
    environment: "production",
    failureModes: ["checkout.*": .failClosed]))

let decision = BugCapsule.emergency.evaluate(featureKey: "checkout.confirm_order")
guard decision.allowed else {
    showUnavailable(decision.publicUserMessage)
    return
}

• Initialization never waits for the network: the SDK loads and re-verifies its cached document and fetches in the background. Evaluation is synchronous and in memory.
• The SDK uses the connection you gave BugCapsule (an SDK token), or one in the emergency configuration. Guards keep working when capture is disabled.
• Partial rollouts bucket by a random installation id. Pass subject to bucket by your own stable, pseudonymous user id instead; never an email address or anything that identifies a person.

Decisions

A policy can target platforms, app version ranges and a percentage of installations. When several policies match a key, the one with the highest priority wins, then the most restrictive.

Failure modes

A verified document is valid for a limited time (one hour by default; the SDK refreshes long before). When there is no usable document, for example a fresh install that has never been online, each feature uses its failure mode:

The whole app never fails closed by default: one key at a time opts in.

Server enforcement

For Node.js servers, @bugcapsule/emergency-guard fetches and verifies the same document and refuses protected operations with a 503 problem response. Check right before the irreversible step, and use idempotency keys so a retried request never charges twice.

const guard = createEmergencyGuard({
  credential: process.env.BUGCAPSULE_SDK_TOKEN!,
  trustAnchor: process.env.BUGCAPSULE_EMERGENCY_ANCHOR!,
  environment: 'production',
  failureModes: { 'checkout.*': 'FAIL_CLOSED' },
});
await guard.start();

// In the request handler, before charging:
await guard.enforce({ featureKey: 'checkout.confirm_order', operation: 'write', subject: userId });

By default the guard refreshes every 5 seconds. With consistency: 'strong' it checks for a newer document right before deciding. Neither stops an operation that passed the check just before a block was published, so record the decision's revision with the operation and reconcile what committed after the block. READ_ONLY lets operation: 'read' through. For other server languages, enforce the same rule in your own code: never rely on the app alone.

During an incident

1. File an incident in Emergency Recovery and link the capsules that show the failure.
2. Create a policy for the narrowest key, platforms and versions that cover the bug. Review its estimated scope.
3. Request activation. In production a second person approves it; project owners can break glass with a reason when nobody else is reachable.
4. Watch the coverage: how many reporting installations applied the new revision.
5. Ship the fix (below), then restore the feature in stages.

Every step is recorded in the incident's timeline and an append-only audit trail.

Ship the fix

A recovery operation takes a fix from a candidate build to your users.

1. BugCapsule replays the incident's capsules on the candidate build.
2. A second person approves the deployment.
3. The approved build ships through a recovery provider, the one delivery mechanism you choose for this fix.

An admin sets providers up once, under Emergency Recovery → Settings → Recovery providers. Credentials are stored sealed and are never shown again.

Before anything ships, the recovery page runs the provider's compatibility checks. Each check passes, warns, fails, or can't tell:

• A failed check blocks the deployment.
• A warning or a check that can't tell needs an explicit acknowledgement, which is recorded with the deployment.

For example, a release built without Shorebird fails the check: it can only be fixed by a store release.

Afterwards the page shows the provider's own answers:

• When a provider doesn't answer, the page says so. The deployment is never marked done on a guess.
• Adoption comes from reporting installations, not from the store.

Deploying, rolling back or resolving never lifts a policy. Guarded features stay guarded until you restore them.

Some things no provider can do:

• Upload builds or submit them for review.
• Predict or speed up a store review.
• Take an installed binary back from devices.
• Deliver a native code change as a patch.

Restore in stages

Once a fix has shipped, you can lift a block gradually instead of all at once. On the incident page, choose Plan restoration on the active restriction.

• Stages. By default 1%, 5%, 25%, then everyone, each held 30 minutes. You can first lift the restriction in an internal environment (for example the one your internal builds use).
• Same users throughout. Users are bucketed the same way on every stage. Someone who gets the feature back at 1% keeps it at 5%. Your server guard computes the same buckets, as long as your app and your server pass the same subject (a pseudonymous user id).
• Guards. A guard is a health signal that would show the fix failing, for example:
  • the crash rate;
  • a transaction's error rate or p95 latency, from BugCapsule monitoring;
  • a value your backend reports, such as a payment failure rate.
  If a guard is breached, the full restriction returns for everyone at once. Optionally, the fix's deployment is rolled back too.
• Manual or automatic. A manual plan waits for someone to advance it after each hold. An automatic plan advances only when every guard has enough data from the current stage and none is breached. It never advances just because time passed or because updates were downloaded.
• Approval. In production a second person approves the plan once. Its stages and its rollback then run under that approval.

You can roll a plan back at any time. A paused plan is still watched. Cancelling stops the plan where it is: stages already lifted stay lifted.

Automation rules

Under Emergency Recovery → Automation, a rule watches one health signal. When the signal crosses its threshold with enough data, the rule files a suspected incident.

A rule can also propose a restriction. A proposal always waits for a person to approve it; automation never publishes or lifts a restriction by itself.

Before you make a rule active, run it in dry-run mode, or simulate it over the last day, to see when it would have fired.

AI assistance

On an incident, BugCapsule can draft:

• likely root causes;
• a risk classification;
• a recovery strategy;
• an explanation of a candidate fix;
• test proposals.

It is advice only. It never changes the incident, publishes a policy, starts a recovery or deploys anything. People act through the normal, approved flows. Proposed test code is a sketch to adapt, never run or committed by BugCapsule.

• Off by default. A project owner turns it on in Emergency Recovery → Settings.
• What is sent. The incident and its policies, recoveries, validation results, deployments and restoration plans, plus up to five linked capsules. You can also include a diff of the fix.
  • Capsule data is redacted again on the server, with your project's redaction keys.
  • Secrets are scrubbed from a pasted diff.
  • Money amounts are never sent.
  Each result shows exactly what was sent.
• Checked, not trusted. Every claim cites the evidence it is based on. Citations to evidence that wasn't sent are removed, and a strategy that can't reach the platform is marked unusable.
• Limits. Advice can be wrong, so verify it before acting. Redaction removes known secrets, not every piece of personal data in free text: keep AI assistance off if your data rules don't allow sending incident evidence to a model provider.

Update and hotfix adapters

A REQUIRE_UPDATE decision carries the first version without the restriction. The adapters help the guarded feature get users there through the platform's own mechanism. They also report what they know (with the policy receipts, at most once a minute, never during replay), which drives the compatibility checks and adoption figures above. No adapter installs a package, downloads native code or restarts your app.

// pubspec: bug_capsule_shorebird (builds made with `shorebird release` only)
await BugCapsule.emergency.initialize(EmergencyConfig(
  trustAnchor: 'bcek.v1.…',
  hotfixProvider: BugCapsuleShorebird(),
));

final result = await BugCapsule.emergency.hotfix!.downloadAvailablePatch();
if (result is HotfixRestartRequired) {
  // Active after the next start. When that happens is up to you.
}

// implementation("io.github.bugcapsule:bugcapsule-android-play:…")
BugCapsule.emergency.initialize(this, EmergencyConfig.builder()
    .trustAnchor("bcek.v1.…")
    .updateProvider(PlayEmergencyUpdateProvider.builder(this).build())
    .build())

if (decision.action == EmergencyAction.REQUIRE_UPDATE) {
    BugCapsule.emergency.updates.startUpdate(activity, decision)
}

BugCapsule.emergency.initialize(EmergencyConfiguration(
    trustAnchor: "bcek.v1.…",
    updateProvider: AppStoreUpdateProvider()))

if decision.action == .requireUpdate,
   let update = try? await BugCapsule.emergency.updates.checkAvailability(for: decision),
   update.isAvailable {
    _ = await BugCapsule.emergency.updates.presentUpdate(from: viewController)
}

Signed patches (early access)

A signed patch changes logic or layout that your app registered as patchable when it was built, without a new store release. It can't reach anything else in the app.

Still needs a store release: Kotlin, Java, Swift or Objective-C code, unregistered Dart, new modules, surfaces, components or actions, plugins and native dependencies, permissions and entitlements, runtime or Flutter updates, and irreversible data migrations.

• Bound to one build. A patch is signed for an exact base build, and the app verifies the signature against a key compiled into it. Any other build refuses the patch.
• Bounded at runtime. Patched logic runs in a restricted interpreter with resource limits. If it fails, the app falls back to the code it shipped with.
• Validated first. Outside development, a patch must pass validation against recorded capsules, regressions included, before it can be deployed.
• Rolled out in stages. Each stage compares sessions on the patch with sessions on the base build. Too little data holds the rollout, and a higher crash rate halts it. In production, a second person approves.
• Rollback is a signed action. You publish it from the dashboard. Devices need a connection to receive it and may apply it on the next start, so a published rollback doesn't mean every device has recovered.

So far this has been tested on physical iPhone release builds (Flutter and native Swift) and on Android emulators, not on physical Android devices.

Signing key rotation

Create the next key in Settings; the trust anchor then lists two keys. Ship apps with the new anchor, and promote the next key once enough installations carry it. Apps that only know the old key keep their last document until it expires, then use their failure modes.