Replay & verify

Regression tests in CI

A capsule you've verified as FIXED becomes a regression test: the exact recorded session, replayed against every new commit. regress run reads only local files, so CI needs no credentials.

Add tests

bugcapsule regress add <id> --name "coupon over subtotal"   # downloads the capsule
bugcapsule regress list
git add bugcapsule/ && git commit -m "Add coupon regression capsule"

This writes bugcapsule/regressions.json and bugcapsule/regressions/<capsuleId>.bugcapsule under the project root (-p). Commit both. Capsules are redacted on device, but read one with bugcapsule inspect --json before committing it to a shared repository.

Run them

bugcapsule regress run -n 2          # every active test, 2 attempts each; exit 1 on any failure

Verify verdict	Recorded as	Fails CI?
FIXED	pass	No
STILL FAILING	fail	Yes
INCONCLUSIVE, NOT REPRODUCED, REPRODUCED	inconclusive	Yes
FLAKY	test status → `quarantined`	No; skipped next time unless `--include-quarantined`
INCOMPATIBLE	test status → `stale`	No; the state schema changed, re-capture it

regress run updates each test's lastRun in regressions.json. In CI, either discard that change or upload the file as an artifact to review quarantined and stale tests.

Because replay is strict, a regression test breaks when a deliberate change makes the app send new requests in that flow (network.unrecorded → inconclusive). Re-capture the capsule on the new behaviour and replace the test; don't switch to --lenient-network in CI, which would send requests to real services.

CI recipes

name: bugcapsule regressions
on: [pull_request]
jobs:
  regress-web:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - uses: subosito/flutter-action@v2        # provides dart for the CLI
        with: { channel: stable }
      - run: npm ci
      - run: npx playwright-core install --with-deps chromium
      - run: dart pub global activate --source git https://github.com/draz26648/bug_capsule_flutter_sdk --git-path packages/bug_capsule
      - run: bugcapsule regress run -n 2        # builds and serves via bugcapsule.json, replays in Chromium

jobs:
  regress-android:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with: { distribution: temurin, java-version: 17 }
      - uses: subosito/flutter-action@v2
        with: { channel: stable }
      - run: dart pub global activate --source git https://github.com/draz26648/bug_capsule_flutter_sdk --git-path packages/bug_capsule
      - name: Enable KVM
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm.rules
          sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
      - uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 35
          arch: x86_64
          script: bugcapsule regress run -n 2   # device must be unlocked; no screen-lock PIN

jobs:
  regress-ios:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4
      - uses: subosito/flutter-action@v2
        with: { channel: stable }
      - run: dart pub global activate --source git https://github.com/draz26648/bug_capsule_flutter_sdk --git-path packages/bug_capsule
      # bugcapsule.json names the scheme, replayTest and simulator; Debug must define DEBUG
      - run: bugcapsule regress run -n 2 -d "iPhone 16"

jobs:
  regress-flutter:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4
      - uses: subosito/flutter-action@v2
        with: { channel: stable }
      - run: flutter pub get
      - name: Boot a simulator
        run: xcrun simctl boot "iPhone 16" || true
      - run: dart run bug_capsule:bugcapsule regress run -n 2 -d "iPhone 16"

These are starting points, not tested workflows: runner images and simulator names change. Keep the per-attempt timeout generous on shared runners (--attempt-timeout or timeouts.attemptSeconds).

Recording replay and verify results from CI

run and verify on a capsule from your project record their result automatically, so it shows in the dashboard (use --report-to <id> when you replay a local file). In CI, give them credentials through the environment rather than bugcapsule login:

      - run: bugcapsule verify ${{ inputs.capsule_id }} -n 5
        env:
          BUGCAPSULE_URL: https://api.bugcapsule.com
          BUGCAPSULE_SECRET_KEY: ${{ secrets.BUGCAPSULE_SECRET_KEY }}   # a CI secret, never in the repo