Execute the isolated synthetic rehearsal after the S08 Identity dark launch,
its audited Project Management receiver prerequisite and S08A session-store
substrate, before ordinary staging authentication changes.
Purpose: Prove the complete native Identity/BFF behaviour and rollback topology without changing Auth0 defaults, real users, live data or production.
Output: Focused SyRF and cluster-gitops PRs, redacted full-matrix evidence, and a separately reviewed/synced teardown.
This was a two-stage teardown, following lessons 1–3 below. Lesson 4 was applied in #1508, which
set mongodb.com/atlas-resource-policy: delete. Lesson 5 is why the DNS records remain:
Stage 1: camaradesuk/cluster-gitops#1525 (merge 840fef7a, 02:29:21Z). It is the inverse of
#1508, keeping the three ClusterExternalSecret selectors, the plugins destination and the
PostDelete hook (without the PostSync create hook).
Stage 2: camaradesuk/cluster-gitops#1526 (merge b9e3ec88, 02:42:22Z) removed those once
absence was proven. All three of its paths are byte-identical to before #1508.
Preflight:preflight.sh --mode isolated-rehearsal-teardown on stage 2's head gives rc 0: 14
assertions, 0 residual references. The redaction check passes.
Read-only discovery after stage 1 synced:
Item
State
Rehearsal Applications
The four service apps were gone by 02:30:14Z, and extra-secrets-rehearsal by 02:31:22Z
AppProject syrf-rehearsal
Its own resources-finalizer held it until no Application referenced it. Gone at 02:32:16Z; root Synced/Healthy
Namespace syrf-rehearsal
Gone by 02:31:22Z, with no stuck ESO finalizer, because the CES still selected it
RabbitMQ vhost syrf-rehearsal
The PostDelete Job completed. The broker log shows exactly one vhost deletion, syrf-rehearsal, at 02:31:13Z
Atlas database users
Deleted. All three took the operator's delete path. There is no "Not removing Atlas database user" line, which on 2026-09-29 appeared for all three. Each logged Removed 1 connection secrets and released its finalizer; a failed Atlas delete keeps the finalizer. The Atlas API itself still answers 401 to read-only access, so this rests on operator logs and object status (camaradesuk/syrf#3834)
Databases syrf_rehearsal, syrf_identity_rehearsal
Absent from the Preview cluster's database list
Mailpit
Rolled at 02:29:40Z; Ready, 0 restarts. The rehearsal SMTP user, its ExternalSecret and its store are gone, and MP_SMTP_AUTH holds only the shared users
Ingresses
None with a rehearsal host
DNS rehearsal., identity.rehearsal., project-management.rehearsal.syrf.org.uk
Still resolve to the shared ingress. They are left for the GitOps-owned fix in camaradesuk/cluster-gitops#1380; nothing was deleted by hand
Ordinary staging and production
Staging API SYRF__IdentityProvider=auth0, Web SYRF__AuthProvider=auth0. Staging web, Identity discovery and API /health/ready all return 200, as does production web. API/PM/Web/Identity staging and API/PM/Web production are Synced/Healthy
Kept on purpose:
the GCP secret camarades-google-oauth-rehearsal and its Google client;
the S08A Valkey rehearsal user;
the operator's run wrapper and Playwright wrapper;
Read-only discovery (kubectl get/describe/logs, DNS lookups, the Preview cluster's database
list) after #1189 synced:
Item
State
Argo Applications (identity-, api-, web-, project-management-rehearsal) and the syrf-rehearsal AppProject/ApplicationSet
Gone by 13:13Z
Workloads, Ingresses, Certificates, ExternalSecrets, AtlasDatabaseUser and DatabaseLifecycle objects in syrf-rehearsal
Gone; no namespaced object remains
Databases syrf_rehearsal and syrf_identity_rehearsal
Dropped by DatabaseLifecycle cleanupOnDelete
Mailpit
Rolled once at 13:11:43Z, ready, 0 restarts. The rehearsal SMTP user, its store and its Secret are gone; the shared staging/preview users are unchanged
Valkey rehearsal ACL user
Retained on purpose (S08A substrate, kept for the redeploy); valkey-nonprod Synced/Healthy
syrf-rehearsal namespace
Gone at 13:31:56Z, after camaradesuk/cluster-gitops#1379 (see the lessons below). It was stuck Terminating from 13:12Z on two ESO ClusterExternalSecret finalizers
RabbitMQ vhost syrf-rehearsal
Removed. After camaradesuk/cluster-gitops#1381 restored the hook, the PostDelete Job syrf-rehearsal-rabbitmq-vhost-delete completed at 13:47Z; it exits 0 only on HTTP 204/404. extra-secrets-rehearsal finalized at 13:47:52Z
Atlas database users
Not removed from Atlas. The operator (2.13.2, --object-deletion-protection=true) logged "Not removing Atlas database user from Atlas as per configuration" for all three. It honours only mongodb.com/atlas-resource-policy, not the atlas.mongodb.com/deletion-protection: "false" annotation our charts use, and previews share the gap. Tracked in camaradesuk/syrf#3834. Read-only Atlas API access returned 401, so this is inferred from the operator logs
DNS rehearsal., identity.rehearsal., project-management.rehearsal.syrf.org.uk
Still resolve to the shared ingress, which returns 404. External-DNS runs upsert-only with registry: noop, so it never deletes records. Nobody deleted them by hand; tracked for a GitOps-owned fix in camaradesuk/cluster-gitops#1380
Ordinary staging and production
API/PM/Web staging and API/PM production Synced/Healthy. Staging API SYRF__IdentityProvider=auth0, Web SYRF__AuthProvider=auth0. Staging Identity discovery 200 and API /health/ready 200
Leftovers kept for the redeploy:
the GCP secret camarades-google-oauth-rehearsal and its Google OAuth client;
the Valkey rehearsal user and its valkey-nonprod-rehearsal store;
the operator's local run files (the synthetic accounts themselves were dropped with the database).
Atlas users left in Atlas (names only): syrf_rehearsal_app, syrf_identity_rehearsal and
syrf_identity_rehearsal_pm_ro. They are left in place for now; see camaradesuk/syrf#3834. The
preview-orphan-sweep CronJob never collects them, because it matches only ^syrf_pr_[0-9]+_app$.
A redeploy's AtlasDatabaseUsers use the same usernames.
Teardown chain:
camaradesuk/cluster-gitops#1189 (b027c886): the inverse, keeping two entries.
camaradesuk/cluster-gitops#1379 (019154cf): re-selected the namespace in two
ClusterExternalSecrets so ESO releases it.
camaradesuk/cluster-gitops#1381 (cda83b18): restored the PostDelete vhost hook.
camaradesuk/cluster-gitops#1382: removes every retained or restored entry once absence is proven.
Teardown lessons (apply to the redeploy's teardown)¶
Keep a namespace in ClusterExternalSecret selectors until the namespace is gone. ESO v1.0.0
puts a per-CES finalizer on each selected namespace. It removes that finalizer only while the
CES still selects the namespace (gatherProvisionedNamespaces); de-selecting only deletes the
ExternalSecret.
Keep AppProject destinations until the Applications have finalized. Argo CD deletes only
live objects the project still permits (getPermittedAppLiveObjects).
Keep PostDelete hooks at their git path until the Application has finalized. Argo CD renders
them from the Application's current target state, so deleting the path strands the
post-delete finalizer. A two-PR teardown avoids all three: first delete the ApplicationSet,
apps and config; then, after read-only proof of absence, delete the selectors, destinations
and hooks.
atlas.mongodb.com/deletion-protection: "false" has no effect. The operator honours only
mongodb.com/atlas-resource-policy (camaradesuk/syrf#3834).
External-DNS never deletes records (camaradesuk/cluster-gitops#1380).
Task 1: Prepare and validate the isolated rehearsal topologySyRF chart/package definitions, rehearsal preflight/tests and live harness; cluster-gitops isolated Application, namespace values, ExternalSecret/operator resources and inverse teardownUse separate isolated SyRF and cluster-gitops PR worktrees and extend only established chart/package plus Argo/Helm/Kustomize/ExternalSecret/operator patterns. Add closed, fixture-tested preflight modes for isolated provision and teardown, and extend the checked-in live harness with closed flags for the isolated route and forwarded-header matrix. Pin immutable validated API and Identity artifacts. Consume only S08A's rehearsal ACL user (~rehearsal:*) on the shared non-production Valkey, with bffAuth.redis.keyPrefix: "rehearsal:" and its namespace-local Secret contract; never use the staging credential or prefix (topology approved 2026-09-22). Declare a new restricted rehearsal route and dedicated namespace, Identity database, encryption/DataProtection material, OAuth clients, allowlisted callbacks, synthetic mail adapter references and least-privilege credentials. Render every resource and prepare a separate inverse GitOps PR before deployment. Reject floating tags, shared staging/production databases or clients, real-user/production/Auth0 exports, default Web/API provider changes, Terraform and manual cloud resources.SYRF_WORKTREE="\({SYRF_WORKTREE:?}" && GITOPS_WORKTREE="\)" && "\(SYRF_WORKTREE/scripts/auth-migration/assert-worktree.sh" "\)SYRF_WORKTREE" && "\(SYRF_WORKTREE/scripts/auth-migration/assert-worktree.sh" "\)GITOPS_WORKTREE" && bats "\(SYRF_WORKTREE/scripts/auth-migration/tests/scripts.bats" --filter 'isolated rehearsal' && "\)SYRF_WORKTREE/scripts/auth-migration/preflight.sh" --mode isolated-rehearsal --syrf-worktree "\(SYRF_WORKTREE" --gitops-worktree "\)GITOPS_WORKTREE" --environment staging --evidence /tmp/m005-s30-preflight.json && "$SYRF_WORKTREE/scripts/auth-migration/check-redacted-evidence.sh" /tmp/m005-s30-preflight.jsonApprove only when the provision and inverse diffs identify exact immutable revisions, bounded synthetic resources and route, separate data/secret/client domains, retained Auth0 defaults, and no production or ordinary staging Web/API mutation.The isolated topology and its inverse are reviewed, render cleanly and cannot receive default staging traffic or real-user data.
Task 2: Sync and run the complete synthetic live matrixNone (isolated live staging evidence only)After the sole operator confirms the restricted callback, mailbox and secret-reference prerequisites, merge/sync only the reviewed rehearsal Application. Create only designated synthetic identities. Run password sign-in, SyRF confirmation/resend, forced reset admission, passkeys, optional MFA and recovery, Google sign-in/link/unlink with step-up and notification, profile completion, token claims, API/BFF/Swagger authorization, sessions and SignalR. Exercise trusted and deliberately untrusted X-Forwarded-Proto/X-Forwarded-Host inputs through the actual ingress; generated callbacks and emailed links must use only the allowlisted rehearsal origin. Capture bounded pass/fail, revision and aggregate evidence; never record credentials, cookies, participant identifiers or secret/resource identifiers.SYRF_WORKTREE="\({SYRF_WORKTREE:?}" && "\)SYRF_WORKTREE/scripts/auth-migration/live-smoke.sh" --environment staging --expected-provider openiddict --isolated-rehearsal --require-two-replicas --require-forwarded-header-matrix --evidence /tmp/m005-s30-live.json && "$SYRF_WORKTREE/scripts/auth-migration/check-redacted-evidence.sh" /tmp/m005-s30-live.jsonApprove only with the exact Synced/Healthy rehearsal revision, all synthetic matrix rows green, no real-user email/data, and independent proof that ordinary staging Web/API still use Auth0.The isolated deployed topology passes every required synthetic parity and ingress scenario while Auth0 remains staging's default.
Task 3: Sync the isolated teardown and prove rollbackcluster-gitops isolated rehearsal Application and operator resources onlyMerge the separately reviewed inverse revision and wait for Argo reconciliation. Remove only the rehearsal Application/revision and synthetic-only resources according to their approved retention class. Do not disable the retained S08 Identity service or alter ordinary staging Web/API, Auth0, shared services, production, or real-user data. Use read-only discovery to prove the rehearsal Application, workloads, route, DNS and disposable synthetic state are absent; verify the retained staging Identity route and Auth0 defaults remain healthy.SYRF_WORKTREE="\({SYRF_WORKTREE:?}" && GITOPS_WORKTREE="\)" && "\(SYRF_WORKTREE/scripts/auth-migration/preflight.sh" --mode isolated-rehearsal-teardown --syrf-worktree "\)SYRF_WORKTREE" --gitops-worktree "\(GITOPS_WORKTREE" --environment staging --evidence /tmp/m005-s30-teardown.json && "\)SYRF_WORKTREE/scripts/auth-migration/check-redacted-evidence.sh" /tmp/m005-s30-teardown.jsonApprove only when the teardown has its own PR/revision/sync, no rehearsal object remains, the S08 Identity service is still healthy, and ordinary staging Web/API are still on Auth0.The synthetic topology is removed through GitOps with no effect on Auth0, retained Identity, ordinary staging, real users or production.
Fixture-tested provision/teardown preflight, exact rendered GitOps resources, full live parity and forwarded-header matrix, separate Argo revisions, and read-only absence evidence all pass.The isolated-rehearsal half of M005-R10 is complete; S09 may separately prepare an ordinary staging switch and Auth0 rollback.