Skip to content

Screening source-write development benchmark

ProjectScreeningWriteBenchmark measures screening source saves on a real MongoDB replica set in three arms, selected per cell by the ScreeningWriteArm enum:

Arm (artifact metric) What one submission executes
SourceOnly (*_source_only) Statistics writes off. With ReviewEligibilityPolicy off: the capacity-save overload with tracking off, a version-filtered FindOneAndReplace (find + findAndModify). With it on: the controller's flag-off branch, an uncached Project read per attempt and TrySaveActivityReviewAsync (its eligibility transaction and Project token write).
Transactional (*_materialized) Unchanged from earlier artifacts: the six-argument capacity save inside the statistics snapshot transaction, with the durable-mode read, envelope and receipt pre-check and the real screening writer and coordinator. ReviewEligibilityPolicy off only.
Fold (*_fold) ProjectScreeningFoldSave exactly as ReviewController.TrySaveScreeningOnFoldPathAsync drives it with materializedProjectStatisticsFold on and the project in fold mode, over a target that mirrors ReviewScreeningFoldTarget: private load with the persisted inclusion statuses, durable-mode and control reads, duplicate checks, deadline, free fold-only retries and the three-attempt budget, then SaveScreeningOnFoldPathAsync (or TrySaveActivityReviewAsync with eligibility on). The real ProjectStatisticsFoldWorker drains after every batch, outside the timed window.

Every cell runs with ReviewEligibilityPolicy off and on. With it off, all three arms run; with it on, SourceOnly and Fold run (*_eligibility_source_only, *_eligibility_fold). The transactional arm is not run with eligibility on: the gate does not compare against it, and running it would need a new driver path rather than the unchanged historical one.

Like-for-like. With tracking off, the fold arm's write is the plain version-filtered FindOneAndReplace; the SourceOnly arm's capacity overload with tracking off issues the same command with the same filter, minus statistics, which is why it is the gate's baseline. The controller's own flag-off branch with tracking off uses TrySaveExistingAsync (ReplaceOne, one update) instead; both are one write round trip. The controller's GetOrDefaultAsync before TrySaveScreeningAsync (cached process-wide for two seconds) is outside every arm: SourceOnly and Transactional time their own source load, which the fold arm replaces with its private load, as the design's command budget counts it. Each request owns a separate repository and cache, reloads after an optimistic conflict and keeps the API's three-attempt limit. Reviewer membership and authorization (except the eligibility stage check), event dispatch and HTTP are outside the fixture.

The minimum corpus, PS-WRITE-01, contains one screening project and ten studies. Each matrix cell uses 1, 2, 5 or 10 reviewers submitting to the same study or separate studies. Later rounds alternate inclusion/exclusion decisions by the same reviewers (the reviewer-correction pattern). With eligibility on, every reviewer is an active member with the stage Review grant. The artifact's corpus counts describe maximum matrix dimensions, not a claim that every cell has ten active reviewers. This is deliberately not a large-project PS-DS-02 benchmark. Scale values other than 1 are refused.

Reproduce

From the PR worktree root:

SYRF_STATS_DATASET=PS-WRITE-01 SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10 \
SYRF_STATS_RESULTS_DIR=/tmp/syrf-screening-write-results \
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj \
  --filter FullyQualifiedName~ProjectScreeningWriteBenchmark

One run measures every arm in every cell, with ReviewEligibilityPolicy off and on; there is no per-arm variable. Command counting is opt-in. Add SYRF_STATS_COUNT_ROUNDTRIPS=1 to the command above for the command breakdown; leave it unset for a no-subscriber control timing. Without the dataset variable, ordinary CI skips this benchmark. The existing replica-set fixture owns and removes its mongo:8.0 container. Existing shared-runner resource limits apply when SYRF_TEST_JOB_KEY is set; the artifact records that policy and the runtime/server environment. No deployed database, credentials or live flags are needed. Run it only on an idle host: the programme's threshold is a one-minute load average per CPU of at most 0.5.

Each cell's gate (b) verdict is in the artifact note <cell>_fold_gate_b (and <cell>_eligibility_fold_gate_b); the historical transactional gate stays in <cell>_p95_gate.

What the harness now records

Following the write-overhead diagnosis (screening-write-overhead-diagnosis.md, #3255), three things changed in what this harness reports. The artifact schema is extended, never renamed, so an existing reader of Durations, Notes or the *_outcomes keys is unaffected.

  • Command counting is opt-in via SYRF_STATS_COUNT_ROUNDTRIPS. A CommandStartedEvent subscriber forces the driver to materialise every command document, and the materialised arm issues many times more commands than the source-only arm, so the recorder's own cost is asymmetric between the two arms being compared. Running once without it gives a no-subscriber control timing. When it is off, commands and commit_attempts read not_counted rather than zero, and no RoundTrips record is emitted.
  • Commands are broken down by driver command name, in the artifact's existing RoundTrips field, so a reduction is attributable to a specific command class rather than only to a batch total. The per-name counts are windowed exactly as the total is, so they sum to commands.
  • Per-attempt command counts are recorded in the *_commands_per_attempt note, for the single-reviewer cells only. With two or more reviewers the submissions run concurrently against one shared counter, so a per-attempt delta would include another reviewer's commands; the artifact says so rather than publishing a figure it cannot support.

The duplicate pre-check's assertion also moved outside the timed window. ResolveByReceiptAsync itself stays inside it, because production runs the same pre-check before opening a transaction.

Evidence and interpretation

The existing BenchmarkResults format records p50/p95 submission duration and repository-attempt duration. Submission time includes source reload, envelope/receipt reads, classification and retries. It excludes seeding, assertion queries, HTTP middleware, assignment, event dispatch and background work. The source-only baseline performs no statistics admission reads. Candidate/baseline ordering alternates by reviewer count. Each variant has its own warmup.

Every batch checks exact persisted decisions against successful submissions, and enabled batches also check exact authoritative/projection parity, operation receipt/delta presence, committed projection revision advancement and the coalesced outbox's client revision. Exhausted retries are reported as failures to save. A repeated decision can save source without a classified delta; these saves are counted separately and correctly have no statistics receipt/delta. There is no silent exception-to-success conversion. Driver command monitoring counts all commands during submissions, commit attempts and UnknownTransactionCommitResult failures; it excludes seeding and post-batch assertion queries. There is no injected unknown-commit failure in this timing run.

The p95 overhead gate is strictly less than 10%. Its result is emitted for each matrix cell, never asserted as a CI correctness test on a shared development host. All-submission latency includes exhausted requests, so a lower p95 with a worse success rate cannot establish a usable performance improvement. The artifact includes saved, conflict, exhausted and no-delta counts so that tradeoff remains visible. Maximum serialized summary bytes are also recorded.

This slice keeps active-reviewer tracking off and enables only ProjectScreening maintenance. It does not establish reviewer-family maintenance overhead, reservation/assignment throughput, annotation write cost, non-capacity API latency, large-project capacity, live rollout consistency or production acceptance. Those required programme measurements remain ordered follow-up evidence. Existing deterministic replica-set tests separately prove the non-capacity transaction conflict response and rollback.

Development result: 5 September 2026

The 100-iteration artifact records 10 warmup batches per variant. All exact correctness assertions passed. Every p95 overhead gate failed. These measurements leave the performance acceptance gate open.

Target Reviewers Baseline p95 ms Materialized p95 ms Overhead Baseline saved Materialized saved
Same study 1 8.57 64.64 654% 100/100 100/100
Same study 2 11.24 47.56 323% 200/200 117/200
Same study 5 20.35 44.77 120% 300/500 141/500
Same study 10 28.90 44.55 54% 329/1000 146/1000
Different studies 1 6.19 48.65 686% 100/100 100/100
Different studies 2 6.14 94.55 1441% 200/200 200/200
Different studies 5 7.71 80.96 950% 500/500 350/500
Different studies 10 21.89 134.89 516% 1000/1000 598/1000

The different-study result exposes contention on shared project statistics despite independent source records: 1,406 optimistic conflicts and 402 exhausted submissions in its ten-reviewer candidate cell. No unknown-commit failure was observed. Summary documents remained at most 1,982 bytes in this small corpus. The artifact's no_delta_saves field counts successful source saves without a statistics transition, so it includes every source-only baseline save; it is not a count of unchanged source data.

Required continuation #3255 owns the bounded write-path performance and contention improvement before claiming activation readiness. Increasing retries alone would trade exhausted requests for more load and latency; it does not establish the required overhead threshold. Re-run this matrix after the chosen change and add the required reviewer-family and active-tracking variants as their writers become available. The present slice does not silently expand maintenance or alter the existing three-attempt source contract to make its numbers pass.

Dependency integration measurement: 13 September 2026

After merging main and reviewer-maintenance prerequisite #3297 at 0803a4ec5, the opt-in smoke run (3 measured batches, 1 warmup) passed. A separate 30-batch artifact then ran all eight cells with five warmup batches per variant against disposable MongoDB 8.0.28 on .NET 10.0.0. Each cell still has one project and ten studies, and only ProjectScreening is enabled. The measured total is 1,080 submissions per variant; warmup is excluded. Every source/projection, receipt/delta and outbox correctness assertion passed. No unknown-commit failure was observed.

All eight overhead gates failed again. Thirty batches are a modest local measurement, not a large-project, current-controller, staging or production acceptance result. The historical 100-iteration artifact above remains unchanged. This run records that dependency revision; it does not establish a comparable host-controlled trend against September 5.

Target Reviewers Baseline p95 ms Materialized p95 ms Overhead Baseline saved Materialized saved
Same study 1 11.10 84.44 660.79% 30/30 30/30
Same study 2 24.68 57.83 134.28% 60/60 36/60
Same study 5 26.49 52.90 99.70% 90/150 46/150
Same study 10 30.73 60.30 96.22% 97/300 44/300
Different studies 1 8.40 62.83 647.97% 30/30 30/30
Different studies 2 8.46 93.91 1010.16% 60/60 60/60
Different studies 5 11.36 96.04 745.78% 150/150 105/150
Different studies 10 14.60 86.96 495.69% 300/300 176/300

Baseline exhausted 263 of 1,080 submissions after 1,005 total optimistic conflicts; materialized exhausted 553 after 1,883 conflicts. Exhaustion is a failed save, never counted as success. During measured submissions the driver recorded 4,649 baseline commands versus 28,104 materialized commands, including 527 materialized commit attempts. Those command counts exclude assertion queries and seeding. The higher command count and contention are observed diagnostic evidence; this slice introduces no unmeasured partitioning or write-protocol changes to chase the threshold.

Reproduce this sample with the command above using SYRF_STATS_ITERATIONS=30 and SYRF_STATS_WARMUP=5. Focused integration validation additionally passed 67 tests covering durable mode saves, optimistic contention and reviewer materialization; with no dataset variable the benchmark was skipped as intended. Performance follow-up remains #3255, and rollout stays gated by the failed write-overhead criterion and the separate live acceptance requirements.

The later integration of #3297 at 266a8dac3 includes Sonar fixes, legacy null-list recovery and a real Project-document admission write for reviewer maintenance. The recorded artifact predates those fixes. Reviewer maintenance remains disabled in this benchmark; measuring its additional write and contention requires a separately enabled workload before rollout acceptance. No fresh performance measurement is claimed by this integration.

Paired run 2026-09-15 (loaded host)

main at 0fe1716f6 and #3475 at 6546bb25d were run back to back, same host, same minute, SYRF_STATS_ITERATIONS=30 SYRF_STATS_WARMUP=5 SYRF_STATS_COUNT_ROUNDTRIPS=1, MongoDB 8.0.28 on .NET 10.0.0. Artifacts: main arm, #3475 arm.

Both the pre-#3475 main arm and the post-#3475 candidate arm ran on the same loaded host; neither was an idle-host fixture. Load average was 20-45 with roughly 130 CI containers running throughout both arms. This measurement therefore fails the hygiene rule this programme set for itself — "run unchanged main and the candidate back to back on the same idle host" — in one specific respect: the p95 columns below are indicative only and no ratio derived from them may be quoted. The command, conflict, exhaustion and saved counts are deterministic work counts rather than timings and are the reliable part of this run.

Cell Commands (main → #3475) Conflicts Exhausted Saved Materialised p95 ms Source-only p95 ms
same_study 1 900 → 630 0 → 0 0 → 0 30 → 30 118.72 → 56.53 58.55 → 12.62
same_study 2 1390 → 1090 82 → 80 25 → 25 35 → 35 95.67 → 53.62 13.29 → 17.18
same_study 5 2812 → 2553 343 → 343 111 → 111 39 → 39 84.33 → 56.52 15.73 → 26.41
same_study 10 5196 → 4974 779 → 770 251 → 246 49 → 54 58.99 → 71.41 41.99 → 30.72
different_studies 1 900 → 630 0 → 0 0 → 0 30 → 30 47.91 → 42.95 6.44 → 8.68
different_studies 2 2612 → 1860 58 → 60 0 → 0 60 → 60 103.98 → 75.32 38.86 → 5.90
different_studies 5 5090 → 3779 193 → 193 44 → 43 106 → 107 103.06 → 93.50 7.31 → 5.85
different_studies 10 9296 → 7142 434 → 428 125 → 118 175 → 182 94.07 → 98.10 9.94 → 10.15

What this establishes

  • The command reductions are real and match the unit measurement. The uncontended cells fall 900 -> 630 over 30 submissions, i.e. 30 -> 21 commands per save, exactly the figure ProjectScreeningWriteRoundTripBudgetTests asserts. The candidate artifact's per-command-name breakdown for that cell is find 390, update 120, insert 60, findAndModify 30, commitTransaction 30 — 13 finds, 4 updates, 2 inserts, 1 findAndModify and 1 commit per save — and its per-attempt note records 16 commands inside the repository save, leaving 5 before the transaction.
  • The contended cells drop by more in absolute terms, because a conflicting attempt also does less wasted work: 2612 -> 1860 at two different-study reviewers, 5090 -> 3779 at five, 9296 -> 7142 at ten.

What this does not establish

  • A4's contention effect is not demonstrated. Conflicts and exhaustion are essentially unchanged (58->60 / 0->0; 193->193 / 44->43; 434->428 / 125->118). Ordering the contended writes first was expected to cut the work a loser does, not the number of losers, and that is what the counts show — a loser still loses. The small movements in saved (175 -> 182 at ten different-study reviewers, 49 -> 54 at ten same-study) are within the noise of a loaded host and are not claimed as an improvement.
  • No latency claim. Materialised p95 is lower in six of the eight cells and higher in two, but the source-only p95 of the same cell moved by as much as 6.6x between arms (different_studies 2: 38.86 ms on main against 5.90 ms on the candidate) purely from host load. That is why the artifacts' own *_p95_gate overhead percentages are not reproduced here: under this load they measure the scheduler, not the change.
  • Every one of the eight <10% overhead gates still fails, in both arms. Nothing here moves the performance acceptance gate.

At the time, an idle-host re-run was owed before any rollout claim; it was run on 2026-09-30 (see Idle-host run 2026-09-30 below) and the gate still fails. The defensible statement about this change remains the one the command counts support: nine fewer round trips per conflict-free save, and proportionally less wasted work per losing attempt.

Idle-host run 2026-09-30

main at 65ae1099b, MongoDB 8.0.28 on .NET 10.0.0, PS-WRITE-01 at SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10, exactly the Reproduce command above. Three sequential runs from one scratch worktree, one build:

  • (a) timing, no command counting: artifact.
  • (b) command count, SYRF_STATS_COUNT_ROUNDTRIPS=1: artifact.
  • © repeat of (a), for run-to-run variance: artifact.

Reviewer maintenance is not an arm and there is no env toggle for it. The harness fixes ActiveReviewerTrackingEnabled = false and toggles only MaterializedProjectStatisticsWrites between the two arms, so no run of this benchmark measures enabled reviewer maintenance. STATUS asks for it; that measurement still needs a separately enabled workload and is not claimed here.

Host conditions

The machine is also the CI runner host (48 CPUs). The doc's idle threshold is load per CPU at most 0.5.

Run Load average at start Peak 1-minute load during/after Peak load per CPU Docker containers running
(a) timing 4.42 7.88 0.16 56-58
(b) counting 7.53 10.75 0.22 57-58
© repeat 7.78 14.85 0.31 57

Load stayed below the threshold in every snapshot, so no waiting was needed. About 57 CI containers were present throughout but mostly idle; the load-average figures are the evidence, not a claim of an empty host. The 15 September runs above had load 20-45.

Results (run a)

Overhead is materialized p95 over source-only p95 minus one. The gate is strictly under 10%.

Target Reviewers Source-only p50 ms Source-only p95 ms Materialized p50 ms Materialized p95 ms Overhead Gate
Same study 1 4.08 7.76 37.51 61.21 689% FAIL
Same study 2 5.75 11.12 31.40 43.85 294% FAIL
Same study 5 10.99 19.07 25.63 43.95 131% FAIL
Same study 10 18.87 30.20 28.80 44.24 46% FAIL
Different studies 1 4.79 5.82 35.40 39.31 575% FAIL
Different studies 2 4.90 5.94 42.82 81.20 1268% FAIL
Different studies 5 4.70 5.92 43.89 81.84 1283% FAIL
Different studies 10 5.15 13.88 43.54 84.29 507% FAIL

Run-to-run variance (overhead per cell in runs a, c and the counting run b, whose subscriber adds its own asymmetric cost): same study 689/611/596, 294/293/289, 131/128/116, 46/51/53; different studies 575/662/766, 1268/1426/1438, 1283/1366/1167, 507/1088/570. The verdict is stable; the individual percentages are not, especially the source-only p95 in the low-millisecond cells (a 1 ms shift in a 5-6 ms baseline moves the ratio by 15-20%). Do not quote a single percentage to more than order of magnitude.

Outcomes and command counts

Saved, conflicts and exhaustion are from run a; commands are from the counting run b (100 batches, so submissions per cell are 100 times the reviewer count).

Target Reviewers Source-only saved Materialized saved Source-only conflicts / exhausted Materialized conflicts / exhausted Source-only commands (per submission) Materialized commands (per submission)
Same study 1 100/100 100/100 0 / 0 0 / 0 200 (2.0) 2500 (25.0)
Same study 2 200/200 116/200 100 / 0 268 / 84 700 (3.5) 4143 (20.7)
Same study 5 302/500 142/500 896 / 198 1125 / 358 3300 (6.6) 9172 (18.3)
Same study 10 310/1000 151/1000 2376 / 690 2612 / 849 7724 (7.7) 17669 (17.7)
Different studies 1 100/100 100/100 0 / 0 0 / 0 200 (2.0) 2500 (25.0)
Different studies 2 200/200 200/200 0 / 0 198 / 0 400 (2.0) 7600 (38.0)
Different studies 5 500/500 350/500 0 / 0 646 / 150 1000 (2.0) 15513 (31.0)
Different studies 10 1000/1000 601/1000 0 / 0 1401 / 399 2000 (2.0) 28782 (28.8)

No unknown-commit failure occurred and every parity, receipt and outbox assertion passed. Exhaustion is a failed save; a lower materialized p95 in a contended cell does not offset its lower success rate.

Verdict: FAIL, on an idle host

All eight cells fail the under-10% p95 gate, with overhead from 46% (ten reviewers, same study) to 1,283% (five reviewers, different studies). The host met the doc's idle definition, so unlike the 15 September paired run these timings are acceptance-grade evidence, and they say the gate is not met. Only the single-reviewer cells are free of contention; they fail by 575-689%, so the cost is structural per-save work and not contention alone.

Different-study cells regress the most in relative terms: source-only saves never conflict there (0 conflicts at every reviewer count), while the materialized arm exhausts 150 of 500 submissions at five reviewers and 399 of 1,000 at ten because every save contends on the shared per-project statistics rows, which drops materialized success to 60% of submissions at ten reviewers.

Cost drivers

Uncontended (one reviewer), the materialized save issues 25 commands against 2 (find + findAndModify) source-only. Names are from the counting run's breakdown; the collection order comes from ProjectScreeningWriteRoundTripBudgetTests output at the same commit.

Command Source-only per save Materialized per save Added
find 1 16 +15
update 0 4 +4
insert 0 2 +2
aggregate 0 1 +1
commitTransaction 0 1 +1
findAndModify 1 1 0

This is 25 commands per save, up from the 21 recorded for #3475 on 15 September: the four added commands are the bounded ledger admission probes (aggregate and find on pmProjectStatisticsRevision) and the two pmProjectStatisticsReceiptRetirement reads (receipt retirement guards), all merged after that measurement. In the table these four are the aggregate row's +1 and three of the find row's +15 (one revision probe plus the two retirement reads); the rest of the table was already present in the 21-command count. Under contention the materialized arm also issues one abortTransaction per lost attempt (for example 1,396 aborts at ten different-study reviewers), each preceded by the full read set again.

Top three added command types per save: find (+15), update (+4), insert (+2). The 15 extra reads dominate: the traced order shows pmProjectStatisticsGlobalControl read 3 times (pre-transaction, envelope authority capture, in-transaction), pmProjectStatisticsControl, pmProjectStatisticsSourceOperationReceipt, pmProjectStatisticsReceiptRetirement and pmProjectStatisticsPublicationGuard each read twice (once for the envelope/pre-check, once again inside the transaction), plus two pmProjectStatisticsRevision reads.

Optimisation candidates (not implemented)

Each is grounded in the traced sequence above and needs its own correctness review, since the duplicate reads exist to close races between the pre-check and the transaction.

  1. Read pmProjectStatisticsGlobalControl once per submission, not three times. Hold the snapshot captured in BeginOperationAsync and re-validate it inside the transaction with the write's own version guard (or one read whose result the transaction reuses) instead of three separate finds.
  2. Fold the pre-transaction duplicate pre-check into the transaction. ResolveByReceiptAsync and receipt-retirement preflight read the receipt and retirement collections, and the transaction reads both again. A single in-transaction read, with the duplicate result returned from the transaction, removes roughly four finds per save at the cost of opening a transaction for a duplicate.
  3. Combine the Control and PublicationGuard checks into the same read as the receipt admission (one query against a single guard document per project, or one $lookup) instead of one find per guard family.
  4. Merge the two update writes on Control and PublicationGuard, and the read-then-update on the NotificationOutbox, into a single upsert (the outbox already uses a coalesced slot). The 4 updates and 2 inserts are the writes the transaction has to keep, so the saving here is the paired read, not the write.
  5. Address contention separately from round trips. In the different-study cells every save writes the same pmProjectStatisticsCurrent and Control rows, so conflicts and exhaustion (399 of 1,000 at ten reviewers) do not fall with command count. Options: shard the statistics row by study bucket, or accumulate deltas in the already-written revision/ledger rows and fold them asynchronously. This is the design question behind #3255 and #3251, not a local tweak.

Even a large round-trip reduction is unlikely to reach under 10% while a save must run a multi-document transaction against a source path that is a single findAndModify; the gate as written may need Chris's explicit decision, as STATUS already notes.

Limits

  • One project, ten studies, capacity-save path only. Not the current controller path, not HTTP, not staging or production.
  • Reviewer maintenance and active-reviewer tracking are off; their overhead remains unmeasured.
  • The source-only p95 in low-millisecond cells is noisy, so percentages differ by up to 2x between identical runs (see above). The pass/fail verdict does not.

Fold arm run 2026-10-02 (slice 6, gate b)

Async point-fold slice 6 (design, section 9): the three-arm matrix with ReviewEligibilityPolicy off and on, at 75498052f on #3949 (stacked on the slice-6a P0 branch, with slices 3, 4 and 5 merged), MongoDB 8.0.28 on .NET 10.0.0, PS-WRITE-01 at SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10, exactly the Reproduce command above, one build:

  • (a) timing, no command counting: artifact.
  • (b) command count, SYRF_STATS_COUNT_ROUNDTRIPS=1: artifact.
  • © repeat of (a), for run-to-run variance: artifact.

Every run passed every per-batch check: exact persisted decisions; for the transactional arm, parity, receipts, deltas and the outbox as before; for the fold arm, acceptance 3 after every batch, warmup included (overlay before the drain = authoritative = the legacy FullStats ProjectScreening section; pending entries = the batch's entry-carrying saves; the real worker drains; stored row = authoritative; one receipt per consumed entry; no pending set left).

Host conditions: not acceptance-grade timing

The machine is the CI runner host (48 CPUs); the threshold is a one-minute load per CPU of at most 0.5 (load 24). From 22:22 to 00:00 UTC the host was polled every 90 seconds and each run was started the moment load fell to 24 or below, but CI bursts arrived within one to three minutes of every start:

Run Load at start 1-minute load sampled each minute during the run Peak per CPU Load after Containers
(a) timing 23.85 37.03, 30.67, 39.95, 38.39, 28.81, 16.29, 31.19 0.89 42.74 73-78
(b) counting 27.25 39.70, 25.79, 17.58, 18.03, 22.17, 26.57, 31.79 0.83 30.52 78-83
© repeat 14.77 17.82, 20.35, 46.86, 50.60, 41.57, 52.81, 46.69 1.10 30.02 77-82

The timings below are therefore not acceptance evidence: load exceeded the threshold for most of every run. They are reported because the verdicts were identical in (a) and © in fifteen of sixteen cells and the failing margins are mostly several milliseconds, not noise-sized. The deterministic results (saved, exhausted, conflict attribution, command counts and the per-batch exactness checks) do not depend on host load and are reliable.

Two earlier attempts at a777e0021 failed after seven minutes with Connection refused because there are too many open connections: a client per arm left about 13 connections open on the server after disposal, and the forty arms of the new matrix reached mongod's limit. The harness now shares one instrumented client across arms (6ff78055c); the whole matrix peaks at 33 open connections.

Results (run a, gate b per cell)

The Verdict column applies the gate to loaded-host timings, so it is provisional (see the provisional verdict).

Target Reviewers Eligibility Source-only p50 / p95 ms Fold p50 / p95 ms Overhead p95 Latency (<10% or <=2 ms) Exhausted fold / source Fold saved / expected Statistics-caused Verdict
Same study 1 off 7.03 / 8.19 7.37 / 8.55 +4.4% (+0.36 ms) pass 0 / 0 100 / 100 0 PASS
Same study 1 on 7.42 / 12.04 13.41 / 16.57 +37.7% (+4.54 ms) FAIL 0 / 0 100 / 100 0 FAIL
Same study 2 off 7.51 / 12.12 11.08 / 20.17 +66.4% (+8.05 ms) FAIL 0 / 0 200 / 200 0 FAIL
Same study 2 on 16.45 / 27.93 15.60 / 38.84 +39.1% (+10.91 ms) FAIL 0 / 0 200 / 200 0 FAIL
Same study 5 off 16.11 / 23.75 26.53 / 42.26 +77.9% (+18.51 ms) FAIL 196 / 200 304 / 300 0 FAIL
Same study 5 on 26.25 / 35.45 33.38 / 46.72 +31.8% (+11.27 ms) FAIL 261 / 277 239 / 223 0 FAIL
Same study 10 off 18.37 / 28.86 34.31 / 49.90 +72.9% (+21.04 ms) FAIL 632 / 677 368 / 323 0 FAIL
Same study 10 on 31.14 / 43.51 44.84 / 53.87 +23.8% (+10.36 ms) FAIL 722 / 769 278 / 231 0 FAIL
Different studies 1 off 5.14 / 6.07 6.95 / 8.01 +32.0% (+1.94 ms) pass 0 / 0 100 / 100 0 PASS
Different studies 1 on 10.74 / 13.05 13.36 / 16.02 +22.8% (+2.98 ms) FAIL 0 / 0 100 / 100 0 FAIL
Different studies 2 off 5.09 / 6.48 7.21 / 9.11 +40.6% (+2.63 ms) FAIL 0 / 0 200 / 200 0 FAIL
Different studies 2 on 16.33 / 20.98 15.78 / 28.17 +34.3% (+7.19 ms) FAIL 0 / 0 200 / 200 0 FAIL
Different studies 5 off 4.93 / 5.97 6.94 / 13.12 +119.7% (+7.15 ms) FAIL 0 / 0 500 / 500 0 FAIL
Different studies 5 on 17.51 / 34.43 31.08 / 44.60 +29.5% (+10.17 ms) FAIL 200 / 199 300 / 301 0 FAIL
Different studies 10 off 5.07 / 7.64 7.04 / 15.48 +102.7% (+7.84 ms) FAIL 0 / 0 1000 / 1000 0 FAIL
Different studies 10 on 28.95 / 41.42 43.45 / 50.14 +21.1% (+8.72 ms) FAIL 688 / 700 312 / 300 0 FAIL

Exhausted and Fold saved are the literal gate (b) equalities. Where they fail, the fold arm exhausted fewer submissions than source-only (for example 632 against 677 at ten same-Study reviewers) in every cell but one, where it exhausted one more (200 against 199 at five different-Study reviewers with eligibility on, where the Project token serializes every save). Contended exhaustion is timing-dependent and differs run to run in both arms. No fold-arm conflict was statistics-caused in any cell. No conflict was fold-caused either, because the worker drains between batches, not during them; the free fold-only retry is covered by ProjectScreeningFoldEndToEndTests.AFoldBetweenTheLoadAndTheWriteIsAFreeRetry.

Run © agrees: PASS in same_study_1 (eligibility off) and FAIL in every other cell; the one difference is different_studies_1 (eligibility off), +1.94 ms in (a) and +2.00 ms in ©, against a +2 ms allowance.

Transactional arm, for reference (run a, eligibility off)

Target Reviewers Source-only p95 ms Transactional p95 ms Transactional saved Historical gate
Same study 1 8.19 43.22 100/100 428% FAIL
Same study 2 12.12 47.52 122/200 292% FAIL
Same study 5 23.75 43.41 136/500 83% FAIL
Same study 10 28.86 45.85 156/1000 59% FAIL
Different studies 1 6.07 38.79 100/100 539% FAIL
Different studies 2 6.48 87.61 200/200 1252% FAIL
Different studies 5 5.97 86.10 350/500 1342% FAIL
Different studies 10 7.64 83.09 600/1000 988% FAIL

Outcomes and conflicts (run a)

Conflicts are attributed per attempt from the reload the save already performs. Source: Study means the Study's version moved by a write that was not a fold; Source: Project means the Study did not move, so the eligibility transaction lost on the Project (its admission token or the first-review marker). Fold-caused would be a reload whose version moved only by folds. Statistics-caused counts statistics write rejections in the fold arm, and for the transactional arm a reload where the Study did not move. Unattributed is a third-attempt conflict with no reload in the timed path.

Target Reviewers Eligibility Arm Saved Exhausted Conflicts Source: Study Source: Project Fold-caused Statistics-caused Unattributed
Same study 1 off SourceOnly 100 0 0 0 0 0 0 0
Same study 1 off Fold 100 0 0 0 0 0 0 0
Same study 1 off Transactional 100 0 0 0 0 0 0 0
Same study 1 on SourceOnly 100 0 0 0 0 0 0 0
Same study 1 on Fold 100 0 0 0 0 0 0 0
Same study 2 off SourceOnly 200 0 100 100 0 0 0 0
Same study 2 off Fold 200 0 100 100 0 0 0 0
Same study 2 off Transactional 122 78 256 22 0 0 156 78
Same study 2 on SourceOnly 200 0 132 100 32 0 0 0
Same study 2 on Fold 200 0 144 100 44 0 0 0
Same study 5 off SourceOnly 300 200 900 900 0 0 0 0
Same study 5 off Fold 304 196 896 896 0 0 0 0
Same study 5 off Transactional 136 364 1134 134 0 0 636 364
Same study 5 on SourceOnly 223 277 1028 489 262 0 0 277
Same study 5 on Fold 239 261 985 707 278 0 0 0
Same study 10 off SourceOnly 323 677 2336 2336 0 0 0 0
Same study 10 off Fold 368 632 2215 2215 0 0 0 0
Same study 10 off Transactional 156 844 2600 477 0 0 1279 844
Same study 10 on SourceOnly 231 769 2505 1169 567 0 0 769
Same study 10 on Fold 278 722 2423 1863 560 0 0 0
Different studies 1 off SourceOnly 100 0 0 0 0 0 0 0
Different studies 1 off Fold 100 0 0 0 0 0 0 0
Different studies 1 off Transactional 100 0 0 0 0 0 0 0
Different studies 1 on SourceOnly 100 0 0 0 0 0 0 0
Different studies 1 on Fold 100 0 0 0 0 0 0 0
Different studies 2 off SourceOnly 200 0 0 0 0 0 0 0
Different studies 2 off Fold 200 0 0 0 0 0 0 0
Different studies 2 off Transactional 200 0 200 0 0 0 200 0
Different studies 2 on SourceOnly 200 0 100 0 100 0 0 0
Different studies 2 on Fold 200 0 100 0 100 0 0 0
Different studies 5 off SourceOnly 500 0 0 0 0 0 0 0
Different studies 5 off Fold 500 0 0 0 0 0 0 0
Different studies 5 off Transactional 350 150 650 0 0 0 500 150
Different studies 5 on SourceOnly 301 199 899 0 700 0 0 199
Different studies 5 on Fold 300 200 900 0 900 0 0 0
Different studies 10 off SourceOnly 1000 0 0 0 0 0 0 0
Different studies 10 off Fold 1000 0 0 0 0 0 0 0
Different studies 10 off Transactional 600 400 1398 0 0 0 998 400
Different studies 10 on SourceOnly 300 700 2400 0 1700 0 0 700
Different studies 10 on Fold 312 688 2379 0 2379 0 0 0

With eligibility on, the source-only arm itself exhausts 199 of 500 submissions at five different-Study reviewers and 700 of 1,000 at ten: every eligibility save writes the Project's admission token, so different Studies contend on one Project document. That is the source-level contention the design records as deferred item 2; the fold arm meets the same contention and adds none.

Command counts (run b)

Target Reviewers Eligibility Source-only per submission Fold per submission Transactional per submission Fold breakdown
Same study 1 off 2.00 5.00 25.00 find 400, findAndModify 100
Same study 1 on 7.00 10.00 n/a commitTransaction 100, find 700, update 200
Same study 2 off 3.50 9.50 20.44 find 1600, findAndModify 300
Same study 2 on 10.67 17.34 n/a abortTransaction 141, commitTransaction 200, find 2587, update 541
Same study 5 off 6.60 18.37 18.39 find 7986, findAndModify 1199
Same study 5 on 13.60 25.04 n/a abortTransaction 988, commitTransaction 234, find 9843, update 1456
Same study 10 off 7.71 20.58 17.65 find 17992, findAndModify 2591
Same study 10 on 14.25 27.70 n/a abortTransaction 2443, commitTransaction 264, find 22027, update 2971
Different studies 1 off 2.00 5.00 25.00 find 400, findAndModify 100
Different studies 1 on 7.00 10.00 n/a commitTransaction 100, find 700, update 200
Different studies 2 off 2.00 5.00 38.00 find 800, findAndModify 200
Different studies 2 on 9.50 14.00 n/a abortTransaction 100, commitTransaction 200, find 2000, update 500
Different studies 5 off 2.00 5.00 31.10 find 2000, findAndModify 500
Different studies 5 on 13.20 20.80 n/a abortTransaction 900, commitTransaction 300, find 7700, update 1500
Different studies 10 off 2.00 5.00 29.04 find 4000, findAndModify 1000
Different studies 10 on 14.11 22.88 n/a abortTransaction 2391, commitTransaction 306, find 17182, update 3003

Per submission, uncontended (one reviewer): source-only find + findAndModify = 2; fold = the private Study find, the durable-mode find and the control find concurrently, then the pre-write durable-mode re-read find, then findAndModify = 5, in three sequential round trips against two; transactional = 25. With eligibility on: 7 against 10 (the global and control reads, and the durable-mode read inside the eligibility transaction). Under contention every fold retry adds the reload, the durable mode, the control, the receipt and two quarantine reads and the pre-write re-read before its write (8 commands, 4 sequential round trips against the source-only retry's 2), which is why the fold arm's contended cells cost as many commands per submission as the transactional arm.

Command budget per save shape (gate b acceptance 1)

FoldSaveCommandBudgetTests and FoldSaveEligibilityCommandBudgetTests (API tests, the real ReviewScreeningFoldTarget, SubmitAnnotationSessionService and AnnotationDeletionFoldTarget) and ProjectStatisticsFoldCommandBudgetTests (Mongo.Data tests, the real repository) run in ordinary CI on a real replica set. Every shape asserts no write to any pmProjectStatistics* collection, the same number of commitTransaction and Project admission-token writes as the same call with statistics and the fold off, and the count below. A positive control asserts that today's transactional screening save does trip the statistics-write detector (24 commands, one commit).

Save shape Design table, first attempt As built, first attempt Statistics-off today Difference from the design
Capacity-guarded or plain screening 4 5 2 SaveScreeningOnFoldPathAsync re-reads the durable mode immediately before the write (a sequential round trip), so a mode flip between the attempt's reads and its write cannot commit an ordinary screening without its capacity guard
Screening retry (all MVP families requested) 6 10 for the retry; the lost attempt also issues the plain write's bulk-update-lock probe 2 + probe The screening save carries its annotation half as a second namespace, so the receipt and quarantine reads are per namespace, plus the unrecorded-drop range read and the pre-write re-read
ReviewEligibilityPolicy screening the above + Project + eligibility transaction 10 7 Matches the design: the transaction's own durable-mode read replaces the separate re-read; same commit and token write as today
Annotation session save, untracked 4 5 n/a The pre-write durable-mode re-read
Annotation session save, tracked 4 in the presence transaction 5 + commit same commit The re-read; the one commit is today's presence transaction
Annotation session delete 4 6 n/a The pre-write re-read, and the deletion target's Project read (cached in the host; one find on a miss)
Reservation release 4 5 2 The Project read the entry's context digest is computed from (cached in the host; one find on a miss)
Reservation claim 3, or 4 without the caller's Project 3 / 4 1 None
Eligibility admission today's transaction + global + control today + 2 5 None

Red first: the screening, release and retry assertions were first written at the design table's counts and failed with the traces above (for example Expected commands to contain 4 item(s) ... but found 5), then set to the as-built counts with the reason recorded beside each.

Provisional verdict, loaded host (not acceptance evidence)

Latency misses gate (b) on these timings; statistics cause no conflicts. This is not the gate (b) verdict. The timings were taken on a loaded host (host conditions), so gate (b) stays open until an idle-host rerun (planned on Bramble). Only the deterministic results below (conflict attribution, saved and exhausted counts, command counts and the per-batch exactness checks) are final.

  • Zero statistics-caused conflicts and failures in every cell, both eligibility settings, every run. No fold-arm save failed for a statistics reason.
  • Latency misses the limits in fourteen of sixteen cells in run (a) (fifteen in run ©), on a host that was not idle; provisional until the idle-host rerun. The uncontended fold save adds about 2 ms at p50 and p95 (one sequential round trip plus a slightly larger majority-acknowledged write). Contended cells add 7-21 ms at p95, because a fold retry costs four sequential round trips against two.
  • Exhaustion equality fails literally in the contended cells: the fold arm exhausted fewer submissions than source-only in all but one of them (one more, 200 against 199).

Cost drivers and the deferred optimisation

  1. The pre-write durable-mode re-read in SaveScreeningOnFoldPathAsync (and the annotation and deletion writes that share it) is the only extra sequential round trip on an uncontended save. The design (section 2, "Durable reviewer-mode check") accepts that the mode check is not atomic with the write, and invariant 13 makes a mode-transition owner wait FoldPreWriteDeadline + the writer grace before treating the fleet as transitioned. Whether that wait makes the re-read redundant is a safety decision for the design owner, not a benchmark change.
  2. Retry cost. Every charged retry runs the stored checks (receipt and quarantine, per namespace, plus the drop ranges) and the re-read, which dominates the contended cells.
  3. Write size. Both writes are a findAndModify that returns the document. The fold write also carries the pending set (a larger document both ways) and an explicit majority write concern, which on this replica set matches the server's implicit default, so this is a minor term.

Deferred item 3 (coalescing the global and project control reads into one $unionWith aggregate, or moving FoldMode onto the Project) was not applied. Both reads already run concurrently with the private Study load, so coalescing them removes one command but no sequential round trip; it cannot close a gap of about one round trip, and it would cross the boundary between the save (which reads the control) and the writer target (which reads the mode). Driver 1 is the lever that matches the measurement.

Limits

  • Host load exceeded the threshold during every run, so no timing here is acceptance-grade.
  • One project, ten studies; reviewer-tracking off (the capacity filter is not exercised by the timed fold write; the command-budget tests cover the tracked shape). Not HTTP, staging or production.
  • The fold worker drains between batches only, so fold-caused conflicts and free retries are not exercised by this benchmark (they are covered by the fold end-to-end tests).
  • The fold target mirrors ReviewScreeningFoldTarget command for command (the benchmark project cannot reference the API host); the API command-budget tests run the real target.