Summary

At camp 454 (Dammapeta Van1) on 5 Nov 2026, counter 2's tablet did not receive the registrar's (counter 1) new registrations for the whole camp day. Its first changes-feed download finished only at 20:32. To keep the queue moving, staff did TB screening on counter 1, so X-ray orders appeared about 2 minutes after registration. The field reported this as "X-ray ordered on registration".

Evidence: Fleet camp queries #12 to #16 on the camp 454 laptop, STOP-TB sync logs from the registrar and counter 2 tablets (4 and 5 Nov), read-only role lookup on central.

1. Field symptom: X-ray order "on registration"

• 124 X-ray orders for 81 beneficiaries on 3 and 5 Nov. For people registered the same day, the median time from registration to first order was about 2 minutes.
• Orders are created when the TB Screening form is saved (every non-pregnant beneficiary). On 5 Nov, the ordering user was the screening user for 20 of 20 beneficiaries.
• Roles at camp 454: TG-258 has Registrar only; TG-263 and TG-264 have Nurse + Registrar; TG-260 has Counsellor + Registrar. In the app, Registrar has TB Screening = FULL (RoleModuleConfig.kt), so the registrar can screen. TG-258 recorded 24 screenings.
• The field confirmed the cause: counter 2 and counter 3 were not getting data, so screening was done on counter 1.

2. Root cause: counter 2 could not finish its changes-feed download

The design is that tablets read only through /flw-api/stoptb/sync/changes (a snapshot first, then deltas). On 5 Nov:

Why it took so long and kept stopping:

1. Large first download: about 5,900 records. The server capped snapshot pages at 25 beneficiaries (MAX_BENEFICIARIES_PER_PAGE = 25), so about 235 pages and 20+ minutes of uninterrupted pulling. The download does resume from the saved token after an interruption.
2. Refresh cancelled the running download: startManualCampRefresh calls triggerAmritPullWorker(force = true), which used ExistingWorkPolicy.REPLACE and cancelled the pull in progress. Staff pressed Refresh about 20 times on counter 2.
3. Pulls skipped while uploads ran: triggerCampQuickPullIfConnected skipped the pull whenever an upload was running. That happened 223 times on counter 2. Uploads ran almost constantly because of items 3 and 4 below.
4. Unstable link to the laptop: counter 2 lost 192.168.1.50:8080 at 09:53, 10:19, 10:43, 12:20 and 13:44, and could not reach it again until 20:31.
5. Runs were also stopped 11 to 16 minutes in (JobCancellationException), consistent with Android stopping a long background job. Sync workers deliberately do not use a foreground service.

3. Side paths that bypassed the changes feed

3a. Per-order result polling (DiagnosticResultPollWorker)

• With the feed active, the worker turned off only its village-wide sweeps. It still called /flw-api/diagnostic/order/result for every pending order every cycle: 9,064 calls on counter 2 and 6,102 on the registrar tablet on 5 Nov.
• That endpoint only reads the laptop DB. The laptop polls the vendors itself (X-ray every 15 s, TrueNat every 60 s), and result changes already reach the tablet through the feed (change-log triggers on tb_diagnostic_order and tb_diagnostic_result; applyFeedDiagnosticOrders uses the same applyOrderResult).

3b. Why about 1,700 orders were being polled

• Counter 2 polled 1,330 beneficiaries for TrueNat, 382 for X-ray and 10 for MDR/RIF. The laptop answered PENDING 11,538 times. The most common state was X-ray COMPLETED, TrueNat PENDING.
• The tablet polls every local order that is PENDING with an order ID, with no date limit. TrueNat orders often never get a result through the integration, and the laptop's daily auto-close of outstanding orders is disabled in FLW-API (DiagnosticPollSchedulerService.expireOutstandingOrders commented out since 4 Aug 2026, "orders are no longer closed automatically"). So stuck orders pile up at long-running camps. Fleet query #16 (orders by type, status and age on the camp 454 laptop) was sent to size this.

3c. TB suspected upload loop

• The laptop's tb_suspected has no columns for hasSymptoms, isNaatConducted and naatResult. The tablet derives them from diagnostics (syncTBSuspectedFromDiagnostics), marks the record UNSYNCED and uploads it. The server drops them, the next feed page brings the record back with them false/null, and the tablet derives them again and treats that as a change.
• About 1,450 TB suspected records per tablet were re-marked UNSYNCED on 5 Nov. On the registrar tablet, 1,139 went round more than once. This kept the push chain busy, which in turn caused the skipped pulls in 2.3.

3d. Fallback to old per-table pulls

• On any feed error other than a retry code, PullFromAmritWorker fell back to the getAll per-table pulls, a second read path competing with the feed. These ran at 08:07 and 08:27 and failed because the laptop could not be reached.

4. Why only some camps

It needs several conditions at once: a large, long-running camp (big first download, many stuck TrueNat orders); a tablet that needs a fresh first download (new install, data cleared, re-login, or an app update that changes the synced type list); X-ray/TrueNat integration enabled; a weak Wi-Fi link from that tablet to the laptop; and staff pressing Refresh during the download.

5. Testing gaps

• Test data too small, so the first download finished in seconds.
• Single-tablet tests only, with no "registered on counter 1, visible on counter 2 within a minute" check.
• No device integration or few pending orders, so polling and the upload loop were invisible.
• No interruption tests: Wi-Fi drop, Android stopping the job, Refresh during a download.
• No upload-and-pull-back comparison per record type, so fields lost on the server were not caught.
• No tablet sync health reporting. Fleet monitors laptops only.

6. Fixes

Local tests: STOP-TB-App 109/109 unit tests pass. FLW-API sync tests pass except StopTBSyncCoverageTest, which fails identically on untouched release-3.12.1 (badge, notification and general check-up tables not classified); separate follow-up.

7. Open items

1. Review and merge #347 and #405, build the 2.3.1 APK, deploy FLW-API release-3.12.1 to camp laptops.
2. Program decision on stuck orders: bring back an expiry (for example, close TrueNat orders with no result after N days, keeping manual-entry orders exempt) or give staff a clean-up list.
3. Program decision on roles: should Registrar keep TB Screening = FULL, and should nurses also hold the Registrar role?
4. Camp 454: check counter 2's Wi-Fi to the laptop.
5. Add the missing tests from section 5, and tablet sync health reporting to Fleet.
6. Classify the unsynced tables flagged by StopTBSyncCoverageTest.