APO — Autonomous Payment Orchestrator

The problem

A payment assistant can produce a convincing answer and still propose the wrong amount. Even a correct proposal leaves harder questions: who authorizes it, what happens when a service fails, and whether retrying creates another payment.

I built APO to explore that entire path. A natural-language request becomes a structured proposal, an independent policy checks it, and authorized execution passes through a durable workflow, a payment core, and double-entry accounting.

The model proposes an action. It does not have the authority to pay. Payment credentials are selected by the application through a separate boundary, not by the model.

APO is an engineering prototype, not a service for paying real invoices. The browser demo uses a deterministic mock for reasoning and a simulated payment provider. HTTP calls between services, PostgreSQL persistence, Inngest execution, policy checks, idempotency, and ledger writes are real.

Open the live demo. It uses mock reasoning and simulated payments; no real money moves. The demo VM stays running to avoid idle cold starts.

Four ways to test the system

The demo is organized around deliberate failures and repeated delivery. Each experiment separates the expected outcome from the evidence actually returned by the backend.

Experiment	What happens	Evidence to inspect
Repeat the same payment	Complete a $500 payment, then explicitly repeat capture and ledger posting ten times, sequentially.	Captured amount stays at $500. Payment ID and both ledger entry IDs stay unchanged. Ten receipts show the returned results.
Recover from temporary failures	The simulator fails the first authorize and first capture calls. Inngest retries the steps.	Failed attempts followed by successful calls, a confirmed $500 capture, and the ledger pair.
Reject the assistant’s amount	The source says $500; the mock proposes $5,000.	The policy rejects the proposal with `amount_not_grounded`. No workflow starts.
Handle a permanent decline	Both source and proposal specify $600; the simulator declines authorization.	A failed payment record and `capturedAmount = 0` when the payment snapshot is available.

Starting a prepared scenario explicitly authorizes that simulated payment. This matters because the $500 and $600 amounts meet the manual-confirmation threshold; the scenarios satisfy the policy rather than bypassing it.

The repeat experiment is a second action, available only after the original payment is confirmed. It tests sequential redelivery, not concurrent load. Reconnecting or retrying evidence loading performs GET requests only; it does not initiate another payment.

Architecture and sources of truth

The execution path is:

Request → proposal → policy → authorization → durable workflow → payment core → ledger.

Runtime services communicate through HTTP contracts. Within packages, domain and application code depend on ports; network clients and SDKs live in adapters.

Package	Responsibility
`agent-orchestrator`	Intents, model proposals, policy checks, human confirmation, and ownership checks. Determines whether execution may begin.
`durable-ledger`	Inngest steps, retries, compensation, double-entry posting, and observation journaling. Ledger records establish whether posting happened.
`pay-core`	Payment state, transactions, versions, idempotency, and atomic state/outbox writes. Its simulator snapshot establishes the captured amount.
`orchestra`	Gateway, browser lab, CLI, and restrictions on optional live-model access. Presents facts from the other services.
`agent-evals`	Hostile scenarios, generated cases, effect assertions, and reports. Runs outside the application runtime.

Inngest owns workflow status. The application does not maintain a competing workflow lifecycle in a second status table.

Decisions that matter

Return the original result on a retry

An idempotency key is bound to a request-body fingerprint. The same key and body return the stored result. Reusing the key with a different body produces a conflict. Payment state and outbox events are written in one transaction.

The outbox has a deliberate limit: a separate dispatcher for delivering pay-core outbox events is not connected. Inngest executes the workflow; the presence of an outbox is not a claim of end-to-end event delivery.

Completed-capture replay is enabled only for the simulation. The browser submits an intent ID. The backend derives the original payment and keys from the authoritative run and canonical ledger posting, rather than accepting arbitrary amounts, payment IDs, or credentials from the browser. The ten sequential replays have a shared nine-second deadline and per-call limits.

Give retries one owner

Inngest manages durable retries. Application code classifies the error and returns a decision without adding another timed retry loop. This prevents nested loops from multiplying attempts and keeps execution status with the system responsible for execution.

Check proposals independently

The proposed amount and merchant must be grounded in the original request or an accepted clarification. Text parsing extracts values for that check; it does not authenticate an invoice or establish that a merchant exists in a verified registry.

Prepared simulator profiles are immutable intent metadata. Per-intent uniqueness keeps one visitor’s simulated failures from affecting another visitor’s experiment.

Keep evidence separate from inference

Two ledger entries represent the debit and credit of one payment. Their sum is zero because the posting balances, not because the customer was charged nothing. The captured amount comes from the payment snapshot. A successful replay must preserve the original entry IDs as well as the balance.

The observation journal is best-effort: a telemetry failure must not change a payment, retry, or compensation. Its history can therefore be incomplete. Replay receipts are separate batch responses; a shared key or nearby timestamp does not establish a one-to-one relationship with journal rows.

An unavailable snapshot means the amount is unknown. An empty journal does not mean zero was captured.

The challenge: building entirely through AI

A second challenge was the development process itself: could I take a multi-service engineering prototype from specifications to a verifiable demo with all code generation and changes delegated to AI?

That meant making the work explicit enough for agents to execute and other agents to review. I used specifications to state the intended behavior, constraints, service boundaries, and acceptance scenarios. Architectural decisions and handoffs carried that context into implementation and verification. I remained responsible for the direction and for deciding whether the result met the goal.

Different roles, explicit handoffs

Code generation and changes were delegated to AI agents throughout the project. I set the goal, directed the interface, clarified constraints, selected decisions, and checked the result.

The workflow separates a coordinator, an architect, one implementer for the complete change, a final verifier, and independent general and security reviewers. The coordinator carries the task and constraints across handoffs; the architect defines the plan and boundaries; the implementer delivers the connected change and runs step-level checks. The final verifier and both reviewers assess the completed package.

Reviews apply to a fixed branch, base SHA, and head SHA. A new commit invalidates earlier approvals; the implementer does not approve or merge its own work.

Keeping specifications and evidence connected

ADRs and handoffs preserve decisions, constraints, the agreed plan, revision identifiers, and command results. This is explicit repository context, not a custom persistent-memory engine.

What the process exposed

One useful lesson was the gap between backend correctness and a convincing demonstration. A CLI PASS did not explain the outcome to a visitor. The interface evolved into a lab with separate payment and replay actions, inspectable evidence, and readable decline states. Other fixes were less visible but just as important: stale reads must not overwrite a newer experiment, telemetry failures must not bring down the payment service, and cents must not be displayed as dollars.

I did not measure development speed or productivity gains from AI, so I make no numerical claim about them.

Verification at the recorded snapshot

The supplied project record covers commit cff901e, dated October 16, 2026. These are recorded results for that revision, not a fresh test run performed for this portfolio page.

pnpm test: 1,784 passing tests in 116 files — pay-core 112, durable-ledger 363, agent-orchestrator 772, orchestra 171, and agent-evals 366.
Typecheck, lint, build, browser JavaScript syntax, and diff checks passed.
All four browser scenarios passed locally against the real service stack with the simulated provider.
Previously passing database integration checks were reused where persistence, dependencies, and environment were unchanged. They are not added to the 1,784 total as new independent results.
No paid live-model calls were made in this verification.

Hostile evaluations cover 31 scenarios and 200 generated cases with a fixed seed. Their oracles inspect effects rather than the wording of model responses. Coverage is bounded: the amount check shares an extractor with the system, and merchant grounding has no independent oracle. Passing the corpus does not establish the absence of vulnerabilities.

Limits and deployment status

X-Customer-Id scopes data; it is not authentication. The daily-limit check has an accepted race: counting completed intents does not reserve capacity for concurrent in-flight requests. Idempotency depends on keys, storage, and delivery boundaries; APO does not claim universal exactly-once behavior for arbitrary providers.

Compensation is covered by process tests. The four browser scenarios do not demonstrate a user-initiated refund.

The public demo was deployed on October 20, 2026, from commit 7927fa3. Four service processes share one Fly VM, with Neon PostgreSQL and Inngest Cloud. Only the gateway and signed Inngest callback are exposed publicly; backend service ports remain internal. The hosted demo uses mock reasoning and the payment simulator, with no live-model instance.

Cloud verification confirmed a completed $500 simulated payment, two ledger entries, ten repeat deliveries preserving the original payment and ledger identities, rejection of unsigned callbacks, and persistence after stopping and waking the VM. One measured wake-and-read took about 16 seconds; this is an observation, not a latency guarantee. The four-scenario verification and test counts above remain tied to their historical snapshot.

The demo is hosted at the root of an HTTPS subdomain. Mounting under a path such as /payments is not supported.

This project does not claim PCI compliance or production readiness. Its purpose is to make execution boundaries, uncertainty, and observable effects concrete enough to inspect and discuss.

Source reference

The implementation discussion and historical test counts describe the recorded October 16 snapshot; the deployment section records the October 20 cloud verification. The links below follow the repository’s current HEAD, including subsequent changes; the test results above remain tied to cff901e.

For a smaller, standalone illustration of the retry ambiguity, see A timeout is not a failed operation. That browser model is separate from APO’s backend experiments.