Design an Idempotent Payment System

Financial payment systems must guarantee that money is never created or destroyed out of thin air, duplicate charges never occur during network timeouts, and every single transaction is immutably audited via double-entry bookkeeping.

1. System Requirements

Functional Requirements

1. Pay-in Processing: Accept customer payment methods (credit card, bank transfer) and process charge authorizations.
2. Double-Entry Financial Ledger: Maintain an immutable accounting record where total debits equal total credits for every transaction.
3. Third-Party Payment Service Provider (PSP) Integration: Dispatch external API calls to providers like Visa, Mastercard, or Stripe.
4. Reconciliation Engine: Asynchronously compare internal transaction records with daily bank settlement files to catch discrepancies.
5. Refunds & Chargebacks: Support full and partial refunds with reversible ledger entries.

Non-Functional Requirements

1. Strict Idempotency: Guarantee exactly-once payment processing even when network retries occur.
2. Zero Financial Discrepancies: Total ledger consistency; transactions must satisfy ACID guarantees.
3. High Availability: 99.999% uptime for the payment ingestion gateway.
4. Auditability: Every modification must be traceable to a cryptographically verifiable transaction log.

2. Core Concepts: The Double-Entry Ledger

In financial accounting, money does not simply change a single balance column. Instead, every movement of capital is represented as two offsetting entries: $$\sum \text{Debits} = \sum \text{Credits}$$

Example: Customer Buys a $100 Item

Account                    | Debit   | Credit
---------------------------|---------|---------
Customer Cash Holding      |         | $100.00
Platform Escrow Holding    | $100.00 | 

Ledger entries are strictly append-only. Corrections are made by creating a new reversing entry, never by modifying past database rows.

3. High-Level Architecture Blueprint

                               +-----------------------+
                               |     Checkout UI       |
                               +-----------+-----------+
                                           |
                                           | POST /payments (with Idempotency-Key)
                                           v
                               +-----------------------+
                               |    Payment Gateway    |
                               | (Rate Limit & Verify) |
                               +-----------+-----------+
                                           |
                                           v
                               +-----------------------+
                               |    Payment Service    |
                               +-----+-----------+-----+
                                     |           |
          +--------------------------+           +--------------------------+
          |                                                                 |
          v (1. Acquire Lock & Check State)                                 v (2. Insert Pending)
+------------------------+                                         +------------------------+
|   Redis Distributed    |                                         |  PostgreSQL Primary DB |
|   Lock (Redlock)       |                                         | (Orders & Payments)    |
+------------------------+                                         +-----------+------------+
                                                                               |
                                                                               v (3. Enqueue Outbox)
                                                                   +------------------------+
                                                                   |  Transactional Outbox  |
                                                                   |  & Kafka Task Queue    |
                                                                   +-----------+------------+
                                                                               |
                                                                               v
                                                                   +------------------------+
                                                                   | Payment Execution Svc  |
                                                                   +-----------+------------+
                                                                               |
                                                                               v Call External API
                                                                   +------------------------+
                                                                   | Third-Party PSP (Visa) |
                                                                   +------------------------+

4. Staff-Level Deep Dives

Deep Dive 1: End-to-End Idempotency Pattern

When a user clicks "Pay", mobile networks can drop right after the credit card is charged, prompting the user to hit "Pay" again.

1. Idempotency Key Generation: The client generates a unique UUID (e.g. Idempotency-Key: c9b8a...) prior to submitting the payment.
2. Atomically Claim Key: The server queries the database within a transaction:

   INSERT INTO payment_idempotency_keys (key, user_id, status)
   VALUES ('c9b8a...', 1234, 'PROCESSING')
   ON CONFLICT (key) DO NOTHING;

1. If the insert succeeds, this is a fresh request; proceed.
2. If conflict occurs:
3. If status is PROCESSING: Return HTTP 409 Conflict ("Transaction in progress, please wait").
4. If status is COMPLETED: Return the previously cached response payload immediately without charging the card again.

Deep Dive 2: The Two-Phase PSP Execution (Auth & Capture)

Never debit an internal account before the card network approves, but never approve without recording the intent.

1. Authorization: Holds the funds on the customer's card (e.g. 7-day reserve).
2. Capture: Transfers the authorized funds into the merchant's merchant account after the order is fulfilled.

Deep Dive 3: The Reconciliation Engine

• External banks deliver end-of-day settlement files (BAI2, MT940, CSV).
• An automated batch pipeline joins internal ledger entries with bank settlement rows.
• Discrepancies are categorized:
• Timing Differences: Transactions settled on different calendar days.
• Missing Fees: Unexpected interchange fee deductions.
• Unreconciled Losses: Charges present in bank records but missing from the internal database (triggered by failed webhooks). Escalated to operations.

Key Architecture Trade-Offs & Decision Matrix