Building Payment Infrastructure That Scales With You

At a hundred payments a day, almost any setup works: someone notices when something goes wrong and fixes it by hand. At a hundred thousand, the same setup quietly loses money, through double charges, missed refunds, orders stuck in 'pending' and settlements nobody can match. Scaling payments is less about speed than about making correctness automatic.

These are the principles that matter most, in roughly the order businesses discover they need them.

Networks time out, and code retries. Every request that moves money (a charge, a refund, a payout) should carry an idempotency key, so that sending it twice can't do it twice. Store the key with the order, and reuse it on every retry.

A customer's browser returning to your site proves nothing; the provider's server-to-server notice (webhook) does. Verify each notice's signature before acting on it, accept that notices can arrive twice or out of order, and design handlers so that processing the same event twice has no extra effect.

A payment moves through states: created, pending, succeeded, failed, refunded, disputed. Write down which transitions are allowed and enforce them, so a late 'pending' notice can never overwrite 'succeeded'. Keep every transition with its timestamp and source; that history is what support and audit teams need.

Reconciliation shouldn't be a month-end project. Match orders to payments, payments to settlements, and settlements to bank credits daily, on references you control. The output should be a short list of exceptions with owners, not a spreadsheet nobody finishes.

Watch success rate by method and provider, pending payments older than normal, notice delivery failures, refund backlog and settlement gaps. Alert on changes, not just thresholds: a success rate falling from 80% to 70% in an hour matters more than a steady 70%.

Every provider has incidents. With one provider, an incident stops your sales; with two, payments can move, if you've built for it. Redundancy means keeping both integrations current, deciding in advance what triggers a switch, and making retries safe so switching can't double-charge anyone.

As teams grow, more people can issue refunds, change bank details or approve payouts. Use maker-checker approvals for outgoing money, separate the right to add a payee from the right to pay one, keep an audit trail of every change, and remove access on the day people leave.

API keys and webhook secrets belong in server-side configuration, scoped per environment, never in browser or app code, and rotated when people with access leave. Test keys stay on test systems.