W. Blanchard-Butter
Senior Systems Engineer
Case study 02 · Payment credential isolation

Isolating Payment Processor Credentials Behind a Minimal Internal API

Rebuilt payment tracking across two automation instances so that credentials capable of creating charges and reading stored card data never sit in an environment the whole team can access.

1 in 9,336
Payments missed
9,063
Payments processed
273
Refunds
19 mo
In production

The problem

Payment tracking ran off an email inbox. A scenario polled it every five minutes and string-matched notification bodies to work out what had been paid, by whom. It missed payments regularly — any change to the notification wording, any message that failed to arrive, and the payment simply never got recorded.

The obvious fix was to integrate the payment processor's API directly. That created a second problem.

Our main automation instance is accessible to the entire systems team and to contractors. Anyone with access can open any scenario and read any credential stored in it. The PaySimple API key is not a read-only key: the same credential that looks up a customer record can create new payments and retrieve stored card details. Putting it in the main instance would have meant every contractor who ever touched an unrelated scenario had, in practice, the ability to charge cards.

What the payment pipeline actually needed from that API was much smaller: given a customer ID, return a name, an email and a company name.

Constraints

  • No VPS or container infrastructure available at the time. A second Make instance was offered at no additional cost.
  • The credential must never be readable from the main instance, by anyone.
  • The pipeline had to be more reliable than inbox polling, which was the bar it replaced.

Architecture

Three scenarios across two instances.

Main instancePayments receives processor webhooks and routes on event type. Settled and refunded payments are reconciled against MySQL and Google Sheets locally, with a duplicate check before insert. Created-payment and created-customer events are forwarded across the instance boundary. No PaySimple credential exists anywhere in this instance.

Restricted instancePayments V2 handles client resolution, deciding whether a payment belongs to a new or existing client, and fans out to Slack, Sheets, Monday and GoHighLevel. Where it needs customer details, it calls the third scenario.

Restricted instanceNew Customer Search is the credential boundary and is deliberately trivial: accept a customer ID, call PaySimple, return three fields. Nothing else.

The important property is the response shape. The PaySimple customer endpoint returns considerably more than three fields. The proxy discards the rest rather than passing it through. That is the difference between an internal API and a tunnel — a pass-through proxy relocates the credential without reducing what an attacker who reaches the endpoint can obtain. Narrowing the response means that even a fully compromised proxy yields a name, an email and a company, not billing details.

Two independent controls

Credential isolation answers what an attacker gets if they reach the endpoint. It does not answer who can reach it. Those are separate problems and the design needed both.

The proxy webhook now requires a shared secret, checked in a filter placed on the outbound PaySimple call itself rather than downstream. A request without the secret is stopped before the credentialed call is made, so a rejected request never reaches the payment processor and receives no response body.

The two controls are deliberately independent. The secret limits who gets through; the narrowed response limits what getting through is worth. Either alone is weak — an authenticated tunnel still exposes card data to anyone who obtains the secret, and a narrow endpoint with no auth is an open customer lookup. Together the failure of either one is survivable.

Results

  • In production since 24 January 2025. Roughly nineteen months, 9,063 payments and 273 refunds processed, one missed — against a predecessor that missed them routinely.
  • PaySimple credentials have never existed in the main instance.

What I would change

The secret comparison should fail closed. It is currently an equality check. If the stored value is ever cleared or the mapping breaks, an empty incoming value compares equal to an empty expected value and the filter passes — the control disappears silently, with no error to alert anyone. Pairing the equality check with an existence check means a missing secret can never satisfy the filter, and the failure mode becomes an outage rather than an open endpoint. For a control sitting in front of payment credentials, an outage is the correct direction to fail in.

A second automation instance is not the right home for this. It was chosen because it was free and because there was no infrastructure available at the time, and that was a reasonable call under those constraints. But the proxy is thirty lines of logic wearing a whole platform, and the platform brings a UI, a user list and an execution history that all have to be secured. On a VPS it would be a small service with a rotating secret, unit tests around the response filtering, and no interactive access at all.

Declines still run on the system this replaced. Failed payments — 1,694 over the same period — are detected by the original inbox poller and then enriched through the same customer-search proxy. They are handled, but by the mechanism the rest of the pipeline moved away from, so that path keeps the failure mode the webhook events no longer have: if a notification email changes wording or never arrives, the decline is missed.

I proposed migrating declines onto the webhook pipeline. It was not taken up — the team was under load and a rebuild that added no new capability to a working path was not worth the time. That was a fair call. The cost is carried quietly rather than visibly, which is the usual shape of this kind of decision: the poller keeps working until the day the wording changes.

Two of the three guarantees are structural; one is administrative. The narrowed response and the secret check hold regardless of who has access — they are properties of the system. The fact that the credential-holding instance has a single user is a property of a user list, and it degrades differently: it survives exactly as long as nobody is added and nobody forgets to remove someone. It is a real control and the right one under the circumstances, but it is the one that decays quietly, and it is worth being explicit about which category each control falls into.

It also makes me a single point of failure on payment tracking. The boundary needs a documented handover — what the second instance contains, how the secret rotates, and who takes it on — for reasons that have nothing to do with security and everything to do with the pipeline outliving whoever built it.

← All work