The most dangerous production failures aren't the ones that crash loudly. They're the ones that succeed - quietly, completely - while breaking everything that matters.
The ticket came in without much alarm. Reviews weren't publishing for one specific account. Probably a quick fix - we'd seen variants of this before. Except the logs looked clean, authentication was valid, and every API request was reaching the server and coming back 200 OK. On paper, the integration was healthy. But the reviews weren't there.
What made it harder: the same integration worked perfectly on every other account. Same code, same API calls, same role configuration — or so we thought. That account-specific behaviour meant conventional debugging was almost useless. You can't diff code that hasn't changed.
Testing our way to a dead end
We started in Postman, firing the same API calls manually against the affected account. Auth worked. Survey fetch returned data. Review submission accepted the payload and returned 200 OK every time. No errors anywhere.
So we ran the integration locally, pointed directly at the failing account, and stepped through the execution. Everything ran without error. The final call to publish the review returned 200 OK — and then silence. No review. No failure. Just a successful-looking response that led nowhere.
At that point, the only honest move was to stop looking at our system and reach out to the third-party platform's POC directly.
One permission. One checkbox. Everything broken.
They came back within a day. The API Admin role on that specific account was missing a required permission scope — something that hadn't been enabled during the account's original onboarding. The credentials were valid, the role existed, but without that scope, their backend was accepting our requests, returning 200 OK, and silently discarding the write operation. No error code. No failure message. Just a success response that meant something different to their system than it meant to ours.
Their team toggled the permission. Reviews started publishing immediately. The fix took seconds. Finding it took two days.
What this failure taught us
The real lesson wasn't about the permission itself. It was about where our visibility ended. Every tool we had — logs, Postman, local execution — confirmed that our side was working. None of them could tell us what happened after the request left our system.
Where silent failures actually live
Account-level permission scopes — configured in a third-party admin panel, invisible to your logs
Undocumented API behaviour — endpoints that return 200 whether or not the write actually succeeded
After this, permission validation became part of our onboarding checklist — probe calls that verify write access actually works, not just that credentials authenticate. And our health checks now measure outcomes, not just operations: did the review appear, not just did the POST return 200.
Loud failures are recoverable. A 500 fires an alert and someone fixes it fast. A silent mismatch between what your system reports and what actually happened — no alarm, everything green — is the one that costs the most time to find and the most trust to explain.
______________________________________________________________________
The API was working perfectly.
That was exactly the problem.
Discussion