The API said the call didn't exist

The setup

The system was a call-attribution pipeline for a logistics company spending six figures a year on Google Ads. The job: every time a prospect calls, figure out which ad campaign produced that call, so the marketing budget follows what actually works.

The architecture was straightforward. Ads used a tracking number that forwarded to the main sales line. A cloud phone system logged every call. My pipeline polled the phone system's API on a schedule, pulled new calls, matched them to campaigns, and wrote the results where the team could see them. Polling is the simple, obvious choice — ask "what's new?" every few minutes, process the answer.

It ran. Health checks green. Calls flowing. Numbers in the dashboard.

The smell

The numbers were plausible but thin. Ad clicks said the phones should be busier than the attribution data claimed. Nothing was erroring — and that's the point of this story: nothing was erroring. If the API had returned failures, alerting would have caught it in minutes. Instead it returned 200 OK and clean JSON that happened to be incomplete.

When I pulled a specific call I could see in the phone system's own dashboard and asked the API for it — filtered by the caller's number — the API returned an empty list. The dashboard showed the call. The API said it didn't exist.

The culprit

After methodically comparing what the dashboard showed against what the API returned, the pattern surfaced: the API's call index was silently omitting forwarded calls — calls that came in through one number and were routed to another.

Now recall the architecture: every ad-driven call came through a tracking number that forwarded to the sales line. The API's blind spot wasn't random. It covered precisely the calls the pipeline existed to track. Organic calls logged fine; paid calls — the ones tied to the ad budget — vanished from the index.

This is the part I'd want a client to understand: no amount of ordinary error handling catches this. Retries, timeouts, status-code alerts — all useless against an API that succeeds confidently with missing data.

The fix

Two changes, both boring, both load-bearing:

• Push instead of pull. The phone system offered webhooks — events pushed at the moment a call happens. Webhooks fired for forwarded calls even though the polling index dropped them. The pipeline became a webhook receiver: idempotent (dedupe on call ID, so a replayed event can't double-count), durable (events logged before processing, so a crash loses nothing).
• Reconciliation. Trust, but verify — a scheduled job compares call volume against independent signals and alerts when reality and the pipeline drift apart. If a data source went quiet again, the system would now notice the absence.

The pipeline has attributed the ad spend reliably since. The diagnosis took longer than the fix — which is usually how it goes.

What it left behind

• Silence is not success. Monitoring that only watches for errors misses the failure mode where nothing errors. Production systems should also alarm on what's missing: "no calls logged in N hours" is an alert, not a quiet day.
• A green health check is a claim, not a fact. Verify pipelines end-to-end against an independent source, on a schedule, forever.
• Prefer push over poll for events that matter. Polling asks an API for its version of history; webhooks hand you the event as it happens. When money rides on completeness, take the event.
• The vendor's dashboard and the vendor's API are different products. Agreement between them is something to test, not assume.

← back to notes · talk to me about your pipeline