The Ghost Traffic Problem: When APIs Keep Talking to Strangers

During a cloud audit for a fintech app, I noticed something strange. The servers were handling thousands of API requests per hour — even when no users were online. At first, everyone assumed it was just background jobs or analytics pings. But when we checked the source of those calls, we found something unexpected: old developer test scripts, expired mobile app versions, and third-party clients that were never properly offboarded. They were still sending valid requests. Nothing was technically 'hacked,' but the system was bleeding resources and exposing unnecessary risk.

• Old API keys that still worked, with no expiration date or usage limits
• Forgotten integrations still hitting production endpoints daily
• No visibility into which users or clients generated specific requests
• Rate limiters that blocked requests but never told us who sent them

1. We started by **mapping every API key** in the system — who owned it, what it did, when it was last used, and whether it should still exist.
2. Introduced **token expiration and rotation policies** — every key now had a defined lifetime and automatic renewal process.
3. Used **AWS API Gateway + CloudWatch** to track each request’s origin, IP, and metadata. We visualized traffic patterns in Grafana.
4. Set up **behavioral monitoring** — if an API client suddenly behaved differently (like changing frequency or region), it triggered an alert.
5. Enabled **WAF adaptive throttling** — instead of simply blocking requests, it analyzed behavior and automatically adjusted limits.
6. Finally, integrated **OpenTelemetry traces** to see how API calls moved across services, making 'invisible' traffic visible again.

Within two weeks, non-human or legacy traffic dropped by over 90%. System load balanced out, and bandwidth costs went down by 30%. The best part? We discovered a small data exposure from an old beta client that could have become a major breach if left unnoticed. By treating each request as something we could explain, not just process, we built confidence in what our APIs were actually doing.

Modern API security isn’t only about stopping hackers — it’s about knowing your own ecosystem. Every connection, key, and request should have a reason to exist. The moment you can’t explain a request, you’ve found your first vulnerability.