Context
During a fintech audit, servers processed thousands of requests per hour even when no customers were online. Legacy test scripts and forgotten clients still had valid keys.
Threats
- API keys never expired and weren’t tied to owners.
- Forgotten integrations hammered prod endpoints daily.
- Rate limiters blocked traffic but offered zero attribution.
- No single view of how requests flowed across microservices.
Approach
- Inventory every API credential, tagging owners, scopes, and last-used timestamps in a central registry.
- Shipped key rotation + expiration policies enforced via CI, so stale keys stop the build.
- Fronted services with API Gateway + WAF adaptive throttling while pushing logs to CloudWatch + Grafana for behavior insights.
- Instrumented OpenTelemetry traces end-to-end so we could explain every successful request and its downstream impact.
- Set up anomaly alerts when frequency or geography shifted outside baselines.
Outcome
Zombie traffic fell by 90% in two weeks, compute cost dropped 30%, and we discovered a beta client still hitting sensitive endpoints. The visibility work turned into a Zero Trust roadmap for the org.
Lessons Learned
Security starts with knowing who is talking to you. If you can’t explain a request, you’ve already lost observability — and maybe data.