Resilience testing checks whether software keeps serving users when a dependency breaks. Teams inject timeouts, error responses, and slow databases, then measure the impact and the time to recover. Keploy brings this into CI: its recorded mocks can be edited into failure responses and replayed on every pull request.
Resilience testing gets skipped for the same reason outages run long: most systems are built for a dependency that goes down, not one that gets slow. A database that answers slowly still passes its health check. Meanwhile, every request handler waits on it, queues fill up, and a service that looks healthy on the dashboard starts turning users away.
Failover plans share that blind spot. They’re usually written for clean outages, so they wait for a dependency to disappear while users are already stuck behind one that’s only degraded. The distance between the failure a team planned for and the one that actually arrives is where an incident’s first hour usually goes.
What Is Resilience Testing?
Resilience testing, also called resiliency testing, is a type of non-functional software testing that deliberately introduces failures into a system, such as dependency outages, latency, error responses, or resource loss, to verify that it degrades gracefully and recovers within agreed limits.
Software resiliency is the property being measured, and resilience testing is how you prove it. You’ll also see the practice called fault tolerance testing or software resilience testing, and it sits under the broader discipline of resilience engineering.
The pass criteria aren’t "nothing broke." Instead, a resilience test asks whether the system stayed within its service level objectives (SLOs) while something was broken, and whether it came back without manual help.
Why Does Resilience Testing Matter?
Resilience testing matters because real dependencies rarely fail cleanly, and the recovery code written for those moments almost never runs until an incident forces it to.
- Partial failures are the dangerous ones. A dependency that’s slow or erroring intermittently is harder to detect than one that’s fully down, so failover logic often misses it.
- Retries can turn a blip into an outage. Clients that retry aggressively without backoff multiply the load on a service that’s already struggling.
- Failures travel in microservices. One slow service holds connections open in every caller, so the degradation moves upstream, one hop at a time.
- Recovery paths go untested. Fallbacks, circuit breakers, and failover scripts only run during incidents, which is the worst possible time to learn they don’t work.
That’s why a good resilience suite tests the recovery path as carefully as the failure itself.
Types of Resilience Testing

Each type targets a different way a system can fail. Most teams start with dependency failure and latency tests, since those cover the incidents that happen most often in API services.
Dependency Failure Testing
A downstream API, database, or queue returns errors or stops responding entirely.
- What you check: fallback responses, clear error messages for users, and circuit breakers that open instead of hammering the failed dependency.
Latency and Timeout Testing
A dependency still answers, just slowly. In practice, this is where to start, because default timeouts are often far longer than any user will wait.
- What you check: timeouts are set and enforced, connection pools don’t run dry, and users get a fast failure instead of a hanging request.
Failover Testing
The primary instance of a component is removed or degraded, and traffic should shift to a backup.
- What you check: failover triggers on degraded performance, not only on a total outage, and the switch finishes within your recovery target.
Recovery Testing
The failure is removed, and the system should return to normal without anyone stepping in.
- What you check: recovery time, data consistency after the event, and whether queued work drains without overloading the dependency again.
Load-Induced Failure Testing
Failures happen while the system is under realistic traffic, which is when retries and restarts do the most damage.
- What you check: retries use exponential backoff with jitter, load shedding protects core paths, and rate limits hold.
Resilience Testing Techniques
Fault Injection
Fault injection introduces a specific, controlled fault at a defined point: an error response, added latency, a dropped connection, or a killed process. Almost every resilience test is built on it, so it’s the technique to learn first.
Chaos Experiments
Chaos experiments run fault injection as a structured experiment. You define a steady state, form a hypothesis, break something, and check whether the steady state held. This loop is the core of chaos testing, and it often runs in production-like environments or in production itself.
Service Virtualization for Failure Simulation
Service virtualization replaces a real dependency with a virtual one that returns failures on demand. As a result, you can test how your code handles a partner API outage without touching the partner, and the same test runs safely on every build. It also fits naturally with dependency mocking when a team already mocks external calls in CI.
How to Do Resilience Testing

- Define steady state. Pick the metrics that show the system is healthy, such as error rate, p95 latency, and throughput, and tie them to your SLOs.
- List the failures that matter. Work from your dependency map and past incidents. Every external API, database, cache, and queue is a candidate.
- Decide the expected behavior first. For each failure, write down what should happen before you run anything: a fallback, a cached response, a graceful error, or a failover.
- Simulate the failure in a safe environment. Start with mocks or virtual services in CI, then move to a staging test environment. Production experiments come last, with a limited blast radius and a clear stop condition.
- Measure degradation and recovery. Compare results against steady state, and record how long the system took to recover.
- Automate the repeatable tests. Dependency failure and timeout tests belong in CI on every pull request, while larger chaos experiments can run on a schedule.
Resilience Testing Examples
A useful resilience test names three things: the failure you inject, the behavior you expect, and the number that tells you whether it held. These examples cover the failures API teams run into most.
|
Failure injected |
Expected behavior |
What to measure |
|
Payment API returns 503 |
Checkout shows a clear error, retries with backoff, and never creates a duplicate order |
Error rate, retry count, duplicate orders |
|
Database responds slowly |
Requests time out fast instead of queuing, and failover triggers on latency |
p95 latency, connection pool usage, time to failover |
|
Auth token service is down |
Existing sessions keep working from cached tokens |
Share of requests still served |
|
Cache cluster is lost |
Requests fall back to the database without overloading it |
Database load, response time |
|
Many instances restart at once |
Jittered backoff keeps startup traffic to shared dependencies within limits |
Dependency load, total recovery time |
|
Message queue rejects writes |
Events are buffered or retried later, and none are lost |
Lost messages, backlog drain time |
|
Downstream API returns malformed JSON |
The parser rejects it safely and the service falls back |
5xx rate, fallback rate |
Resilience Testing Tools
Most teams pair a tool that simulates dependency failures close to the code with one that breaks infrastructure on purpose.
|
Tool |
Layer |
What it simulates |
License |
|
Keploy |
API and dependencies |
Error responses and unexpected payloads from recorded mocks |
Open-source core |
|
Gremlin |
Infrastructure |
Resource, network, and host failures |
Commercial |
|
LitmusChaos |
Kubernetes |
Pod, node, and network faults |
Open source |
|
Chaos Mesh |
Kubernetes |
Pod, network, I/O, and time faults |
Open source |
|
Toxiproxy |
Network (TCP) |
Latency, timeouts, and dropped connections |
Open source |
|
AWS Fault Injection Service |
AWS resources |
Faults in EC2, ECS, EKS, and RDS |
Pay as you go |
|
k6 |
Load |
Realistic traffic while faults run |
Open source, paid cloud |
Keploy
Keploy records API calls along with the database queries and downstream calls behind them, then turns them into tests and mocks. Because the mocks are plain files, a recorded success can be edited into a failure, such as a 503 or a malformed payload, and replayed in CI.
- Best for: testing how an API handles failing dependencies on every pull request, without a live environment
- Simulates: error responses and unexpected payloads from databases and downstream APIs
- Limitation: works at the application dependency level, so it doesn’t kill nodes or degrade networks
- License: open-source core under Apache 2.0, with paid plans
Gremlin
Gremlin is a commercial chaos engineering platform for running controlled failure experiments across hosts, containers, and Kubernetes.
- Best for: teams that want managed chaos experiments with built-in safety controls
- Simulates: CPU, memory, and disk exhaustion, network latency and packet loss, and process or host shutdown
- Limitation: infrastructure-level faults need careful blast radius planning, especially in production
- License: commercial
LitmusChaos
LitmusChaos is an open-source, CNCF chaos engineering project for Kubernetes that defines experiments as Kubernetes resources.
- Best for: Kubernetes teams that want chaos experiments managed like other cluster resources
- Simulates: pod deletion, node failures, network latency, and resource stress
- Limitation: focused on Kubernetes, so it’s less useful for services running elsewhere
- License: open source
Chaos Mesh
Chaos Mesh is another open-source CNCF project that injects faults into Kubernetes workloads.
- Best for: fine-grained fault types inside Kubernetes clusters
- Simulates: pod failures, network partitions and delays, I/O faults, and clock skew
- Limitation: Kubernetes only, and cluster-wide permissions need careful control
- License: open source
Toxiproxy
Toxiproxy, from Shopify, is a TCP proxy that sits between an application and its dependencies to simulate network conditions in tests.
- Best for: lightweight latency and timeout tests in local development and CI
- Simulates: added latency, bandwidth limits, timeouts, and dropped connections
- Limitation: works at the TCP level, so it can’t return HTTP-specific failures like a 503
- License: open source
AWS Fault Injection Service
AWS Fault Injection Service is a managed service for running fault injection experiments against AWS resources, with stop conditions tied to CloudWatch alarms.
- Best for: teams running on AWS that want managed experiments with automatic stop conditions
- Simulates: faults in EC2, ECS, EKS, and RDS, including instance stops and network disruption
- Limitation: only covers AWS resources
- License: pay as you go, based on usage
k6
k6 is an open-source load testing tool from Grafana Labs. It doesn’t inject faults by itself, but it generates the realistic traffic that load-induced failure tests need, and its thresholds can fail a CI job.
- Best for: testing retries, backoff, and load shedding under realistic traffic
- Simulates: user load, which pairs with a fault injection tool running at the same time
- Limitation: needs another tool to create the failure
- License: open source, with a paid cloud service
Resilience Testing vs Performance Testing vs Chaos Engineering

|
Aspect |
Resilience testing |
Performance testing |
Chaos engineering |
|
Main question |
Does it degrade gracefully and recover? |
Is it fast enough under load? |
What unknown weakness appears when something breaks? |
|
Input |
Planned, specific failures |
Load profiles |
Experiments with a hypothesis |
|
Where it runs |
CI and staging |
Staging or a performance environment |
Staging and production |
|
Pass criteria |
SLOs held and recovery time met |
Latency and throughput targets met |
Steady state held |
Performance testing asks how the system behaves when everything works but demand is high. Resilience testing, however, asks how it behaves when something doesn’t work at all, so the two often run together in load-induced failure tests.
Conclusion
Dependency graphs keep getting wider. AI features add model APIs to request paths, and AI agents call other services with their own retry logic, so every new integration is another partial failure waiting to happen.
Resilience tests that only run during an annual game day can’t keep up with that. The teams that stay ahead will run the cheap tests (timeouts, error responses, backoff) on every pull request and save the big experiments for failures that truly need real infrastructure.
FAQs
Is resilience testing functional or non-functional testing?
Resilience testing is non-functional testing. It doesn’t check whether a feature returns the right result. Instead, it checks how the system behaves, and how quickly it recovers, when something it depends on fails.
What is the difference between resilience testing and recovery testing?
Recovery testing is one type of resilience testing. It focuses only on what happens after a failure ends, such as recovery time and data consistency. Resilience testing also covers how the system behaves while the failure is still happening.
What is the difference between resilience testing and reliability testing?
Reliability testing measures how consistently a system works over time under normal conditions. Resilience testing measures how it behaves when conditions are not normal, such as during a dependency outage or a sudden slowdown.
Can resilience testing run in CI/CD?
Yes. Dependency failure and timeout tests run well in CI when dependencies are mocked or virtualized, because they don’t need real infrastructure to break. Larger chaos experiments usually run on a schedule in staging or production.

Leave a Reply