Why Every Link Was Healthy And The Service Still Dropped
A bonded connection with four carriers loses the feed for eleven seconds. Nobody has touched anything. Afterwards the monitoring shows four green lines with no gap in any of them.
This is the most common escalation we are called into, and the most commonly misdiagnosed. The equipment is usually fine. What failed was the definition of “up”.
What A Health Check Actually Proves
Most link monitoring answers one question: can this interface reach the next hop. It is normally an ICMP probe to the local gateway, or to a public resolver, at a fixed interval.
That is a useful question. It is not the question anybody cares about.
Reaching the carrier’s gateway proves the radio attached, the SIM authenticated and an address was issued. It proves nothing about whether traffic can cross that carrier’s network, reach your tunnel endpoint, and come back. Congestion at an aggregation point, a saturated backhaul, or a routing problem two hops upstream will all leave your gateway probe answering happily while the path is unusable.
Green means the probe succeeded. It does not mean the service worked.
Probe the far end of the path you actually depend on, over the protocol you actually use. If the service is a tunnel to a hub, the check belongs at the hub. If the check and the traffic take different routes, the check is measuring a route nobody is using.
Detection Time Is Arithmetic, Not A Feature
Datasheets advertise failover in milliseconds. That figure describes the switch, not the detection.
The arithmetic is straightforward. A probe every five seconds, with three consecutive failures required before a path is declared down, is fifteen seconds before failover begins. The switch itself may well be instant. The outage was fifteen seconds long.
Tighten the interval and you reduce detection time and increase false positives, because a single lost probe on a congested cellular path is normal. Loosen it and you get stability and a longer hole. There is no setting that gives both, and the right answer depends on whether the traffic can tolerate a short gap or a wrong decision.
State the detection budget in seconds before choosing equipment. Work backwards from what the application survives. Then set interval and threshold to meet it, and write down the false positive rate you accepted.
Four Carriers Is Often One Carrier
Path diversity is counted in contracts and delivered in infrastructure. The two frequently disagree.
Several things collapse apparent diversity. Mobile virtual operators resell capacity on a host network, so two SIMs from different retailers can ride the same radio access network. Distinct carriers frequently backhaul from the same mast site, and at a venue or a remote location there may only be one mast worth using. Fixed circuits ordered from different suppliers can share a duct, a chamber or a building entry.
When the common element fails, every path fails together, and each one honestly reports itself as down. The design had four paths. The site had one.
Ask for the host network behind every SIM, not the retail brand. For fixed circuits ask for the physical route and the building entry point. Diversity that has not been evidenced is an assumption, and assumptions are what the debrief will be about.
The Failover Worked. The Session Did Not.
Switching a path and preserving a service are different problems.
If failover changes the public address that traffic egresses from, every established TCP session breaks. They will re-establish, and the re-establishment is the outage. Streaming protocols carried over UDP survive the address change more gracefully, but an encoder or a receiver may not recover cleanly on its own, and some will sit in a failed state until something restarts them.
There is a second version of this that is harder to see. Failover succeeds onto a path with different characteristics, most obviously a satellite path where latency rises by hundreds of milliseconds. Every link is up. Throughput is adequate. The application is unusable because it was designed around a latency budget that no longer holds.
Decide whether the requirement is path redundancy or session continuity. They cost differently and they are built differently. If sessions must survive, the design needs an overlay that keeps the address stable across a path change, and the failover paths need to be specified against the application’s latency and jitter tolerance, not just its bandwidth.
Asymmetry Hides From Symmetric Tests
A small bidirectional probe tests both directions equally. Real traffic rarely does.
A contribution feed is heavily upstream. A monitoring probe is a few dozen bytes each way. Upstream congestion can degrade a path to the point where it cannot carry the feed, while the probe continues to pass comfortably in both directions.
Measure in the direction that carries the load, at a volume that resembles the load. A test that succeeds under conditions the service never operates in has told you nothing.
The Eleven Seconds, Explained
Most of these incidents are not one fault. They are a detection interval, plus a diversity assumption that was never evidenced, plus a health check pointed at the wrong place. Each one is defensible in isolation. Together they are an outage that nobody can find afterwards, because every component behaved exactly as configured.
The monitoring was not wrong. It answered the question it was asked. The question was the wrong one.
What To Ask Before Anybody Quotes
- What exactly does the health check probe, and is it the thing the service depends on?
- What is the detection time, as interval multiplied by threshold, in seconds?
- What is the host network behind every SIM, and the physical route of every circuit?
- Do sessions need to survive a path change, or only traffic?
- What is the application’s latency tolerance, and does every failover path meet it?
- In which direction is the load, and is that the direction being measured?
None of these require a product decision. All of them change which product is correct.