Connection state¶
Every recipe extends EtcdConnector, and every EtcdConnector maintains one coarse
health signal: connectionState.
It is passive. Nothing polls, nothing pings, nothing runs on a timer. The state is
derived entirely from events the recipe's own watch and lease streams already reported
on their way past — which is why a connector with no active watch and no active lease
reports nothing at all, and sits at CONNECTED forever.
| State | Meaning |
|---|---|
CONNECTED |
The initial state. No failure has been observed yet. |
SUSPENDED |
A stream reported an error; recovery — jetcd's or the recipe layer's — is in progress. |
RECONNECTED |
A stream was re-established. For a lease, ownership was re-established too. |
LOST |
A lease expired, or recovery was abandoned. Ownership may be gone. |
CONNECTED is not a health check
It is the state a recipe starts in, not one it is promoted back to. A recipe that
has never talked to etcd and a recipe with a perfectly healthy watch look
identical from here. To ask whether etcd is actually reachable, use
ping().
What drives the transitions¶
The mapping is small enough to memorise, and worth memorising, because it is the whole reason the states mean what they mean:
| Event | New state |
|---|---|
WatchRecoveryEvent.Suspended |
SUSPENDED |
WatchRecoveryEvent.Resubscribed / Resynced |
RECONNECTED |
WatchRecoveryEvent.Failed |
LOST |
LeaseEvent.Suspended |
SUSPENDED |
LeaseEvent.Expired |
LOST |
LeaseEvent.Restored |
RECONNECTED |
LeaseEvent.Failed |
LOST |
A stream that jetcd recovers by itself (a transient watch or keep-alive error) reports that
too: the watch emits Resubscribed and the lease Restored with the same old and new id,
so SUSPENDED always resolves.
A recipe often runs several streams (a leader election has a participation lease and a
leader watch, for example). A LOST from a stream that is gone for good — a Failed
event, or a start that failed — sticks: until the recipe restarts, no event from its
other, healthy streams can move it off LOST, which would otherwise hide the dead one. A
lock loss (Expired) does not stick; the next acquisition's lease reports afresh.
A recipe built from other recipes reports theirs as its own: a LeaderLatch its
current term's selector, a ServiceProvider its cache.
Note the sharpest line in that table: LeaseEvent.Expired → LOST, immediately,
before any healing is attempted. That is not pessimism. An expired lease means etcd
already deleted your keys, and for a lock or a leadership claim it means a successor
may already be running. LOST says ownership may not be yours any more — and it says
it at the only moment when saying it is useful. If healing succeeds, Restored moves
you to RECONNECTED afterwards. See Leases and loss.
Listening¶
fun interface ConnectionStateListener {
fun stateChanged(newState: ConnectionState, previous: ConnectionState)
}
Register with addConnectionStateListener and drop with removeConnectionStateListener:
TransientKeyValue(client, "/services/api", "up", leaseTtlSecs = 5L).use { tkv ->
tkv.addConnectionStateListener { newState, previous ->
logger.info { "Connection state: $previous -> $newState" }
when (newState) {
// A stream errored; recovery is in progress. Nothing is lost yet.
ConnectionState.SUSPENDED -> logger.warn { "Degraded; holding off on new work" }
// The stream — and, for a lease, the ownership it backs — is re-established.
ConnectionState.RECONNECTED -> logger.info { "Back; safe to resume" }
// A lease expired or recovery was abandoned. Ownership may be someone else's.
ConnectionState.LOST -> logger.error { "Ownership may be gone; stop acting on it" }
ConnectionState.CONNECTED -> logger.info { "Initial state" }
}
}
}
Both the new and the previous state are handed to you, because the transition is often
more informative than the destination — SUSPENDED → RECONNECTED is a blip that healed,
while LOST → RECONNECTED means you got your key back but somebody may have seen it
missing in between.
Transitions are deduplicated, ordered, and delivered on the notifier
Repeated Suspended reports during one outage notify once: an equal state is
dropped, atomically. Listeners run on the recipe's notifier thread, one at a time
and in the order the state changed, never on the reporting thread (which can be
jetcd's event loop). A listener that blocks delays only that recipe's later
notifications, and a listener that throws is recorded on exceptions under the
context connection-state-listener rather than escaping.
isHealthy() vs ping()¶
Two questions that look the same and are not:
isHealthy() |
ping() |
|
|---|---|---|
| Cost | Free — no RPC | One round trip |
| Answers | "Has anything gone irrecoverably wrong that I already know about?" | "Can I reach etcd right now?" |
| Implementation | connectionState != LOST && !closed |
One count-only GET, bounded at 2 seconds by default |
| Fails how | Cannot fail | Returns false rather than throwing |
DistributedMutex(client, "/locks/orders").use { mutex ->
// Passive: derived from events the recipe already observed. No RPC, so it is
// cheap enough to call on every scrape of a liveness endpoint.
val live = mutex.isHealthy()
// Active: one count-only GET, bounded at 2 seconds (RpcResilience.PROBE).
// Costs a round trip, so save it for a readiness probe.
val ready = mutex.ping()
logger.info { "live=$live ready=$ready state=${mutex.connectionState}" }
}
The distinction maps neatly onto Kubernetes-style probes:
- Liveness wants
isHealthy(). It is called often, must not add load, and is asking whether this process is still in a workable state. A liveness probe that issues an RPC turns an etcd hiccup into a fleet-wide restart storm — precisely the wrong response to a shared dependency being briefly slow. - Readiness wants
ping(). It is asking whether this instance can serve traffic that will need etcd, and that genuinely requires asking etcd. Being told "no" here removes one pod from a load balancer; it does not kill anything.
ping() writes nothing: it is a count-only GET against a fixed probe key that need not
exist. It makes a single attempt bounded at 2 seconds (RpcResilience.PROBE, reported to
the recipe's metrics sink), so it answers promptly during the outage it exists to detect
instead of retrying for minutes; ping(rpc) takes a different budget. A definite refusal
from etcd, such as PERMISSION_DENIED for a user whose RBAC doesn't cover the probe key,
still counts as reachable. Only failing to get an answer returns false.
There is also a Client-level counterpart for code that holds a client but no recipe —
a readiness endpoint wired up before any recipe is constructed, say:
// The same probe without a recipe: useful for a readiness check that owns a
// Client but no recipe yet. It defaults to RpcResilience.PROBE: one attempt,
// bounded at 2 seconds.
logger.info { "etcd reachable: ${client.ping()}" }
// A probe with its own deadline, when 2 seconds is wrong for your endpoint.
val patient = RpcResilience(RetryPolicy.never, operationTimeout = 5.seconds)
logger.info { "etcd reachable within 5s: ${client.ping(patient)}" }
A probe gets its own budget, not the recipe's
The 30-second default operationTimeout is right for a lock acquisition and wrong
for a health endpoint. A probe that retries for half a minute is not reporting
health, it is hiding it. That is why both ping()s default to RpcResilience.PROBE
— RetryPolicy.never with a 2-second deadline — rather than the recipe's
RpcResilience. Pass a different one only when your probe's own timeout calls for it.
As a Flow¶
connectionStateAsFlow() emits the current state on collection and then every
transition. It is conflated, so a slow collector sees the latest state rather than every
intermediate one, and it is cold — collection registers the listener, cancellation
removes it, and neither starts nor closes the recipe.
TransientKeyValue(client, "/services/api", "up").use { tkv ->
// Emits the current state immediately, then every transition. Conflated, so a
// slow collector sees the latest state rather than every intermediate one.
tkv.connectionStateAsFlow().collect { state ->
logger.info { "state=$state" }
}
}
Use .stateIn(scope) for a hot StateFlow shared across collectors. See
Flows.
Where to go next¶
- Leases and loss — what
LOSTactually costs you, per recipe. - Resilience — the retry policies that decide how long
SUSPENDEDlasts. - Observability — exporting this alongside the recovery counters.
- Core concepts — the
EtcdConnectorlifecycle.