Skip to content

Connection state

Every recipe extends EtcdConnector, and every EtcdConnector maintains one coarse health signal: connectionState.

enum class ConnectionState {
  CONNECTED,
  SUSPENDED,
  RECONNECTED,
  LOST,
}

It is passive. Nothing polls, nothing pings, nothing runs on a timer. The state is derived entirely from events the recipe's own watch and lease streams already reported on their way past — which is why a connector with no active watch and no active lease reports nothing at all, and sits at CONNECTED forever.

State Meaning
CONNECTED The initial state. No failure has been observed yet.
SUSPENDED A stream reported an error; recovery — jetcd's or the recipe layer's — is in progress.
RECONNECTED A stream was re-established. For a lease, ownership was re-established too.
LOST A lease expired, or recovery was abandoned. Ownership may be gone.

CONNECTED is not a health check

It is the state a recipe starts in, not one it is promoted back to. A recipe that has never talked to etcd and a recipe with a perfectly healthy watch look identical from here. To ask whether etcd is actually reachable, use ping().

What drives the transitions

The mapping is small enough to memorise, and worth memorising, because it is the whole reason the states mean what they mean:

Event New state
WatchRecoveryEvent.Suspended SUSPENDED
WatchRecoveryEvent.Resubscribed / Resynced RECONNECTED
WatchRecoveryEvent.Failed LOST
LeaseEvent.Suspended SUSPENDED
LeaseEvent.Expired LOST
LeaseEvent.Restored RECONNECTED
LeaseEvent.Failed LOST

A stream that jetcd recovers by itself (a transient watch or keep-alive error) reports that too: the watch emits Resubscribed and the lease Restored with the same old and new id, so SUSPENDED always resolves.

A recipe often runs several streams (a leader election has a participation lease and a leader watch, for example). A LOST from a stream that is gone for good — a Failed event, or a start that failed — sticks: until the recipe restarts, no event from its other, healthy streams can move it off LOST, which would otherwise hide the dead one. A lock loss (Expired) does not stick; the next acquisition's lease reports afresh.

A recipe built from other recipes reports theirs as its own: a LeaderLatch its current term's selector, a ServiceProvider its cache.

Note the sharpest line in that table: LeaseEvent.Expired → LOST, immediately, before any healing is attempted. That is not pessimism. An expired lease means etcd already deleted your keys, and for a lock or a leadership claim it means a successor may already be running. LOST says ownership may not be yours any more — and it says it at the only moment when saying it is useful. If healing succeeds, Restored moves you to RECONNECTED afterwards. See Leases and loss.

Listening

fun interface ConnectionStateListener {
  fun stateChanged(newState: ConnectionState, previous: ConnectionState)
}

Register with addConnectionStateListener and drop with removeConnectionStateListener:

TransientKeyValue(client, "/services/api", "up", leaseTtlSecs = 5L).use { tkv ->
  tkv.addConnectionStateListener { newState, previous ->
    logger.info { "Connection state: $previous -> $newState" }
    when (newState) {
      // A stream errored; recovery is in progress. Nothing is lost yet.
      ConnectionState.SUSPENDED -> logger.warn { "Degraded; holding off on new work" }

      // The stream — and, for a lease, the ownership it backs — is re-established.
      ConnectionState.RECONNECTED -> logger.info { "Back; safe to resume" }

      // A lease expired or recovery was abandoned. Ownership may be someone else's.
      ConnectionState.LOST -> logger.error { "Ownership may be gone; stop acting on it" }

      ConnectionState.CONNECTED -> logger.info { "Initial state" }
    }
  }
}

Both the new and the previous state are handed to you, because the transition is often more informative than the destination — SUSPENDED → RECONNECTED is a blip that healed, while LOST → RECONNECTED means you got your key back but somebody may have seen it missing in between.

Transitions are deduplicated, ordered, and delivered on the notifier

Repeated Suspended reports during one outage notify once: an equal state is dropped, atomically. Listeners run on the recipe's notifier thread, one at a time and in the order the state changed, never on the reporting thread (which can be jetcd's event loop). A listener that blocks delays only that recipe's later notifications, and a listener that throws is recorded on exceptions under the context connection-state-listener rather than escaping.

isHealthy() vs ping()

Two questions that look the same and are not:

isHealthy() ping()
Cost Free — no RPC One round trip
Answers "Has anything gone irrecoverably wrong that I already know about?" "Can I reach etcd right now?"
Implementation connectionState != LOST && !closed One count-only GET, bounded at 2 seconds by default
Fails how Cannot fail Returns false rather than throwing
DistributedMutex(client, "/locks/orders").use { mutex ->
  // Passive: derived from events the recipe already observed. No RPC, so it is
  // cheap enough to call on every scrape of a liveness endpoint.
  val live = mutex.isHealthy()

  // Active: one count-only GET, bounded at 2 seconds (RpcResilience.PROBE).
  // Costs a round trip, so save it for a readiness probe.
  val ready = mutex.ping()

  logger.info { "live=$live ready=$ready state=${mutex.connectionState}" }
}

The distinction maps neatly onto Kubernetes-style probes:

  • Liveness wants isHealthy(). It is called often, must not add load, and is asking whether this process is still in a workable state. A liveness probe that issues an RPC turns an etcd hiccup into a fleet-wide restart storm — precisely the wrong response to a shared dependency being briefly slow.
  • Readiness wants ping(). It is asking whether this instance can serve traffic that will need etcd, and that genuinely requires asking etcd. Being told "no" here removes one pod from a load balancer; it does not kill anything.

ping() writes nothing: it is a count-only GET against a fixed probe key that need not exist. It makes a single attempt bounded at 2 seconds (RpcResilience.PROBE, reported to the recipe's metrics sink), so it answers promptly during the outage it exists to detect instead of retrying for minutes; ping(rpc) takes a different budget. A definite refusal from etcd, such as PERMISSION_DENIED for a user whose RBAC doesn't cover the probe key, still counts as reachable. Only failing to get an answer returns false.

There is also a Client-level counterpart for code that holds a client but no recipe — a readiness endpoint wired up before any recipe is constructed, say:

// The same probe without a recipe: useful for a readiness check that owns a
// Client but no recipe yet. It defaults to RpcResilience.PROBE: one attempt,
// bounded at 2 seconds.
logger.info { "etcd reachable: ${client.ping()}" }

// A probe with its own deadline, when 2 seconds is wrong for your endpoint.
val patient = RpcResilience(RetryPolicy.never, operationTimeout = 5.seconds)
logger.info { "etcd reachable within 5s: ${client.ping(patient)}" }

A probe gets its own budget, not the recipe's

The 30-second default operationTimeout is right for a lock acquisition and wrong for a health endpoint. A probe that retries for half a minute is not reporting health, it is hiding it. That is why both ping()s default to RpcResilience.PROBE — RetryPolicy.never with a 2-second deadline — rather than the recipe's RpcResilience. Pass a different one only when your probe's own timeout calls for it.

As a Flow

connectionStateAsFlow() emits the current state on collection and then every transition. It is conflated, so a slow collector sees the latest state rather than every intermediate one, and it is cold — collection registers the listener, cancellation removes it, and neither starts nor closes the recipe.

TransientKeyValue(client, "/services/api", "up").use { tkv ->
  // Emits the current state immediately, then every transition. Conflated, so a
  // slow collector sees the latest state rather than every intermediate one.
  tkv.connectionStateAsFlow().collect { state ->
    logger.info { "state=$state" }
  }
}

Use .stateIn(scope) for a hot StateFlow shared across collectors. See Flows.

Where to go next