Connection state¶

Every recipe extends EtcdConnector, and every EtcdConnector maintains one coarse health signal: connectionState.

enum class ConnectionState {
  CONNECTED,
  SUSPENDED,
  RECONNECTED,
  LOST,
}

It is passive. Nothing polls, nothing pings, nothing runs on a timer. The state is derived entirely from events the recipe's own watch and lease streams already reported on their way past — which is why a connector with no active watch and no active lease reports nothing at all, and sits at CONNECTED forever.

What drives the transitions¶

The mapping is small enough to memorise, and worth memorising, because it is the whole reason the states mean what they mean:

Note the sharpest line in that table: LeaseEvent.Expired → LOST, immediately, before any healing is attempted. That is not pessimism. An expired lease means etcd already deleted your keys, and for a lock or a leadership claim it means a successor may already be running. LOST says ownership may not be yours any more — and it says it at the only moment when saying it is useful. If healing succeeds, Restored moves you to RECONNECTED afterwards. See Leases and loss.

Listening¶

fun interface ConnectionStateListener {
  fun stateChanged(newState: ConnectionState, previous: ConnectionState)
}

Register with addConnectionStateListener and drop with removeConnectionStateListener:

TransientKeyValue(client, "/services/api", "up", leaseTtlSecs = 5L).use { tkv ->
  tkv.addConnectionStateListener { newState, previous ->
    logger.info { "Connection state: $previous -> $newState" }
    when (newState) {
      // A stream errored; recovery is in progress. Nothing is lost yet.
      ConnectionState.SUSPENDED -> logger.warn { "Degraded; holding off on new work" }

      // The stream — and, for a lease, the ownership it backs — is re-established.
      ConnectionState.RECONNECTED -> logger.info { "Back; safe to resume" }

      // A lease expired or recovery was abandoned. Ownership may be someone else's.
      ConnectionState.LOST -> logger.error { "Ownership may be gone; stop acting on it" }

      ConnectionState.CONNECTED -> logger.info { "Initial state" }
    }
  }
}

Both the new and the previous state are handed to you, because the transition is often more informative than the destination — SUSPENDED → RECONNECTED is a blip that healed, while LOST → RECONNECTED means you got your key back but somebody may have seen it missing in between.

isHealthy() vs ping()¶

Two questions that look the same and are not:

DistributedMutex(client, "/locks/orders").use { mutex ->
  // Passive: derived from events the recipe already observed. No RPC, so it is
  // cheap enough to call on every scrape of a liveness endpoint.
  val live = mutex.isHealthy()

  // Active: a bounded, count-only GET through the usual retry/timeout funnel.
  // Costs a round trip, so save it for a readiness probe.
  val ready = mutex.ping()

  logger.info { "live=$live ready=$ready state=${mutex.connectionState}" }
}

The distinction maps neatly onto Kubernetes-style probes:

• Liveness wants isHealthy(). It is called often, must not add load, and is asking whether this process is still in a workable state. A liveness probe that issues an RPC turns an etcd hiccup into a fleet-wide restart storm — precisely the wrong response to a shared dependency being briefly slow.
• Readiness wants ping(). It is asking whether this instance can serve traffic that will need etcd, and that genuinely requires asking etcd. Being told "no" here removes one pod from a load balancer; it does not kill anything.

ping() writes nothing: it is a count-only GET against a fixed probe key that need not exist. It routes through the same retry and timeout funnel as every other RPC, so the recipe's RpcResilience decides how long it may take before returning false.

There is also a Client-level counterpart for code that holds a client but no recipe — a readiness endpoint wired up before any recipe is constructed, say:

// The same probe without a recipe: useful for a readiness check that owns a
// Client but no recipe yet. A tight per-probe deadline keeps the endpoint honest.
val probe = RpcResilience(RetryPolicy.never, operationTimeout = 2.seconds)
logger.info { "etcd reachable: ${client.ping(probe)}" }

As a Flow¶

connectionStateAsFlow() emits the current state on collection and then every transition. It is conflated, so a slow collector sees the latest state rather than every intermediate one, and it is cold — collection registers the listener, cancellation removes it, and neither starts nor closes the recipe.

TransientKeyValue(client, "/services/api", "up").use { tkv ->
  // Emits the current state immediately, then every transition. Conflated, so a
  // slow collector sees the latest state rather than every intermediate one.
  tkv.connectionStateAsFlow().collect { state ->
    logger.info { "state=$state" }
  }
}

Use .stateIn(scope) for a hot StateFlow shared across collectors. See Flows.

Where to go next¶

• Leases and loss — what LOST actually costs you, per recipe.
• Resilience — the retry policies that decide how long SUSPENDED lasts.
• Observability — exporting this alongside the recovery counters.
• Core concepts — the EtcdConnector lifecycle.