Observability¶

Distributed coordination fails quietly. A watch that stopped delivering, a lease that healed three times an hour, a lock whose median wait crept from 2 ms to 800 ms — none of these throw, and all of them matter. This page covers the three surfaces the library gives you for seeing them: a metrics SPI, a background-exception channel, and the logging context that ties background threads back to the recipe that spawned them.

The EtcdMetrics SPI¶

EtcdMetrics is a dependency-free interface the library calls at its seams. The library itself ships exactly one implementation — EtcdMetrics.NoOp — and no metrics dependency at all.

Every method has an empty default body. That single decision is what makes the SPI cheap: an implementation overrides only the seams it cares about, adding a method to the interface never breaks an existing implementation, and a caller who wants no metrics pays nothing.

// Every EtcdMetrics method has a no-op default body, so an implementation overrides
// only the seams it cares about. Implementations must be thread-safe and cheap: they
// run on RPC-caller threads, the watch dispatcher, and the lease healer.
class CountingEtcdMetrics : EtcdMetrics {
  val retries = AtomicLong(0L)
  val leaseHeals = AtomicLong(0L)

  override fun recordRpc(
    opName: String,
    duration: Duration,
    attempts: Int,
    failed: Boolean,
  ) {
    if (attempts > 1) retries.fetchAndAdd((attempts - 1).toLong())
    if (failed) logger.warn { "$opName failed after $attempts attempt(s) in $duration" }
  }

  override fun incrementKeepAlive(
    kind: String,
    leaseId: Long,
  ) {
    // kind is one of renewal / suspended / expired / restored / failed.
    if (kind == "restored") leaseHeals.incrementAndFetch()
  }

  override fun incrementWatchRecovery(
    kind: String,
    key: String,
  ) {
    logger.info { "Watch recovery on $key: $kind" }
  }
}

Install a sink with ResilienceConfig.withMetrics(...), which returns a copy routing all three funnels — RPC, watch, and lease — plus every recipe seam to it:

// withMetrics routes the RPC, watch, and lease funnels — plus every recipe seam —
// to one sink. Pass the result wherever a recipe takes `resilience`.
val metrics = CountingEtcdMetrics()
val resilience = ResilienceConfig.DEFAULT.withMetrics(metrics)

DistributedMutex(client, "/locks/orders", resilience = resilience).use { mutex ->
  logger.info { "isLocked=${mutex.isLocked} retriesSoFar=${metrics.retries.load()}" }
}

Without that call, the sink is NoOp:

// The default. Nothing is recorded and callers pay nothing when metrics are off.
val resilience = ResilienceConfig.DEFAULT
logger.info { "metrics is NoOp: ${resilience.metrics === EtcdMetrics.NoOp}" }

Background exceptions¶

A recipe cannot throw at you from its own healer thread — there is no stack of yours for the exception to land on. Throwing there would kill the healer and lose the failure. So recipes never do it. Instead every background failure — a keep-alive death, an abandoned watcher, a lost lock, a user callback that threw — goes to one sink, which offers both a push and a pull interface.

fun interface BackgroundExceptionListener {
  fun onException(context: String, throwable: Throwable)
}

data class BackgroundException(val context: String, val throwable: Throwable)

context is the recipe's own identity — "DistributedMutex[/locks/orders]", "PathChildrenCache[/cache/orders]", "TransientKeyValue[/services/api]" — so one handler registered across a dozen recipes can still say which one failed.

TransientKeyValue(client, "/services/api", "up").use { tkv ->
  // Push: fires on the recipe's own thread as each background failure happens.
  // `context` is the recipe's identity, e.g. "TransientKeyValue[/services/api]",
  // so one handler shared across recipes can still attribute the failure.
  tkv.addBackgroundExceptionListener { context, throwable ->
    logger.error(throwable) { "Background failure in $context" }
  }

  // Pull: the same failures accumulate here for a caller that would rather poll.
  if (tkv.hasExceptions) {
    tkv.exceptions.forEach { e -> logger.warn(e) { "Recorded earlier" } }
    tkv.clearExceptions()
  }
}

TransientKeyValue(client, "/services/api", "up").use { tkv ->
  tkv.backgroundExceptionsAsFlow().collect { failure ->
    logger.error(failure.throwable) { "[${failure.context}] background failure" }
  }
}

Listeners run on the reporting recipe's own thread — a healer or dispatcher, never jetcd's event loop. A listener that throws is logged and dropped, never re-recorded, so notification cannot recurse. See Core concepts.

Logging context¶

Recipes wrap their background-thread work with the recipe's identity in the SLF4J MDC, under the key EtcdConnector.RECIPE_MDC_KEY ("etcd.recipe"). Any prior value is restored afterwards, so it composes with an MDC you already use.

The value is the same string as the exception context above. Put it in your log pattern and a stray warning from a healer thread stops being anonymous:

<pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %X{etcd.recipe} %logger{36} - %msg%n</pattern>

21:04:12.881 WARN  [etcd-lease-healer] TransientKeyValue[/services/api] i.e.r.c.SelfHealingKeepAlive - Lease 7587883142 expired; attempting to re-grant and re-establish

Without it, that line arrives from a thread named etcd-lease-healer with no clue which of your recipes owns it — and the healer threads of every recipe in the process share that name.

The Micrometer meter catalog¶

The etcd-recipes-micrometer module supplies MicrometerEtcdMetrics, an EtcdMetrics backed by a MeterRegistry. Install it the same way as any other sink — ResilienceConfig.withMetrics(MicrometerEtcdMetrics(registry)) — and you get:

All of them carry whatever common tags the registry already has.

Gauges¶

The SPI is push-only, which cannot express current values. EtcdGauges fills the gap with binders that poll a specific recipe instance on every scrape. Micrometer holds only a weak reference, so a bound gauge does not keep the recipe alive.

Binding several instances of the same gauge to one registry needs distinguishing tags — every binder takes them. See Integrations for setup, including the Spring Boot starter's auto-configuration.

What to watch¶

If you take four things from this page onto a dashboard, take these:

• etcd.keepalive{kind="expired"} — a lease expired, which for a lock or a leadership claim means ownership changed hands unexpectedly. Should be flat at zero.
• etcd.watch.recovery{kind="resynced"} — a compaction ate events. Anything with derived state may now be wrong. See Resilience.
• etcd.watch.recovery{kind="failed"} — a watcher was abandoned. It is dead and will not come back on its own.
• etcd.lock.wait{outcome="timeout"} relative to acquired — contention rising before it becomes an outage. See Locks.