Skip to content

Locks

etcd-recipes ships three mutual-exclusion recipes:

Recipe Use it when
DistributedMutex One holder at a time
DistributedReadWriteLock Many readers, or one writer
DistributedSemaphore At most N holders at a time

All three extend EtcdConnector, so they share the lifecycle, exception, and connection-state surface described in Core concepts.

The EtcdLock interface

DistributedMutex and the two views of DistributedReadWriteLock implement EtcdLock:

interface EtcdLock {
  fun lock()
  fun tryLock(timeout: Duration): Boolean
  fun tryLock(timeout: Long, timeUnit: TimeUnit): Boolean
  fun unlock(): Boolean
  val isHeldByCurrentThread: Boolean
  val isLocked: Boolean
  val holdCount: Int
  val fencingToken: Long
  fun addLockLostListener(listener: LockLostListener)
  fun removeLockLostListener(listener: LockLostListener)
}

lock() and both tryLocks declare @Throws(InterruptedException::class), so Java can catch an interrupted wait.

Why not java.util.concurrent.locks.Lock?

EtcdLock deliberately does not implement Lock. Three of Lock's promises cannot be honoured over a network: newCondition() has no distributed meaning, a no-arg tryLock() cannot be answered without a round trip, and — most importantly — Lock has no vocabulary for losing a lock you already hold. A distributed lock can evaporate underneath you when its lease expires. Pretending otherwise by implementing Lock would make that failure invisible.

DistributedMutex

Built on etcd's native lock service, so ordering is server-side FIFO by revision.

DistributedMutex(client, "/locks/orders").use { mutex ->
  mutex.withLock {
    // Only one client across the cluster runs this at a time.
    logger.info { "Critical section" }
  }
}
try (DistributedMutex mutex = new DistributedMutex(client, "/locks/orders")) {
  mutex.lock();
  try {
    // Only one client across the cluster runs this at a time.
    System.out.println("Critical section");
  } finally {
    mutex.unlock();
  }
}

withLock { } is an inline Kotlin extension. It does compile to a static EtcdLockKt.withLock(lock, Function0) that Java can call, but a Java lambda would have to return Unit.INSTANCE, so it buys nothing — Java callers take the lock and release it in a finally. See the Java guide.

Acquiring with a timeout

Blocking forever is rarely what a service wants. tryLock bounds the wait, and every RPC the attempt makes along the way (the lease grant, reads, and transactions), so during an etcd brownout it still returns false close to its timeout rather than minutes later. A failure that retrying can't fix, such as permission denied on the lock path, is thrown rather than reported as a timeout. The same goes for lock(), which would otherwise retry it forever.

DistributedMutex(client, "/locks/orders").use { mutex ->
  if (mutex.tryLock(5.seconds)) {
    try {
      logger.info { "Acquired within the timeout" }
    } finally {
      mutex.unlock()
    }
  } else {
    logger.info { "Someone else holds the lock; moving on" }
  }
}
try (DistributedMutex mutex = new DistributedMutex(client, "/locks/orders")) {
  // Java uses the (long, TimeUnit) overload; the Duration one is Kotlin-facing.
  if (mutex.tryLock(5, TimeUnit.SECONDS)) {
    try {
      System.out.println("Acquired within the timeout");
    } finally {
      mutex.unlock();
    }
  } else {
    System.out.println("Someone else holds the lock; moving on");
  }
}

Kotlin takes Duration, Java takes (long, TimeUnit)

Both overloads exist on every lock. The kotlin.time.Duration one reads better from Kotlin; the (long, TimeUnit) one is the one to use from Java.

Losing the lock

This is the part that has no single-JVM analogue, and it is the part worth reading twice.

A mutex hold is backed by a lease. If the process stalls or the network partitions long enough for that lease to expire, etcd has already handed the lock to the next waiter. The recipe does not try to heal that lease or reclaim the lock: doing so would race the new holder and defeat the entire point of the lock.

Instead, loss is cooperative. You are told, and you decide:

DistributedMutex(client, "/locks/orders").use { mutex ->
  // The acquisition lease is deliberately never healed. If it expires, etcd has
  // already promoted the next waiter, so the hold is gone and cannot be reclaimed.
  mutex.addLockLostListener { cause ->
    logger.warn { "Lost the lock, abandoning work: $cause" }
  }

  mutex.withLock {
    // unlock() returns false if the hold was lost while inside the section.
    logger.info { "Working" }
  }
}
try (DistributedMutex mutex = new DistributedMutex(client, "/locks/orders")) {
  // The acquisition lease is deliberately never healed. If it expires, etcd has
  // already promoted the next waiter, so the hold is gone and cannot be reclaimed.
  mutex.addLockLostListener(cause ->
    System.out.println("Lost the lock, abandoning work: " + cause));

  mutex.lock();
  try {
    System.out.println("Working");
  } finally {
    // Returns false if the hold was lost while inside the section.
    mutex.unlock();
  }
}

Three things happen on loss:

  • every registered LockLostListener fires,
  • connectionState moves to LOST,
  • the eventual unlock() returns false rather than throwing.

Listeners run on the lock's notifier thread, never on jetcd's lease callback thread, so a listener may block or make an RPC; it delays only the lock's later notifications. A loss applies only to the hold whose lease expired, not to a newer hold of the same thread.

If you would rather the holding thread be interrupted the instant the lock is lost — so it cannot keep mutating state it no longer owns — opt in with interruptOnLockLoss:

// Opt in to having the holding thread interrupted the moment the lock is lost,
// rather than letting it run on against state it no longer owns.
DistributedMutex(
  client = client,
  lockPath = "/locks/orders",
  interruptOnLockLoss = true,
).use { mutex ->
  mutex.withLock {
    logger.info { "Interrupted if the lease expires under us" }
  }
}
// Opt in to having the holding thread interrupted the moment the lock is lost,
// rather than letting it run on against state it no longer owns. Java has no
// named arguments, so every preceding parameter must be supplied positionally.
try (DistributedMutex mutex =
       new DistributedMutex(
         client,
         "/locks/orders",
         10L,                                 // leaseTtlSecs (the lock default)
         io.etcd.recipes.common.ResilienceConfig.DEFAULT,
         "worker-1",                          // clientId
         true)) {                             // interruptOnLockLoss
  mutex.lock();
  try {
    System.out.println("Interrupted if the lease expires under us");
  } finally {
    mutex.unlock();
  }
}

It defaults to false because interrupting a thread that is midway through non-idempotent work is not automatically safer than letting it finish and fail its own commit. Choose deliberately.

Fencing tokens

Neither listeners nor interrupts close the gap entirely: a holder learns of the loss on the client, after etcd has already granted the lock to the next waiter. For a resource where two writers must never both land, use the hold's fencingToken. It is a number etcd assigned to the hold, larger than any earlier conflicting holder's. A resource that keeps the largest token it has seen and refuses smaller ones then turns away a holder that lost the lock but hasn't noticed yet:

DistributedMutex(client, "/locks/orders").use { mutex ->
  mutex.withLock {
    // Send the token with every write. If this hold was lost (a pause past the lease) and
    // another client has the lock now, its larger token already reached the store, which
    // refuses this stale write.
    store.write("order-42 shipped", mutex.fencingToken)
  }
}
try (DistributedMutex mutex = new DistributedMutex(client, "/locks/orders")) {
  mutex.lock();
  try {
    // Send the token with every write; the store refuses one smaller than it has seen.
    store.write("order-42 shipped", mutex.getFencingToken());
  } finally {
    mutex.unlock();
  }
}

fencingToken is -1 on a thread that holds nothing. The read-write lock's readLock and writeLock have one each, and a semaphore's is the token of the permit the calling thread's release() would give up.

The lease TTL

The lock recipes default to a 10-second lease (leaseTtlSecs), where most other recipes use 2 seconds. A lapsed lease loses the lock, so the longer lease rides out a GC pause or network blip of a few seconds. The trade-off: when a holder's process dies, its lock is freed only when the lease runs out, up to 10 seconds later. Set leaseTtlSecs for your own balance, and use a fencing token where a lost hold must not write.

Reentrancy

Holds are per-thread and reentrant, tracked by holdCount:

DistributedMutex(client, "/locks/orders").use { mutex ->
  mutex.withLock {
    mutex.withLock {
      // Reentrant within the same thread; holdCount is now 2.
      logger.info { "holdCount=${mutex.holdCount}" }
    }
  }
}

A thread that dies holding the lock does not release it

The hold is released when unlock() is called or the recipe is closed — not when the acquiring thread dies. A thread that terminates inside the critical section leaves the lock held until close() or lease expiry. This matches Curator's behaviour. Always release in a finally (or use withLock).

DistributedReadWriteLock

Many concurrent readers, or exactly one writer. Fair: waiters are served FIFO by create revision, so a steady stream of readers cannot starve a waiting writer.

DistributedReadWriteLock(client, "/locks/catalog").use { rwLock ->
  // Any number of readers may hold the read lock at once...
  rwLock.readLock.withLock {
    logger.info { "Reading the catalog" }
  }

  // ...but a writer excludes every reader and every other writer.
  rwLock.writeLock.withLock {
    logger.info { "Rewriting the catalog" }
  }
}
try (DistributedReadWriteLock rwLock = new DistributedReadWriteLock(client, "/locks/catalog")) {
  // Any number of readers may hold the read lock at once...
  rwLock.getReadLock().lock();
  try {
    System.out.println("Reading the catalog");
  } finally {
    rwLock.getReadLock().unlock();
  }

  // ...but a writer excludes every reader and every other writer.
  rwLock.getWriteLock().lock();
  try {
    System.out.println("Rewriting the catalog");
  } finally {
    rwLock.getWriteLock().unlock();
  }
}

DistributedReadWriteLock does not itself implement EtcdLock — it exposes two properties that do, readLock and writeLock. Everything from the mutex section (timeouts, lock-lost listeners, reentrancy) applies to each view independently.

DistributedReadWriteLock(client, "/locks/catalog").use { rwLock ->
  if (rwLock.writeLock.tryLock(5.seconds)) {
    try {
      logger.info { "Got the write lock" }
    } finally {
      rwLock.writeLock.unlock()
    }
  }
}
try (DistributedReadWriteLock rwLock = new DistributedReadWriteLock(client, "/locks/catalog")) {
  if (rwLock.getWriteLock().tryLock(5, TimeUnit.SECONDS)) {
    try {
      System.out.println("Got the write lock");
    } finally {
      rwLock.getWriteLock().unlock();
    }
  }
}

Downgrade works, upgrade throws

Taking the read lock while holding the write lock (downgrade) is safe and supported:

DistributedReadWriteLock(client, "/locks/catalog").use { rwLock ->
  rwLock.writeLock.withLock {
    logger.info { "Writing" }

    // Downgrading write -> read is safe: take the read lock before releasing
    // the write lock, and no other writer can slip in between.
    rwLock.readLock.withLock {
      logger.info { "Still holding the write lock while reading back" }
    }
  }
}

The downgraded read keeps the write hold's place in line, so it is admitted at once even with another process's writer queued behind the write, and that writer keeps waiting until the read is released too. Its fencing token is the write hold's. The rank rides in the read entry's value, so a clientId starting with rank: is rejected, and clients from before 0.13.0 don't honor it: avoid downgrading while a mixed-version fleet shares a lock.

Taking the write lock while holding the read lock (upgrade) is not:

DistributedReadWriteLock(client, "/locks/catalog").use { rwLock ->
  rwLock.readLock.withLock {
    // DON'T: upgrading read -> write self-deadlocks, because the write lock
    // waits on a predecessor this very thread is holding. The recipe throws
    // rather than hanging forever.
    // rwLock.writeLock.lock()
    logger.info { "Release the read lock first, then take the write lock" }
  }
}

Upgrade throws instead of hanging

A read→write upgrade would make the write lock wait on a predecessor that the calling thread itself holds — a self-deadlock that no timeout can distinguish from ordinary contention. Rather than hang, the recipe throws EtcdRecipeRuntimeException. Release the read lock, then take the write lock, and re-validate whatever you read: another writer may have run in between.

DistributedSemaphore

At most N holders across the cluster.

// At most three clients cluster-wide run the guarded section at once.
DistributedSemaphore(client, "/semaphores/api-quota", permits = 3).use { semaphore ->
  semaphore.withPermit {
    logger.info { "Calling the rate-limited API" }
  }
}
// At most three clients cluster-wide run the guarded section at once.
try (DistributedSemaphore semaphore = new DistributedSemaphore(client, "/semaphores/api-quota", 3)) {
  semaphore.acquire();
  try {
    System.out.println("Calling the rate-limited API");
  } finally {
    semaphore.release();
  }
}
DistributedSemaphore(client, "/semaphores/api-quota", permits = 3).use { semaphore ->
  if (semaphore.tryAcquire(2.seconds)) {
    try {
      logger.info { "Got a permit; ${semaphore.availablePermits()} left (advisory)" }
    } finally {
      semaphore.release()
    }
  } else {
    logger.info { "All permits taken; shedding load" }
  }
}
try (DistributedSemaphore semaphore = new DistributedSemaphore(client, "/semaphores/api-quota", 3)) {
  if (semaphore.tryAcquire(2, TimeUnit.SECONDS)) {
    try {
      System.out.println("Got a permit; " + semaphore.availablePermits() + " left (advisory)");
    } finally {
      semaphore.release();
    }
  } else {
    System.out.println("All permits taken; shedding load");
  }
}

Semaphore holds are not lock holds

DistributedSemaphore deliberately does not implement EtcdLock, because its holds follow java.util.concurrent.Semaphore's rules rather than a lock's:

EtcdLock DistributedSemaphore
Ownership The acquiring thread The instance
Who may release Only the holder Any thread
Reentrant Yes No
Release order — The caller's own permit first
DistributedSemaphore(client, "/semaphores/api-quota", permits = 3).use { semaphore ->
  // Holds are instance-level, not thread-owned: unlike EtcdLock, any thread may
  // release a permit taken by another. Releases are LIFO.
  semaphore.acquire()
  semaphore.acquire()
  logger.info { "Holding two permits" }
  semaphore.release()
  semaphore.release()
}

Permit loss mirrors lock loss — a PermitLostListener on the notifier thread, connectionState → LOST, release() returning false, and an opt-in interruptOnPermitLoss. A release() gives up a live permit the calling thread acquired, else the one it lost (returning false), so a thread whose permit was lost never gives up another thread's live permit. Only when the releasing thread acquired nothing does it fall back to any of the instance's permits.

close() aborts acquisitions in flight: one that close() lands on throws rather than acquiring on a closed recipe. That holds for locks too.

DistributedSemaphore(client, "/semaphores/api-quota", permits = 3).use { semaphore ->
  semaphore.addPermitLostListener { cause ->
    logger.warn { "Permit lost, stop doing the guarded work: $cause" }
  }
  semaphore.withPermit { logger.info { "Working" } }
}

The permit count is fixed by the first writer

The canonical count is CAS-created at <semaphorePath>/permits by whichever instance reaches the path first. An instance that later names the same path with a different count throws rather than silently reconfiguring the semaphore under everyone else:

// The permit count is CAS-created at <path>/permits by whoever gets there first.
// A later instance naming the same path with a different count is a programming
// error, not a silent reconfiguration, so it throws.
try {
  DistributedSemaphore(client, "/semaphores/api-quota", permits = 5).use { semaphore ->
    semaphore.withPermit { logger.info { "Never reached if 3 was established first" } }
  }
} catch (e: SemaphorePermitMismatchException) {
  logger.error { "Requested ${e.requestedPermits}, but the path is fixed at ${e.canonicalPermits}" }
}

availablePermits() is advisory

So is isLocked on a lock. Both are true-at-some-recent-revision, and another client may act between your read and your next line. Use them for logging and dashboards; never branch on them to decide whether an acquire will succeed. Use tryAcquire/tryLock for that — they are atomic. availablePermits() is also an RPC under the semaphore's own budget; availablePermits(rpc) takes a different one.

Coroutines

Every lock has suspending twins that release the thread while waiting. Note the asymmetry: withLock is scoped-only for locks (a lock is thread-owned, so the suspending version pins a confined dispatcher and releases under NonCancellable), while the semaphore also exposes split awaitAcquire/awaitRelease because its holds are instance-level. With interruptOnLockLoss / interruptOnPermitLoss, a lost hold cancels the suspending withLock / withPermit body, which then throws HoldLostException. See Coroutines.

Observability

With the Micrometer module wired in, locks report etcd.lock.wait (tagged acquired/timeout) and etcd.lock.hold timers. Lock paths never become tags — that would blow up cardinality. See Observability.