Skip to content

Counters

DistributedAtomicLong is a single Long in etcd that any number of clients can safely mutate. It is the distributed answer to java.util.concurrent.atomic.AtomicLong — with one honest difference that runs through this whole page: every operation is a network round trip, and the value is contended by machines you cannot see.

It extends EtcdConnector, so it shares the lifecycle, exception, and connection-state surface described in Core concepts.

DistributedAtomicLong(client, "/counters/orders").use { counter ->
  // Optional: start() is invoked automatically on first use. Call it explicitly
  // when you want the create-if-absent round trip to happen at a point you chose,
  // rather than inside whatever your first increment() turns out to be.
  counter.start()

  logger.info { "Current: ${counter.get()}" }
}
try (DistributedAtomicLong counter = new DistributedAtomicLong(client, "/counters/orders")) {
  // Optional: start() is invoked automatically on first use. Call it explicitly
  // when you want the create-if-absent round trip to happen at a point you chose.
  counter.start();

  System.out.println("Current: " + counter.get());
}

start() is optional

Unusually for this library, you do not have to call start(). It is invoked automatically on the first call that needs the counter to exist, and calling it explicitly is also fine.

What start() does is create the key with default if it is absent — a single If(doesNotExist) Then(setTo default) transaction, so concurrent first-users cannot double-create it. The reason it is lazy is that the constructor used to do this work, which made a DistributedAtomicLong impossible to construct without a live etcd. Deferring it keeps construction I/O-free without changing what existing call sites see.

Call it explicitly when you want that round trip to happen at a point you chose — during startup, say — rather than inside whatever your first increment() turns out to be.

The default value belongs to whoever creates the key

// The default is written only by whichever client creates the key first, and is
// ignored by every instance that finds the counter already present.
DistributedAtomicLong(client, "/counters/seats", default = 100L).use { counter ->
  logger.info { "Seats left: ${counter.decrement()}" }
}
// Java has no named arguments: the third positional parameter is `default`, and
// it is honoured only by whichever client creates the key first.
try (DistributedAtomicLong counter = new DistributedAtomicLong(client, "/counters/seats", 100L)) {
  System.out.println("Seats left: " + counter.decrement());
}

default is only written by whichever client wins the create. Every instance that finds the counter already present ignores its own default entirely. Two clients naming the same path with different defaults do not conflict, do not throw, and do not reconfigure anything — the first one to arrive simply decides, and the other never learns it disagreed.

Mutating

DistributedAtomicLong(client, "/counters/orders").use { counter ->
  // Each mutator returns the value THIS call committed, not a re-read. By the time
  // you look at it another client may already have moved the counter past it.
  val afterIncrement: Long = counter.increment()
  val afterDecrement: Long = counter.decrement()
  val afterAdd: Long = counter.add(10L)
  val afterSubtract: Long = counter.subtract(4L)

  logger.info { "$afterIncrement $afterDecrement $afterAdd $afterSubtract" }

  // get() is a plain read: true at some recent revision, possibly stale on the
  // next line. Never branch on it to decide whether a mutation will succeed.
  logger.info { "Now: ${counter.get()}" }
}
try (DistributedAtomicLong counter = new DistributedAtomicLong(client, "/counters/orders")) {
  // Each mutator returns the value THIS call committed, not a re-read. By the time
  // you look at it another client may already have moved the counter past it.
  long afterIncrement = counter.increment();
  long afterDecrement = counter.decrement();
  long afterAdd = counter.add(10L);
  long afterSubtract = counter.subtract(4L);

  System.out.println(afterIncrement + " " + afterDecrement + " " + afterAdd + " " + afterSubtract);

  // get() is a plain read: true at some recent revision, possibly stale on the
  // next line. Never branch on it to decide whether a mutation will succeed.
  System.out.println("Now: " + counter.get());
}

increment(), decrement(), add(n), and subtract(n) all return a Long, and which Long is the most important thing on this page.

Mutators return the value this call committed

Not a fresh read. The recipe reads the counter, computes value + delta, and commits it with a compare-on-modRevision transaction; on success it hands back the number it just wrote.

It deliberately does not re-read the key afterwards. An earlier version did, and under contention another client could mutate the counter between the commit and the re-read — so callers were handed a value they had not committed, which is exactly the guarantee an atomic counter exists to provide. If ten clients each increment() a counter from 0, the ten return values are 1 through 10 in some order, every value distinct. That property is what lets you use increment() to hand out unique IDs.

The corollary: the returned value is not "the counter's current value". It was, for an instant, at the revision your transaction committed at. By the time you read it, it may not be.

get() is a plain read with no such guarantee: true at some recent revision, possibly stale on the next line. Use it for logging and dashboards. Never branch on it to decide whether a mutation will succeed — that is what the mutators' atomicity is for.

How the optimistic concurrency works

There is no lock. Each mutation is:

  1. GET the counter, keeping its modRevision.
  2. Compute the new value locally.
  3. Txn: if modRevision is still what step 1 saw, then write the new value.
  4. If the transaction succeeded, return the value written. If not, another client wrote first — back off and start over from step 1.

This is compare-and-swap over the network, and it has the same shape as AtomicLong.incrementAndGet()'s retry loop. The difference is the cost of a lost race: a CAS loss in a JVM costs nanoseconds, and a CAS loss here costs a round trip plus a backoff sleep.

A hot counter is a contention point, not a throughput device

The retry loop is unbounded (only close() ends it early), and its backoff is a random sleep from a window that widens with each failed attempt, up to one second. Under low contention that is invisible. Under high contention — many clients hammering one path — throughput collapses: every client is serialized through one etcd key, and each loser pays a full round trip plus a sleep before trying again. There is no fairness and no queue, so an unlucky client can lose repeatedly while others make progress.

A single DistributedAtomicLong is a coordination primitive, not a metrics counter. If you are counting requests, count them locally and flush periodically. If you need per-shard counts, use per-shard paths — N uncontended counters cost the same as one uncontended counter and scale linearly; one counter shared by N clients does not.

Note also that a mutation is two round trips, not one: the GET and the Txn. That is the price of returning the value you committed.

A mutator that throws may still have counted

If increment() or another mutator throws because an RPC failed or timed out, the outcome is unknown. The transaction may have been applied even though its response was lost, so blindly repeating the call can count twice. When that matters, read get() to reconcile, or make the increment idempotent at a higher level.

Deleting

// Static, and deliberately not an instance method: removing the key out from under
// live instances is a destructive act, not a step in a counter's lifecycle.
DistributedAtomicLong.delete(client, "/counters/orders")
// Static, and deliberately not an instance method: removing the key out from under
// live instances is a destructive act, not a step in a counter's lifecycle.
DistributedAtomicLong.delete(client, "/counters/orders");

delete is @JvmStatic and takes a Client and a path. It is deliberately not an instance method: removing the key out from under live instances is a destructive administrative act, not a step in a counter's lifecycle. Live instances keep working after it: get() reads the absent counter as that instance's default, and the next update from any instance re-creates the key from its default.

Closing a DistributedAtomicLong does not delete anything. The counter is durable etcd state with no lease behind it; it outlives every process that ever touched it, until someone deletes it.

Scoped use

val committed: Long = withDistributedAtomicLong(client, "/counters/orders") { increment() }
logger.info { "Committed: $committed" }

Passing resilience

withDistributedAtomicLong takes resilience too, as does the constructor:

withDistributedAtomicLong(client, "/counters/orders", resilience = ResilienceConfig.DEFAULT) {
  logger.info { "Current: ${get()}" }
}

// Or on the constructor:
DistributedAtomicLong(client, "/counters/orders", resilience = ResilienceConfig.DEFAULT).use { counter ->
  logger.info { "Current: ${counter.get()}" }
}

Note what resilience does and does not cover here. It governs the individual RPCs — retries and a timeout on the GET, and a single bounded attempt at the Txn. The CAS retry loop is separate and always on; you cannot turn it off, because a lost CAS is not a failure, it is the algorithm working. See Resilience.

Java

DistributedAtomicLong is one of the friendliest recipes from Java: no generics, no codecs, no lambdas required. The only thing to watch is that Java has no named arguments, so reaching default means passing it positionally. See the Java guide.