Service Discovery

Prometheus Proxy supports Prometheus HTTP service discovery, allowing Prometheus to automatically discover scrape targets as agents register paths.

Enabling Service Discovery

# Enable service discovery via CLI:
java -jar prometheus-proxy.jar \
  --sd_enabled \
  --sd_path discovery \
  --sd_target_prefix proxy-host.example.com:8080

proxy {
  service.discovery {
    enabled = true
    path = "discovery"
    targetPrefix = "proxy-host.example.com:8080"
  }
}

# Enable service discovery via environment variables:
SD_ENABLED=true
SD_PATH=discovery
SD_TARGET_PREFIX=proxy-host.example.com:8080

Discovery Endpoint

Once enabled, the proxy exposes a JSON endpoint at the configured path (default: /discovery).

Query it with:

curl -s http://proxy-host.example.com:8080/discovery | jq '.'

Response Format

The endpoint returns a JSON array in the format expected by Prometheus HTTP SD:

[
  {
    "targets": ["proxy-host.example.com:8080"],
    "labels": {
      "__metrics_path__": "/app1_metrics",
      "agentName": "agent-01",
      "hostName": "internal.host.com",
      "env": "production",
      "team": "backend"
    }
  },
  {
    "targets": ["proxy-host.example.com:8080"],
    "labels": {
      "__metrics_path__": "/app2_metrics",
      "agentName": "agent-01",
      "hostName": "internal.host.com",
      "env": "staging",
      "team": "backend"
    }
  }
]

Each entry contains:

Prometheus Configuration

Configure Prometheus to use HTTP service discovery:

scrape_configs:
  - job_name: 'prometheus-proxy-sd'
    http_sd_configs:
      - url: 'http://proxy-host.example.com:8080/discovery'
        refresh_interval: 30s

The refresh_interval controls how often Prometheus queries the discovery endpoint for changes.

Target Prefix

The targetPrefix setting is the address Prometheus scrapes the advertised paths at: the external host:port of your proxy, as Prometheus reaches it. The default is localhost:8080.

proxy.service.discovery.targetPrefix = "proxy.example.com:8080"

It can also be an http or https URL, such as https://proxy.example.com/prometheus/. Prometheus doesn't accept a URL as a target, so the proxy splits it: the host and port become the target, the scheme becomes each target's __scheme__ label, and any path goes in front of __metrics_path__. A value that is neither fails the proxy's startup when service discovery is enabled.

Labels

Labels configured in agent pathConfigs are included in the service discovery response. Prometheus can use these for relabeling, filtering, or target grouping:

agent.pathConfigs: [
  {
    name: "Production API"
    path: prod_api_metrics
    labels: "{\"env\": \"production\", \"team\": \"backend\", \"service\": \"api\"}"
    url: "http://api.internal:9100/metrics"
  }
]

The labels appear in the discovery response and can be used in Prometheus relabeling rules. They must be a JSON object whose keys are label names that every Prometheus version accepts ([a-zA-Z_][a-zA-Z0-9_]*) and whose values are strings. Prometheus decodes the labels as a map of string to string, and one entry that isn't fails its whole discovery response, so the proxy leaves such entries out. The proxy logs a warning naming them when the path registers, and the agent logs one at startup for a pathConfigs entry. The path itself still registers.

A path can't be the discovery path, favicon.ico, or the blitz path: the proxy serves those itself, so it refuses to register them.