Skip to content

Service Discovery

Prometheus Proxy supports Prometheus HTTP service discovery, allowing Prometheus to automatically discover scrape targets as agents register paths.

Enabling Service Discovery

# Enable service discovery via CLI:
java -jar prometheus-proxy.jar \
  --sd_enabled \
  --sd_path discovery \
  --sd_target_prefix http://proxy-host.example.com:8080/

proxy {
  service.discovery {
    enabled = true
    path = "discovery"
    targetPrefix = "http://proxy-host.example.com:8080/"
  }
}

# Enable service discovery via environment variables:
SD_ENABLED=true
SD_PATH=discovery
SD_TARGET_PREFIX=http://proxy-host.example.com:8080/

Discovery Endpoint

Once enabled, the proxy exposes a JSON endpoint at the configured path (default: /discovery).

Query it with:

curl -s http://proxy-host.example.com:8080/discovery | jq '.'

Response Format

The endpoint returns a JSON array in the format expected by Prometheus HTTP SD:

[
  {
    "targets": ["proxy-host.example.com:8080"],
    "labels": {
      "__metrics_path__": "/app1_metrics",
      "agentName": "agent-01",
      "hostName": "internal.host.com",
      "env": "production",
      "team": "backend"
    }
  },
  {
    "targets": ["proxy-host.example.com:8080"],
    "labels": {
      "__metrics_path__": "/app2_metrics",
      "agentName": "agent-01",
      "hostName": "internal.host.com",
      "env": "staging",
      "team": "backend"
    }
  }
]

Each entry contains:

Field Description
targets Array containing the proxy endpoint (from targetPrefix)
labels.__metrics_path__ The path to scrape on the proxy
labels.agentName Name(s) of agent(s) serving this path
labels.hostName Hostname(s) of agent(s) serving this path
labels.* Custom labels from agent pathConfigs

Prometheus Configuration

Configure Prometheus to use HTTP service discovery:

scrape_configs:
  - job_name: 'prometheus-proxy-sd'
    http_sd_configs:
      - url: 'http://proxy-host.example.com:8080/discovery'
        refresh_interval: 30s

The refresh_interval controls how often Prometheus queries the discovery endpoint for changes.

Dynamic target management

With service discovery enabled, you don't need to update prometheus.yml when agents register new paths. Prometheus automatically picks up new targets on the next refresh interval.

Target Prefix

The targetPrefix setting determines the base URL in the targets array. Set this to the external address of your proxy:

proxy.service.discovery.targetPrefix = "http://proxy.example.com:8080/"

This ensures Prometheus can reach the proxy from its network location.

Labels

Labels configured in agent pathConfigs are included in the service discovery response. Prometheus can use these for relabeling, filtering, or target grouping:

agent.pathConfigs: [
  {
    name: "Production API"
    path: prod_api_metrics
    labels: "{\"env\": \"production\", \"team\": \"backend\", \"service\": \"api\"}"
    url: "http://api.internal:9100/metrics"
  }
]

The labels appear in the discovery response and can be used in Prometheus relabeling rules.