Monitoring & Observability

Both the proxy and agent expose their own operational metrics and admin endpoints.

Enabling Metrics

Metrics are disabled by default. Enable them via CLI, environment variables, or config file.

proxy {
  metrics {
    enabled = true
    port = 8082
    path = "metrics"
  }

  admin {
    enabled = true
    port = 8092
  }
}

agent {
  metrics {
    enabled = true
    port = 8083
    path = "metrics"
  }

  admin {
    enabled = true
    port = 8093
  }
}

Scraping Internal Metrics

The proxy runs next to Prometheus, so Prometheus scrapes its metrics port directly. The agents sit behind firewalls, so give each agent a path for its own /metrics and scrape that path through the proxy, one job per agent (the Agents dashboard tells agents apart by job):

agent {
  pathConfigs: [
    # ... the agent's targets, plus its own metrics:
    { name: "Agent metrics", path: cluster_a_agent_metrics, url: "http://localhost:8083/metrics" }
  ]
}

Then add the scrape jobs to your prometheus.yml:

scrape_configs:
  # Scrape proxy internal metrics
  - job_name: 'prometheus-proxy'
    metrics_path: /metrics
    static_configs:
      - targets: ['proxy-host.example.com:8082']

  # Scrape each agent's internal metrics through the proxy, one job per agent. The agent
  # registers its own /metrics as a path (cluster_a_agent_metrics here).
  - job_name: 'agent-cluster-a'
    metrics_path: /cluster_a_agent_metrics
    static_configs:
      - targets: ['proxy-host.example.com:8080']

JVM Metrics

Both components support optional JVM metrics:

proxy.metrics {
  enabled = true

  // Optional JVM metrics
  standardExportsEnabled = true
  memoryPoolsExportsEnabled = true
  garbageCollectorExportsEnabled = true
  threadExportsEnabled = true
  classLoadingExportsEnabled = false
  versionInfoExportsEnabled = false
  bufferPoolExportsEnabled = true
  compilationExportsEnabled = true
  nativeMemoryExportsEnabled = false  // Needs the JVM's Native Memory Tracking; see below
}

The same options are available under agent.metrics.

The JVM metrics come from the Prometheus Java client 1.x. Since the move to it, the memory metrics put the unit last (jvm_memory_used_bytes, not jvm_memory_bytes_used; jvm_memory_pool_used_bytes, not jvm_memory_pool_bytes_used; the same for committed, max and init), jvm_info is jvm_runtime_info, and memoryPoolsExportsEnabled also adds jvm_memory_pool_allocated_bytes_total.

Since 4.3.0, three more groups are available, also off by default:

• bufferPoolExportsEnabled adds jvm_buffer_pool_used_bytes, jvm_buffer_pool_capacity_bytes and jvm_buffer_pool_used_buffers, by pool (direct and mapped): the memory the JVM's NIO buffer pools hold outside the heap.
• compilationExportsEnabled adds jvm_compilation_time_seconds_total, the time the JIT compiler has spent.
• nativeMemoryExportsEnabled adds jvm_native_memory_reserved_bytes and jvm_native_memory_committed_bytes, by pool. It reads the JVM's Native Memory Tracking, which is off unless the JVM starts with -XX:NativeMemoryTracking=summary (for the Docker images, set JAVA_TOOL_OPTIONS=-XX:NativeMemoryTracking=summary). Without it the flag exports nothing, and tracking adds some overhead of its own.

Proxy Metrics

Counters

proxy_scrape_requests_total type labels:

Histograms

A path's series are removed when its last registration goes away, whether it is unregistered or its agent disconnects, so a retired path stops appearing on /metrics. A scrape still in flight when that happens is not recorded, so it cannot bring the series back.

The latency histogram's outcome label takes the proxy_scrape_requests_total type values above, so latency can be split by result.

Latency buckets: 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 60s, 90s

Response size buckets: 1KB, 10KB, 100KB, 500KB, 1MB, 5MB, 10MB

Gauges

Agent Metrics

Counters

The launch_id label uniquely identifies each agent process lifetime. unsuccessful means the target answered with a non-2xx status or a response over agent.http.maxContentLengthMBytes; timed_out means the target didn't answer within the agent's scrape timeout; failure means the fetch failed without an answer, such as a refused connection or a DNS failure; invalid_path means the proxy asked for a path the agent doesn't serve; dropped is a result lost because the connection to the proxy closed before it was sent.

The two agent_filter_* counters only create series for paths that have a filter configured, so they are absent entirely unless metric filtering is in use. See that section for what the filter does and does not drop.

Histograms

It has the proxy latency histogram's buckets, up to 90s, past the agent's 15s scrape timeout.

Gauges

Metric Flow

Prometheus --- HTTP GET ---> Proxy                        Agent
                              |                             |
                  records the start time                    |
                              |                             |
                  writeScrapeRequest() -- gRPC stream --> fetchScrapeUrl()
                              |                     agentLatency.startTimer()
                              |                             |
                              |                     HTTP GET to target
                              |                             |
                              |                     agentLatency.observeDuration()
                              |                     scrapeRequestCount.inc()
                              |                     scrapeResultCount.inc()
                              |                             |
                  assignScrapeResults() <-- gRPC -----------+
                              |
                  responseBytes.observe()
                  latency.observe(elapsed)
                  scrapeRequestCount.labelValues(outcome).inc()
                              |
                <-- HTTP response ---

PromQL Examples

Scrape Success Rate

# Scrape success rate (last 5 minutes):
sum(rate(proxy_scrape_requests_total{type="success"}[5m]))
  / sum(rate(proxy_scrape_requests_total[5m])) * 100

P99 Scrape Latency

# P99 scrape latency:
histogram_quantile(0.99,
  sum by (le) (rate(proxy_scrape_request_latency_seconds_bucket[5m]))
)

P99 Latency Per Path

# P99 latency per path:
histogram_quantile(0.99,
  sum by (le, path) (rate(proxy_scrape_request_latency_seconds_bucket[5m]))
)

Error Rate by Type

# Error rate by type:
sum by (type) (rate(proxy_scrape_requests_total{type!="success"}[5m]))

Agent Latency by Name

# Agent scrape latency by agent name:
histogram_quantile(0.99,
  sum by (le, agent_name) (rate(agent_scrape_request_latency_seconds_bucket[5m]))
)

Admin Endpoints

Admin endpoints (when admin is enabled):

Proxy (default port 8092):
  GET /ping         - Returns "pong" (liveness check)
  GET /healthcheck  - Returns health status JSON
  GET /version      - Returns version info
  GET /threaddump   - Returns JVM thread dump
  GET /debug        - Returns proxy debug info (if debug enabled)

Agent (default port 8093):
  GET /ping         - Returns "pong" (liveness check)
  GET /healthcheck  - Returns health status JSON
  GET /version      - Returns version info
  GET /threaddump   - Returns JVM thread dump
  GET /debug        - Returns agent debug info (if debug enabled)

Enable admin endpoints:

java -jar prometheus-proxy.jar --admin
java -jar prometheus-agent.jar --admin

Grafana Dashboards

Importing Grafana dashboards:

1. In Grafana, go to Dashboards > Import
2. Upload the JSON file or paste its contents
3. Open the dashboard and pick your Prometheus datasource in its Datasource drop-down

Dashboard files in the repository:
  grafana/prometheus-proxy.json   - Proxy health, throughput, latency, errors
  grafana/prometheus-agents.json  - Agent health, scrape activity, per-agent latency

Requirements:
  - Grafana 10.0 or later
  - A Prometheus datasource scraping the proxy's and the agents' metrics endpoints,
    with each agent in its own job

Proxy Dashboard

Key panels to monitor:

The Job and Instance pickers pick out one proxy when Prometheus scrapes more than one, such as a high-availability pair.

Agents Dashboard

Key panels to monitor:

Every agent panel keys on job, so scrape each agent in its own job, as in Scraping Internal Metrics.