Agent Configuration

The agent runs inside the firewall and scrapes local metrics endpoints on behalf of the proxy.

Path Configs

The pathConfigs array defines which metrics endpoints the agent exposes through the proxy:

Basic Configuration

agent {
  proxy.hostname = "proxy-host.example.com"

  pathConfigs: [
    {
      name: "My App metrics"
      path: my_app_metrics
      url: "http://localhost:9100/metrics"
    }
  ]
}

Multiple Endpoints

agent {
  pathConfigs: [
    {
      name: "App1 metrics"
      path: app1_metrics
      url: "http://app1.local:9100/metrics"
    },
    {
      name: "App2 metrics"
      path: app2_metrics
      url: "http://app2.local:9100/metrics"
    },
    {
      name: "App3 metrics"
      path: app3_metrics
      url: "http://app3.local:9100/metrics"
    }
  ]
}

With Labels

Labels are included in service discovery responses and can be used for filtering:

agent {
  pathConfigs: [
    {
      name: "Production API"
      path: prod_api_metrics
      labels: "{\"env\": \"production\", \"team\": \"backend\"}"
      url: "http://api.internal:9100/metrics"
    },
    {
      name: "Staging API"
      path: staging_api_metrics
      labels: "{\"env\": \"staging\", \"team\": \"backend\"}"
      url: "http://api.staging:9100/metrics"
    }
  ]
}

Each path config entry has these fields:

Dynamic Target Discovery

By default pathConfigs is read once at startup, so adding or removing a target means editing the config and restarting the agent. Enable dynamic discovery to have the agent reconcile its registered paths against a watched file at runtime — no restart, and paths that did not change keep scraping:

agent {
  discovery {
    enabled = false                                    // Enable dynamic discovery
    file.path = "/etc/prometheus-proxy/targets.conf"   // HOCON/JSON list of paths
    reconcileIntervalSecs = 30                         // Poll and full-resync interval
  }
}

The discovery file holds a paths list of the same { name, path, url, labels } entries as pathConfigs:

paths = [
  { name = "app1", path = "app1_metrics", url = "http://app1:9090/metrics" }
  { name = "app2", path = "app2_metrics", url = "http://app2:9090/metrics" }
]

Every interval the agent registers newly-listed paths, unregisters removed ones, and re-registers a path whose URL or labels changed.

Because the file is a plain list, it can be generated rather than hand-maintained — a Kubernetes ConfigMap mounted into the agent pod (see the Kubernetes walkthrough), an Ansible template, or a cron job querying a service registry. Automation then needs permission only to write a file, never to restart the agent, and the churning target list stays separate from the stable config holding ports, TLS, and the proxy address.

A fully annotated version of this file -- required versus defaulted fields, the single-segment path rule, HOCON substitutions, and the empty-versus-deleted distinction -- ships as examples/discovery-targets.conf and is shown verbatim on the Example Configs page.

Dynamic target discovery is distinct from Prometheus service discovery, which exposes an endpoint so Prometheus can find proxied targets; discovery instead lets the agent pick up target changes behind the firewall without a restart.

On the proxy's dashboard, each path is tagged cfg (static) or disc (discovered), so when a target is unexpectedly present or absent you can tell at a glance whether a human or the automation registered it.

Proxy Connection

agent {
  proxy {
    hostname = "proxy-host.example.com"   // Proxy hostname
    port = 50051                          // Proxy gRPC port
  }
}

Or specify on the command line:

java -jar prometheus-agent.jar --proxy proxy-host.example.com:50051 --config agent.conf

Proxy failover

For redundancy, give the agent an ordered list of proxy endpoints instead of a single hostname. It connects to the first that answers and moves to the next on a failed connect; when a working connection drops it returns to the head of the list, so a recovered primary is picked up on the next reconnect. Only one connection is active at a time.

agent {
  proxy {
    port = 50051                                    // Default port for entries that omit one
    endpoints = [
      "proxy-a.example.com:50051",                  // Tried first
      "proxy-b.example.com:50051"                   // Used when proxy-a is unreachable
    ]
  }
}

--proxy and PROXY_HOSTNAME accept the same list as a comma-separated value, and either one replaces endpoints entirely rather than adding to it:

java -jar prometheus-agent.jar --proxy proxy-a.example.com:50051,proxy-b.example.com:50051

A single value behaves exactly as it always has, so existing configurations need no change. hostname is used only when endpoints is empty; the two are never merged. An unparseable entry fails at startup rather than surfacing later as a connection error.

See High availability for the Prometheus side — in particular, scrape an HA pair with static_config, never http_sd_config.

Agent Authentication

If the proxy requires a pre-shared agent token, set the matching value on the agent. It is presented as a gRPC metadata header on every call and is never logged. Resolved from --agent_token → AGENT_TOKEN → agent.agentToken; empty (the default) sends no token.

agent {
  agentToken = "shared-secret"   // Must match the proxy's proxy.agentToken
}

HTTP Client Settings

Configure how the agent makes HTTP requests to scrape endpoints:

agent {
  http {
    maxConcurrentClients = 5          // Parallel scrape limit
    clientTimeoutSecs = 30            // HTTP client timeout
    maxContentLengthMBytes = 10       // Max response size in MB
    enableTrustAllX509Certificates = false
    trustStorePath = ""               // JKS/PKCS12 trust store for HTTPS targets (custom CA); empty = JDK default
    trustStorePassword = ""           // Password for trustStorePath
  }
}

HTTP Client Cache

The agent caches HTTP clients keyed by authentication credentials (for basic auth / bearer token scenarios). Configure cache behavior:

agent {
  http.clientCache {
    maxSize = 100                     // Maximum cached HTTP clients
    maxAgeMins = 30                   // Maximum age of cached clients
    maxIdleMins = 10                  // Idle time before eviction
    cleanupIntervalMins = 5           // Cleanup run interval
  }
}

Scraping HTTPS Endpoints

For HTTPS scrape targets signed by a private or internal CA, point the agent at a trust store that contains that CA (--https_truststore / HTTPS_TRUST_STORE_PATH / agent.http.trustStorePath, with the matching *_password) so certificates are still validated. An empty path uses the JDK default trust store, and --trust_all_x509 (which disables verification entirely) takes precedence. See Scraping HTTPS Endpoints for details.

Scrape Settings

agent {
  scrapeTimeoutSecs = 15              // Total scrape timeout including retries
  scrapeMaxRetries = 0                // 0 disables retries
  chunkContentSizeKbs = 32            // Chunking threshold in KB
  minGzipSizeBytes = 512              // Minimum size for gzip compression
}

Metric Filtering

The agent-to-proxy hop is usually the expensive one — a WAN link, metered egress, a constrained tunnel — yet filtering with Prometheus metric_relabel_configs discards data only after it has crossed it. Filtering at the agent drops unwanted metric families before they are sent, and it runs before the payload is gzipped and chunked, so the saving composes with chunkContentSizeKbs and minGzipSizeBytes above. Filters are declared in a top-level agent.filters list, keyed by path rather than nested inside pathConfigs:

agent {
  pathConfigs: [
    {
      name: "app1"
      path: "app1_metrics"
      url: "http://app1:9090/metrics"
    }
  ]

  // Drop Go runtime GC metrics for app1 before they cross the WAN.
  // Regexes are fully anchored, matching Prometheus relabel_config semantics:
  // "go_" matches nothing; "go_.*" matches go_gc_duration_seconds.
  filters: [
    {
      path: "app1_metrics"
      metricNameAllow: []
      metricNameDeny: [ "go_gc_.*", "go_memstats_.*" ]
    }
  ]
}

Both list fields must be present on every element -- write metricNameAllow: [] for a deny-only filter. The default is filters: [] (no filtering), so an existing config with no filters key loads unchanged.

Regexes are fully anchored (Regex.matches()), matching Prometheus relabel_config / metric_relabel_configs semantics: deny: [ "go_" ] matches nothing, deny: [ "go_.*" ] is required to match go_goroutines.

Matching is against the metric family, not each series. A # HELP or # TYPE line (or # UNIT, for OpenMetrics) opens a family and the allow/deny verdict is computed once against the family name; every sample line that belongs to that family -- the exact name, or the name plus a recognized histogram/summary/OpenMetrics suffix (_bucket, _sum, _count, _created, _total, _gsum, _gcount, _info) -- inherits that verdict instead of being matched again. A histogram's _bucket, _sum, and _count series are therefore always kept or dropped together, along with the family's metadata lines. One consequence worth knowing: once a family is open, a series-level rule such as metricNameDeny: [ "http_req_duration_seconds_bucket" ] is a silent no-op, since the _bucket lines inherit the family's verdict rather than being matched themselves. A sample line with no open family (a payload with no TYPE lines at all) is judged literally on its own full name.

An empty metricNameAllow allows everything. metricNameDeny is applied after allow, so deny wins on overlap.

To make family scoping concrete, given this filter:

{ path: "app1_metrics", metricNameAllow: [], metricNameDeny: [ "go_.*" ] }

a scraped payload is transformed like this:

# HELP go_goroutines Number of goroutines
# TYPE go_goroutines gauge
go_goroutines 12
# HELP http_req_duration_seconds Request duration
# TYPE http_req_duration_seconds histogram
http_req_duration_seconds_bucket{le="0.1"} 5
http_req_duration_seconds_bucket{le="+Inf"} 9
http_req_duration_seconds_sum 3.2
http_req_duration_seconds_count 9

# HELP http_req_duration_seconds Request duration
# TYPE http_req_duration_seconds histogram
http_req_duration_seconds_bucket{le="0.1"} 5
http_req_duration_seconds_bucket{le="+Inf"} 9
http_req_duration_seconds_sum 3.2
http_req_duration_seconds_count 9

The go_goroutines family goes entirely, # HELP and # TYPE lines included. The histogram is untouched: deny never matched it, and its _bucket / _sum / _count series are evaluated as one family rather than individually — which is what stops a filter from delivering a histogram with some of its series missing.

Filters apply by path to both statically configured paths and paths added by dynamic discovery; a filter's path is matched the same way regardless of which route registered it. An invalid regex fails agent startup rather than surfacing later at scrape time.

Registered http://app1:9090/metrics as /app1_metrics with labels {} (static) with a metric filter

Two counters, both labeled by launch_id and path, track filtering and are only created for paths that actually have a filter configured:

Not implemented: dropLabels, metric renaming/relabeling, and an agent-global filter -- every filter is per-path.

Consolidated Mode

By default, each path is owned by a single agent. Enable consolidated mode to allow multiple agents to register the same path:

agent {
  consolidated = true                 // Allow multiple agents per path

  proxy.hostname = "proxy-host.example.com"

  pathConfigs: [
    {
      name: "Shared metrics"
      path: shared_metrics
      url: "http://localhost:9100/metrics"
    }
  ]
}

This is useful for redundancy -- if one agent goes down, another can serve the same path.

Agent Naming

Give agents descriptive names for easier identification in logs and metrics:

java -jar prometheus-agent.jar --name production-agent-01 --config agent.conf

Or in the config file:

agent.name = "production-agent-01"

If no name is provided, the agent uses Unnamed-<hostname>.

Full Example

Here is a real-world config from the examples/ directory:

proxy {
  admin.debugEnabled = false

  admin.enabled: true
  metrics.enabled: true

  http.requestLoggingEnabled: true
}

agent {
  proxy.hostname = localhost
  admin.enabled: true
  metrics.enabled: true

  pathConfigs: [
    {
      name: "Proxy metrics"
      path: proxy_metrics
      labels: "{\"key1\": \"value1\", \"key2\": 2}"
      url: "http://localhost:8082/metrics"
      //url: "http://"${?HOSTNAME}":8082/metrics"
    }
    {
      name: "Agent metrics"
      path: agent_metrics
      labels: "{\"key3\": \"value3\", \"key4\": 4}"
      url: "http://localhost:8083/metrics"
      //url: "http://"${?HOSTNAME}":8083/metrics"
    }
  ]
}

See also:

• examples/myapps.conf -- multiple endpoints
• examples/federate.conf -- Prometheus federation
• examples/discovery-targets.conf -- a dynamic discovery targets file (watched, not passed to --config)