Security

Security Model

Prometheus Proxy is designed to be firewall-friendly:

• The agent initiates an outbound gRPC connection to the proxy
• No inbound ports need to be opened on the firewall
• Agent connections can be authenticated with an optional pre-shared token — optionally scoped to per-agent identities — and/or mutual TLS
• Stale agent connections are automatically cleaned up
• The proxy's scrape port, which Prometheus scrapes, has no authentication, so keep it where only Prometheus can reach it (see Scrape Port Access)

TLS Encryption

Agents connect to the proxy using gRPC, which supports TLS with or without mutual authentication.

See TLS Setup for detailed configuration instructions.

Agent Authentication (Pre-Shared Token)

By default the proxy accepts agent gRPC connections with no application-level authentication. Set a shared pre-shared token so the proxy rejects agents that do not present it:

Both sides must use the same value. When set, the agent attaches the token as a gRPC metadata header on every call and the proxy rejects any call with a missing or mismatched token (UNAUTHENTICATED). When the token is empty (the default), the open behavior is preserved and the proxy logs a startup warning — unless mutual TLS is configured, which already authenticates agents. The token is never logged. If a token or per-agent identities are configured without TLS, the proxy and the agent each log a startup warning that tokens are sent in cleartext.

# Proxy requiring a token
java -jar prometheus-proxy.jar --agent_token "$AGENT_TOKEN"

# Agent presenting the token
java -jar prometheus-agent.jar --config myconfig.conf --agent_token "$AGENT_TOKEN"

Per-Agent Identities and Path Authorization

A single shared agentToken authenticates agents but cannot tell them apart: any agent holding the token can register any path, including one already served by another agent — silently taking over its metrics. Nothing fails and nothing alerts; the dashboard keeps drawing a line, it is just another agent's numbers. The shared secret also makes revocation all-or-nothing (rotating it means reconfiguring every agent at once) and leaves the proxy unable to say who registered a path.

Define named identities under proxy.auth, each with its own token and a list of allowed path glob patterns, to scope what each agent may register. This closes the takeover hole and makes a shared proxy safe for multiple teams, none of which can step on another's paths:

proxy {
  auth = [
    { name = team-a, token = "team-a-token", paths = ["team_a_*"] }
    { name = team-b, token = "team-b-token", paths = ["team_b_*"] }
    { name = infra,  token = "infra-token",  paths = [] }          // empty paths = may register any path
  ]
}

Each agent presents its identity's token the usual way (--agent_token, AGENT_TOKEN, or agent.agentToken) — no agent-side change is needed. The proxy resolves the token to an identity and enforces, on every path registration, that the requested path matches one of the identity's patterns.

Patterns are single-segment globs: * matches any run of characters and ? matches exactly one (e.g. team_a_*). A path is checked against the patterns only once it has passed the path limits (proxy.internal.maxPathLength and the rest; see Proxy configuration), so a path that is too long or has more than one segment is refused as invalid, whatever the patterns. Because authorization is per-identity-per-path, consolidated mode still works as long as each participating agent's identity permits the shared path. A consolidated path belongs to the identity that registered it, though: another identity whose patterns match is refused while agents still serve the path, and joins only once they are gone.

A non-consolidated path is protected the same way. While a live agent serves it, only an agent of the same identity may take it over, which keeps redeploys working; an agent of another identity is refused with a reason, logged at WARN, until the serving agent is gone. With no agent authentication, or only the legacy shared token, every agent has the same identity, so takeovers behave as they always have. The legacy token and a proxy.auth identity cannot take over each other's live paths.

Migrating from a shared token

Setting proxy.auth does not disable a legacy proxy.agentToken. When both are present, the shared token is honored as an additional allow-all identity (the proxy logs a warning that it is active), so you can adopt per-agent identities incrementally:

1. Add a proxy.auth entry per agent while leaving proxy.agentToken in place — existing agents keep connecting with the shared token.
2. Move each agent onto its own identity token, one at a time, stopping the old agent before starting its replacement. While the old agent still serves a path, the proxy rejects the new identity's registration of it; the new agent retries a rejected static path, first after agent.internal.rejectedPathRetrySecs and then on a backoff of up to agent.internal.rejectedPathRetryMaxSecs, and registers it once the old agent is gone.
3. Once every agent presents an identity token, remove proxy.agentToken to close the shared allow-all path.

Step 3 is the one that actually secures anything: until the shared token is removed, anyone holding it still has allow-all access — the per-agent entries constrain only the agents that use them.

Isolation Between Agents

Authenticated agents share one proxy, so the proxy keeps each agent to its own traffic:

• Scrape results come only from the agent the scrape was sent to. Scrape IDs come from one counter shared by every agent, so the proxy checks each result, failure, and chunked-transfer message against the agent that owns the scrape. A message for another agent's scrape is dropped and logged at WARN, and the rest of the stream is still processed.
• Heartbeats are bound to their connection. A heartbeat naming another agent is refused and does not keep that agent from being evicted as stale.
• Behind a reverse proxy, calls are bound to an identity. With transportFilterDisabled there is no connection to tie a call to, so the proxy records the auth identity an agent connected with and refuses a call naming that agent under a different identity.

Credentials in URLs

A target URL can carry credentials, such as http://user:pass@host/metrics?api_key=.... The agent redacts the user info and query values everywhere a target URL is sent, logged, or displayed: in its logs, in what it sends the proxy, and so on the dashboard and /debug pages. It also redacts them inside HTTP client error messages, which embed the request URL. The proxy redacts target URLs from older agents as well. A config URL that fails to load (--config http://user:pass@host/agent.conf) is redacted in the error log and in the ConfigLoadException an embedding application catches.

The agent sends a URL's credentials as HTTP Basic auth with the first request to the target, and follows a redirect only within the target's own origin, so they reach only that host. An Authorization header that Prometheus sends with the scrape takes precedence; the URL's credentials then answer only a 401 challenge.

Scrape Port Access

A client of the scrape port can:

• Have the agents scrape their targets. A GET of a registered path sends a scrape to the agent serving it, or to every agent serving it in consolidated mode. The agent fetches the path's url from inside its own network and the proxy returns what the target answered.
• Choose the query string. The request's query parameters are appended to the path's url, after any it already has. The agent doesn't filter them, so every parameter reaches the target.
• Choose the Authorization header. The proxy forwards it to the agent, which sends it to the target ahead of any credentials in the path's url (see Credentials in URLs).
• List the paths. With service discovery enabled, its endpoint on this port lists every registered path, with the names and host names of the agents serving it.

For most exporters that is a read of metrics Prometheus already collects. An exporter that takes its target as a query parameter is different: blackbox_exporter's and snmp_exporter's /probe?target=…, for example, turn the query string into a choice of host, so a client of the scrape port can have the exporter probe any address it can reach, from inside the agent's network, and read the result. Don't rely on the target in the path's url to prevent that, since which of two values for a repeated parameter an exporter uses is up to the exporter.

To keep the port to Prometheus:

• Bind it to a private address with proxy.http.host, one Prometheus can reach and the wider network can't (-Dproxy.http.host=10.0.0.5 sets it from the command line). In a container, which listens on all of its interfaces, publish the port on a private address instead (-p 10.0.0.5:8080:8080). The proxy's Helm chart makes its Service a ClusterIP, and its agentService exposes only the gRPC port, so in Kubernetes the scrape port stays inside the cluster unless you expose it; a NetworkPolicy keeps it from the cluster's other pods as well.
• Firewall it so that only Prometheus can connect.
• Take care with exporters that take a target parameter. Serve them only through a proxy whose scrape port nothing but Prometheus can reach, and limit what the exporter can reach, with its own settings or the network around it, to the hosts it should probe.

Scrape Port Responses

The proxy passes a target's Content-Type through only for the Prometheus exposition formats (text/plain, application/openmetrics-text, and the protobuf format); anything else is served as text/plain. Every scrape-port response also carries X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox. A compromised target therefore can't serve a page that runs as script on the proxy's origin in an operator's browser.

Auth Header Forwarding

When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards the Authorization header to the agent over the gRPC channel. The agent then includes this header when fetching metrics from the target endpoint.

scrape_configs:
  # Bearer token authentication
  - job_name: 'app1'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJhbGciOiJIUzI1NiIs...'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # Basic auth
  - job_name: 'app2'
    metrics_path: '/app2_metrics'
    basic_auth:
      username: 'user'
      password: 's3cr3t'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # No auth
  - job_name: 'app3'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

When Prometheus scrape configs include basic_auth or bearer_token,
the proxy forwards the Authorization header to the agent over gRPC.

Without TLS, credentials are transmitted in plaintext!

Protect forwarded credentials with TLS:
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key

java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt

Redirects from Scrape Targets

The agent follows a redirect only to the target's own origin — the same scheme, host, and port. A redirect to any other origin is not followed: the scrape reports the 3xx status and the agent logs a warning. This keeps a target, or an open redirect reached through forwarded query parameters, from sending the agent to another host that would receive the target's basic-auth credentials, or to an internal address whose response would come back through the proxy. Point the path's url at the final location if a target redirects elsewhere.

Scraping HTTPS Endpoints

For HTTPS scrape targets signed by a custom or private CA (e.g. an internal corporate CA), point the agent at a trust store containing that CA so certificates are still validated:

Trust a custom or private CA for HTTPS scrape targets (validation stays enabled):

Via CLI:
java -jar prometheus-agent.jar \
  --https_truststore /etc/agent/truststore.jks \
  --https_truststore_password changeit \
  --config myconfig.conf

Via environment variables:
HTTPS_TRUST_STORE_PATH=/etc/agent/truststore.jks
HTTPS_TRUST_STORE_PASSWORD=changeit

Via config file:
agent.http.trustStorePath = "/etc/agent/truststore.jks"
agent.http.trustStorePassword = "changeit"

An empty path uses the JDK default trust store. The trust store is process-wide — it applies to every HTTPS target the agent scrapes, and it is ignored when trust_all_x509 is enabled.

As a last resort, you can disable SSL verification entirely:

Disable SSL verification for HTTPS endpoints (development only!):

Via CLI:
java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf

Via environment variable:
TRUST_ALL_X509_CERTIFICATES=true

Via config file:
agent.http.enableTrustAllX509Certificates = true

Reporting a Vulnerability

Report a security problem privately, not in a public issue, pull request, or discussion: use GitHub's private vulnerability reporting. The report stays visible only to you and the maintainer until an advisory is published with the fix. Security fixes go into the latest release only, so upgrade to receive them.

SECURITY.md lists what a useful report includes, and which documented behavior is by design rather than a vulnerability: the agent port accepting any agent without a token, identities, or mutual TLS; the scrape port having no authentication, so that whoever can reach it can trigger scrapes with their own query string and Authorization header; and the admin, metrics, and dashboard ports having no authentication.