Security

Security Model

Prometheus Proxy is designed to be firewall-friendly:

• The agent initiates an outbound gRPC connection to the proxy
• No inbound ports need to be opened on the firewall
• Agent connections can be authenticated with an optional pre-shared token — optionally scoped to per-agent identities — and/or mutual TLS
• Stale agent connections are automatically cleaned up

TLS Encryption

Agents connect to the proxy using gRPC, which supports TLS with or without mutual authentication.

See TLS Setup for detailed configuration instructions.

Agent Authentication (Pre-Shared Token)

By default the proxy accepts agent gRPC connections with no application-level authentication. Set a shared pre-shared token so the proxy rejects agents that do not present it:

Both sides must use the same value. When set, the agent attaches the token as a gRPC metadata header on every call and the proxy rejects any call with a missing or mismatched token (UNAUTHENTICATED). When the token is empty (the default), the open behavior is preserved and the proxy logs a startup warning — unless mutual TLS is configured, which already authenticates agents. The token is never logged.

# Proxy requiring a token
java -jar prometheus-proxy.jar --agent_token "$AGENT_TOKEN"

# Agent presenting the token
java -jar prometheus-agent.jar --config myconfig.conf --agent_token "$AGENT_TOKEN"

Per-Agent Identities and Path Authorization

A single shared agentToken authenticates agents but cannot tell them apart: any agent holding the token can register any path, including one already served by another agent. Define named identities under proxy.auth, each with its own token and a list of allowed path glob patterns, to scope what each agent may register:

proxy {
  auth = [
    { name = team-a, token = "team-a-token", paths = ["team_a_*"] }
    { name = team-b, token = "team-b-token", paths = ["team_b_*"] }
    { name = infra,  token = "infra-token",  paths = [] }          // empty paths = may register any path
  ]
}

Each agent presents its identity's token the usual way (--agent_token, AGENT_TOKEN, or agent.agentToken) — no agent-side change is needed. The proxy resolves the token to an identity and enforces, on every path registration, that the requested path matches one of the identity's patterns.

Patterns are single-segment globs: * matches any run of characters and ? matches exactly one (e.g. team_a_*). Because authorization is per-identity-per-path, consolidated mode still works as long as each participating agent's identity permits the shared path.

Migrating from a shared token

Setting proxy.auth does not disable a legacy proxy.agentToken. When both are present, the shared token is honored as an additional allow-all identity (the proxy logs a warning that it is active), so you can adopt per-agent identities incrementally:

1. Add a proxy.auth entry per agent while leaving proxy.agentToken in place — existing agents keep connecting with the shared token.
2. Move each agent onto its own identity token, one at a time.
3. Once every agent presents an identity token, remove proxy.agentToken to close the shared allow-all path.

Auth Header Forwarding

When Prometheus scrape configurations include basic_auth or bearer_token, the proxy forwards the Authorization header to the agent over the gRPC channel. The agent then includes this header when fetching metrics from the target endpoint.

scrape_configs:
  # Bearer token authentication
  - job_name: 'app1'
    metrics_path: '/app1_metrics'
    bearer_token: 'eyJhbGciOiJIUzI1NiIs...'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # Basic auth
  - job_name: 'app2'
    metrics_path: '/app2_metrics'
    basic_auth:
      username: 'user'
      password: 's3cr3t'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

  # No auth
  - job_name: 'app3'
    metrics_path: '/app3_metrics'
    static_configs:
      - targets: ['proxy-host.example.com:8080']

When Prometheus scrape configs include basic_auth or bearer_token,
the proxy forwards the Authorization header to the agent over gRPC.

Without TLS, credentials are transmitted in plaintext!

Protect forwarded credentials with TLS:
java -jar prometheus-proxy.jar \
  --cert /path/to/server.crt \
  --key /path/to/server.key

java -jar prometheus-agent.jar \
  --config myconfig.conf \
  --trust /path/to/ca.crt

Scraping HTTPS Endpoints

For HTTPS scrape targets signed by a custom or private CA (e.g. an internal corporate CA), point the agent at a trust store containing that CA so certificates are still validated:

Trust a custom or private CA for HTTPS scrape targets (validation stays enabled):

Via CLI:
java -jar prometheus-agent.jar \
  --https_truststore /etc/agent/truststore.jks \
  --https_truststore_password changeit \
  --config myconfig.conf

Via environment variables:
HTTPS_TRUST_STORE_PATH=/etc/agent/truststore.jks
HTTPS_TRUST_STORE_PASSWORD=changeit

Via config file:
agent.http.trustStorePath = "/etc/agent/truststore.jks"
agent.http.trustStorePassword = "changeit"

An empty path uses the JDK default trust store. The trust store is process-wide — it applies to every HTTPS target the agent scrapes, and it is ignored when trust_all_x509 is enabled.

As a last resort, you can disable SSL verification entirely:

Disable SSL verification for HTTPS endpoints (development only!):

Via CLI:
java -jar prometheus-agent.jar --trust_all_x509 --config myconfig.conf

Via environment variable:
TRUST_ALL_X509_CERTIFICATES=true

Via config file:
agent.http.enableTrustAllX509Certificates = true