Architecture

Components

Proxy

The Proxy runs outside the firewall alongside the Prometheus server. It provides:

• HTTP Service (default port 8080) -- serves metrics endpoints that Prometheus scrapes
• gRPC Service (default port 50051) -- accepts persistent connections from agents
• Service Discovery -- optional HTTP SD endpoint for Prometheus auto-discovery
• Admin/Metrics -- optional health checks and operational metrics

Agent

The Agent runs inside the firewall with the monitored services. It provides:

• gRPC Client -- establishes and maintains an outbound connection to the proxy
• HTTP Client -- scrapes actual metrics endpoints on behalf of the proxy
• Path Registration -- registers configured endpoints with the proxy
• Heartbeat -- sends periodic keep-alive messages to maintain the connection

Request Flow

sequenceDiagram
    participant P as Prometheus
    participant Proxy as Proxy :8080
    participant gRPC as Proxy gRPC :50051
    participant Agent as Agent
    participant App as App :9100/metrics

    Note over Agent, gRPC: Agent connects on startup (outbound)
    Agent->>gRPC: connectAgent()
    Agent->>gRPC: registerAgent()
    Agent->>gRPC: registerPath("/app_metrics")

    Note over P, Proxy: Prometheus scrapes as normal
    P->>Proxy: GET /app_metrics
    Proxy->>gRPC: Lookup agent for path
    gRPC->>Agent: ScrapeRequest (via stream)
    Agent->>App: GET /metrics
    App-->>Agent: Metrics response
    Agent-->>gRPC: ScrapeResponse (via stream)
    gRPC-->>Proxy: Route response
    Proxy-->>P: Return metrics

gRPC Protocol

The communication between Proxy and Agent uses a bidirectional gRPC protocol defined in proxy_service.proto. Key RPCs:

Chunking

Large metric responses are automatically split into chunks to stay within gRPC message size limits.

• The agent checks if a response exceeds chunkContentSizeKbs (default: 32 KB)
• If so, the response is split into ChunkedScrapeResponse messages containing:
  • A header with metadata (status, content type, URL)
  • One or more chunks with payload bytes and CRC32 checksums
  • A summary with total byte count and final checksum for validation
• The proxy reassembles chunks and validates integrity before responding to Prometheus

Configure the chunk threshold:

agent.chunkContentSizeKbs = 64   // Larger chunks for big payloads

Compression

Responses exceeding minGzipSizeBytes (default: 512 bytes) are gzip-compressed before transmission over gRPC, reducing bandwidth between agent and proxy.

agent.minGzipSizeBytes = 256     // Compress more aggressively

Metric Filtering

Chunking and compression both manage the size of a payload; neither reduces the content itself. An optional per-path filter drops whole metric families at the agent, so unwanted series never cross the network boundary at all — cutting bandwidth and downstream Prometheus cardinality at the source.

Filtering runs between the scrape and the gzip decision, so the three mechanisms compose in pipeline order:

scrape target -> filter -> gzip (minGzipSizeBytes) -> chunk (chunkContentSizeKbs) -> gRPC

Because the filter runs first, gzip and chunking both see the already-reduced payload. The maxContentLengthMBytes guard is the deliberate exception: it applies to the raw response, so filtering can never be a way around that limit.

Filtering is off by default and configured per path. See Metric Filtering for the rules and configuration.

Heartbeat

The agent sends periodic heartbeat messages to maintain the gRPC connection during periods of inactivity. This prevents load balancers and firewalls from closing idle connections.

agent.internal {
  heartbeatEnabled = true
  heartbeatCheckPauseMillis = 500
  heartbeatMaxInactivitySecs = 5   // Send heartbeat after 5s of inactivity
}

Connection Lifecycle

1. Startup -- Agent reads configuration and initializes services
2. Connection -- Agent establishes outbound gRPC connection to proxy
3. Registration -- Agent registers its identity and configured paths
4. Operation -- Agent processes scrape requests and sends heartbeats
5. Reconnection -- On disconnect, agent automatically reconnects with rate limiting
6. Shutdown -- Graceful cleanup of connections and resources

Stale Agent Cleanup

The proxy runs a background service that evicts agents that have been inactive for too long:

proxy.internal {
  staleAgentCheckEnabled = true
  maxAgentInactivitySecs = 60       // Evict after 60s of inactivity
  staleAgentCheckPauseSecs = 10     // Check every 10s
}

Consolidated Mode

By default, each path is owned by a single agent. In consolidated mode, multiple agents can register the same path for redundancy. When a scrape request arrives for a consolidated path, the proxy routes the request to one of the available agents.

agent.consolidated = true

See Advanced Topics for more details.