AI Gateway Architecture

Overview of AI Gateway architecture, processing flow, and deployment components on AWS EKS.

AI Gateway is deployed as an umbrella Helm chart that bundles the gateway and its chart-managed runtime components into a single release. Sensitive Data Guardrails and Semantic Guardrails are enabled by default as chart sub-charts. The gateway processes LLM requests by scanning outbound prompts before they reach the LLM and inbound responses before they are returned to the client.

AI Gateway uses Kubernetes for deployment at scale. This deployment guide covers AWS EKS.

Communication Ports

The following table lists the ports used for communication between the components.

Components

The umbrella chart deploys the following components into the Helm release namespace. The standard namespace is federated-guardrails.

• AI Gateway (federated-guardrails): Forms the core AI security gateway. Receives supported LLM API requests, runs configured outbound scanners (including PII tokenization and Semantic Guardrails checks), forwards policy-compliant requests to the upstream LLM, scans the inbound response, and returns the processed result to the client. Helm renders the scanner and routing configuration from policyConfig.policyYaml into the gateway ConfigMap. The gateway retrieves Cedar policy from Policy.

• Sensitive Data Guardrail (sensitive-data-guardrails): Performs reversible PII tokenization on outbound prompts and restores original values from LLM responses using the Vault-Based Tokenizer. Depends on the Classification Service for PII detection.

• Vault-Based Tokenization (vault-based-tokenization): Stores PII tokens in a Valkey cluster. Tokens are keyed by session and expire automatically via TTL.

• Valkey Cluster: High-availability key-value store for token persistence. Deployed as a sharded cluster (3 primaries by default).

• Semantic Guardrails: Provides PII and topic-level policy enforcement. It is deployed by the chart into the semantic-guardrails-ns namespace and accessed by the gateway.

• Classification Service (Data Discovery): Identifies PII spans in text. Deployed into the data-discovery namespace and accessed by both Sensitive Data Guardrails and Semantic Guardrails.

• PPC API Gateway: Provides the Envoy-based API gateway included in the Protegrity Provisioned Cluster (PPC). It terminates TLS and routes client requests to the AI Gateway API prefix, /pty/federated-guardrails/v1.

Policy Configuration

Helm renders the scanner and routing configuration into the gateway ConfigMap. The gateway retrieves the authoritative Cedar policy from Policy.

The gateway retrieves the authoritative Cedar policy from Policy during startup and, when configured, refreshes it on a background interval. The chart sets FG_POLICY_POLL_INTERVAL_S to 60 seconds.

• FG_POLICY_SYNC_URL: Policy endpoint exposed by the current Helm chart.
• FG_POLICY_POLL_INTERVAL_S: Background synchronization interval in seconds.