Protegrity AI Gateway on

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Organizations are rapidly adopting large language models (LLMs) to power applications, automate workflows, and extract insights from unstructured data. Sending sensitive prompts to external LLMs or exposing raw LLM responses to end users introduces data privacy, compliance, and security risks.

AI Gateway addresses these risks by acting as an AI security gateway between client applications and supported LLM APIs. Every outbound prompt and inbound response passes through configurable scanners before reaching its destination. Scanners can detect and block PII leakage, identify prompt injection attacks, enforce topic restrictions, and apply graduated responses from logging to redaction to blocking based on the Cedar policy managed by Policy.

AI Gateway Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview of AI Gateway architecture, processing flow, and deployment components on AWS EKS.

AI Gateway is deployed as an umbrella Helm chart that bundles the gateway and its chart-managed runtime components into a single release. Sensitive Data Guardrails and Semantic Guardrails are enabled by default as chart sub-charts. The gateway processes LLM requests by scanning outbound prompts before they reach the LLM and inbound responses before they are returned to the client.

Gateway Configuration

Mon, 01 Jan 0001 00:00:00 +0000

AI Gateway uses a YAML policy to configure the upstream LLM, scanners, profiles, and request routing. In a Helm deployment, manage this policy through the chart values and gateway ConfigMap. Cedar authorization policy is retrieved from Policy.

The examples below are complete minimal policies. They use a local regex scanner so they do not require a remote scanner service. Choose one upstream configuration for the deployed policy.

OpenAI-Compatible Upstream

Set FG_UPSTREAM_API_KEY in the gateway environment, then use this policy as the YAML policy value:

Uninstallation

Mon, 01 Jan 0001 00:00:00 +0000

1. Configure kubectl and uninstall the Helm release

Configure kubectl for the cluster where AI Gateway is installed:

aws eks update-kubeconfig \
 --region {{AWS_REGION}} \
 --name {{CLUSTER_NAME}}

az aks get-credentials \
 --resource-group "{{RESOURCE_GROUP}}" \
 --name "{{CLUSTER_NAME}}"

Verify that the current context points to the intended cluster:

kubectl config current-context
kubectl get nodes

Remove the AI Gateway Helm release and all bundled sub-charts from the cluster:

Troubleshoot

Mon, 01 Jan 0001 00:00:00 +0000

This section helps you diagnose and resolve common issues with an AI Gateway deployment. Use the following guidance to inspect pods, services, policy configuration, and scanner connectivity.

Common Issues

The following sections describe frequently encountered problems and how to resolve them.

One or more pods not reaching the Running state

Inspect the affected pod’s events and logs to identify why it failed to start, such as scheduling or image pull errors.