F5 gateway works to protect and manage AI applications

A new gateway from F5 is aimed at helping enterprises to better manage AI security and delivery.

The F5 AI Gateway offers an automated way for customers to secure and manage interactions among AI applications, APIs, and large language models (LLMs). It’s a containerized Kubernetes service that can be deployed on its own or integrated with existing F5 software, hardware or services, the company stated. The gateway supports popular AI models such as OpenAI, Anthropic, Ollama as well as generic HTTP upstream LLMs and small language model (SLM) services.

AI applications present unique challenges with respect to security, monitoring, and delivery, according to Lori MacVittie, a distinguished engineer in F5’s CTO office.

“A significant challenge is due to the non-deterministic nature of AI applications. That is, inbound input and outbound responses vary greatly and evolve over time. AI applications can also pull in data from both structured and unstructured sources that can span multiple locations,” MacVittie wrote in a blog about the gateway.

“The trouble with unstructured, unpredictable input and output is the difficulty inherent in determining whether a request or response contains sensitive or incorrect information. While API security and web application firewalls are often used for exactly this purpose, they operate on deterministic content. That is, the format and content of requests and responses are understood and therefore it is easier to detect malicious or sensitive content. With AI, though it leverages the same constructs as APIs and web applications, the content is highly variable, which makes it difficult to craft policies that scan or scrub requests and responses,” MacVittie wrote.

The F5 AI Gateway offers a number of features for handling AI environments, including the ability to offload duplicate tasks from LLMs with semantic caching, according to MacVittie.

It also provides traffic routing and rate limiting for local and third-party large language models (LLM) to maintain service availability and performance and control costs. F5 stated. Semantic caching drives faster response time and reduces operational costs by removing duplicate tasks from LLMs, according to the vendor.

The AI Gateway can inspect, identify, and block inbound attacks such as prompt injection, insecure output handling, model denial-of-service, sensitive information disclosure, and model theft. “For outbound responses, AI Gateway identifies and scrubs PII data and prevents hallucinations. Software development kits (SDKs) enable additional enforcement of operational rules and compliance requirements for both prompts and responses to further align to operational needs,” F5 stated.

“Additional capabilities such as reporting of a wide array of metrics via OpenTelemetry, careful attention to audit log requirements, semantic caching, rate-limiting, and content-based model routing ensure support for all three AI delivery and security requirements: observe, protect, and accelerate,” MacVittie wrote.

The AI Gateway can be integrated with F5’s NGINX application security suite and BIG-IP application delivery platforms offering customers legacy integration and access.

Read more about F5