Announcing Tetrate Agent Operations Director for GenAI Runtime Visibility and Governance

Learn more
< Back

The Beginning of a Scalable GenAI Future: The First Release of Envoy AI Gateway

Today marks an exciting milestone for enterprises eager to integrate Generative AI (GenAI) into their applications at scale. With the first release of

The%20Beginning%20of%20a%20Scalable%20GenAI%20Future%3A%20The%20First%20Release%20of%20Envoy%20AI%20Gateway

Today marks an exciting milestone for enterprises eager to integrate Generative AI (GenAI) into their applications at scale. With the first release of Envoy AI Gateway, Tetrate and Bloomberg are pioneering a standardized and scalable approach to AI service integration in the Envoy Open Source community.

This release is the opening chapter of the journey. At Tetrate, we envision it as a foundation for organizations to embrace GenAI while ensuring control, enhancing security, and optimizing cost efficiency.

The Challenge: AI at Scale Needs a New Approach

Every enterprise looking to integrate GenAI faces similar hurdles:

  • Managing multiple GenAI service providers
  • Controlling costs with granular rate limits
  • Ensuring security and compliance across GenAI integrations

Many early GenAI adopters have resorted to Python-based gateways. However the nature of Python and it’s global interpreter lock makes it difficult to handle many concurrent connections.

Envoy AI Gateway changes that with its foundation in Envoy Proxy, the modern cloud-native proxy designed for efficient concurrent request handling, and Envoy Gateway for scalable Kubernetes-based deployment.

A Unified Interface for LLM Service Integration

Built on Envoy Gateway, Envoy AI Gateway provides a unified layer for easily integrating multiple LLM providers.

The 0.1 release introduces:

  • Unified LLM API: A single API interface for integrating with AWS Bedrock and OpenAI, with more providers coming soon.
  • Upstream Authorization: Streamlined authentication management for different LLM services.
  • Usage-Based Rate Limiting: Fine-grained control over token-based usage is needed to manage LLM usage costs efficiently.

This release is not the finish line; it’s the first step toward a standardized, scalable enterprise AI integration layer.

Why This Matters: AI as an Enterprise Capability

At Tetrate, we are focused on helping enterprises manage complex, distributed architectures, from microservices to Kubernetes networking to service mesh security.

As organizations adopt GenAI for product features, automation, and knowledge retrieval, the need for enterprise-ready infrastructure is pressing. Envoy AI Gateway provides that foundation.

Bloomberg is already leveraging it to centralize GenAI integration, ensuring consistent access control, usage quotas, and provider-agnostic integration. This approach streamlines AI adoption across the organization while enabling governance at scale.

What’s Next? Expanding Enterprise AI Enablement

The 0.1 release is just the start for Envoy AI Gateway. The roadmap ahead includes:

  • Google Gemini 2.0 integration to expand multi-provider capabilities.
  • Provider and model fallback logic to ensure high availability.
  • Prompt templating and semantic caching to optimize AI cost and performance.

Enabling Enterprises Beyond Envoy AI Gateway

Tetrate is committed to building enterprise-ready solutions on Envoy, and AI is no exception. As this project evolves, we will continue building capabilities to help organizations maximize their AI investments.

Ready to Get Started?

We’d love to connect with your organization if it wants to enable GenAI adoption and needs a scalable, secure way to manage AI service integrations.

Contact us at Tetrate to explore how we can help you deploy and scale AI in production. Let’s shape the future of enterprise GenAI Integration together.

Website: Learn more about Envoy AI Gateway

Workshop: Join us for a hands-on workshop in London on 31st March at 2pm. Register now

Product background Product background for tablets
New to service mesh?

Get up to speed with free online courses at Tetrate Academy and quickly learn Istio and Envoy.

Learn more
Using Kubernetes?

Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed via the Kubernetes Gateway API.

Learn more
Getting started with Istio?

Tetrate Istio Subscription (TIS) is the most reliable path to production, providing a complete solution for running Istio and Envoy securely in mission-critical environments. It includes:

  • Tetrate Istio Distro – A 100% upstream distribution of Istio and Envoy.
  • Compliance-ready – FIPS-verified and FedRAMP-ready for high-security needs.
  • Enterprise-grade support – The ONLY enterprise support for 100% upstream Istio, ensuring no vendor lock-in.
  • Learn more
    Need global visibility for Istio?

    TIS+ is a hosted Day 2 operations solution for Istio designed to streamline workflows for platform and support teams. It offers:

  • A global service dashboard
  • Multi-cluster visibility
  • Service topology visualization
  • Workspace-based access control
  • Learn more
    Decorative CTA background pattern background background
    Tetrate logo in the CTA section Tetrate logo in the CTA section for mobile

    Ready to enhance your
    network

    with more
    intelligence?