SAN FRANCISCO, October 3, 2024 — Driven by the surge in applications built to utilize large language models (LLMs), the open source AI gateway space is heating up. In response, engineers from Bloomberg and Tetrate have partnered to develop an innovative, community-led set of core AI gateway features for enterprise AI integration. This effort will expand the capabilities of the CNCF’s Envoy Gateway project, one of the Kubernetes Gateway API implementations.
As the emerging open standard for handling Kubernetes ingress traffic, Envoy Gateway is designed for at-scale operation and is extensible, making it a solid choice to underpin this new set of features, as well as future innovation in the AI API gateway space. Furthermore, Envoy Gateway is a community-led open source project with no commercially licensed features, and where community members drive decisions about upstream feature development.
This sets it apart from both existing vendor-led open source AI gateway offerings and fully proprietary, commercial AI gateway solutions that have sought to address this problem. Both of these approaches can create greater complexity for some enterprises and hamper innovation, which is why the Envoy community is developing an option without vendor lock-in or features that can only be accessed through additional, paid enterprise licenses.
“Historically, when shared problems arise in the software industry, the open source community rallies to solve them, accelerating innovation,” said Varun Talwar, Founder of Tetrate. “Our collaboration with Bloomberg and the CNCF aims to achieve precisely that: designing and delivering a community-led, fully open source AI gateway, powered by the leading contender to replace legacy models for Kubernetes ingress. It’s a solution the market is asking for, and we’re excited to be part of the team of maintainers and contributors creating it.”
AI Gateways enable organizations to integrate AI functionality into workflows and applications. They route requests to multiple AI service providers and models through a single reverse proxy layer (often referred to as a gateway). AI Gateways simplify AI integration by providing a single unified API layer with which developers interact, and can provide additional functionality, such as rate limiting, caching and observability.
The initial idea for this project arose when Dan Sun, Engineering Team Lead for Bloomberg’s Cloud Native Compute Services – AI Inference team and co-founder/maintainer of the KServe project, came to the Envoy community and outlined his views of the problem space and a potential path forward for solving it. Tetrate, a major upstream contributor to the Envoy project, stepped forward to express interest in helping Sun and Bloomberg turn their vision for the Envoy AI Gateway API into reality.
“Bloomberg has more than 15 years of experience delivering value to our customers by incorporating artificial intelligence (AI) – in particular, machine learning and natural language processing – in enterprise applications,” said Steven Bower, Engineering Lead for Bloomberg’s Cloud Native Compute Services group. “When we looked to the community for someone to collaborate with to start building gateway features that accelerate AI integration in our products, we immediately identified the engineering team at Tetrate. Their people are leading contributors to Envoy Gateway, and they bring significant expertise to the project around handling cloud-native, scalable traffic. Beyond that, as an ‘open source first’ company, Bloomberg believes in the power and collaborative nature of the open source community to develop web scale solutions, and that important difference makes this project a valuable alternative to other ongoing efforts.”
Envoy Gateway and KServe can be used together to allow traffic routing to both self-hosted and vendor-hosted LLMs. In this case, the AI gateway sits on the top and routes open source LLM model traffic to self-hosted endpoints using KServe, and vendor-hosted model traffic is routed to AWS Bedrock or other, similar cloud-based services.
The first features to be included in Envoy AI Gateway will provide:
- application traffic management to LLM providers with high-availability routing strategies;
- LLM usage monitoring and control at the application, organization, and enterprise levels, to help users manage costs; and
- a unified interface for LLM requests through which the gateway handles back-end connectivity to various LLM providers.
The open source Envoy Gateway extensions and enhancements will offer usage control for applications that are integrated with multiple LLM providers and models, robust authorization mechanisms, and intelligent fallback options to ensure continued operation even when cloud providers are unavailable or too expensive.
This open source initiative, part of the Cloud Native Computing Foundation (CNCF), is more than just a tool; it’s a strategic response to challenges enterprises face in adopting and integrating AI in their applications at scale. By laying the groundwork for scalable AI platforms, Tetrate and Bloomberg engineers are addressing the immediate needs of today’s enterprises and setting the stage for the future of AI applications within cloud-native environments.
“The Envoy project continues to impress with its flexibility to support new and valuable use cases,” said Chris Aniszczyk, CTO of the CNCF. “Bloomberg and Tetrate have done exactly what our community is designed to do: bring people and organizations together to solve a common problem. That they’re doing it with Envoy Gateway only validates the power and extensibility of the project.”
Learn More About the Envoy AI Gateway Project
Join the “Enabling AI Adoption at Scale – The AI Platform with Envoy AI Gateway” webinar, an online panel discussion of Bloomberg and Tetrate engineers hosted by the CNCF on Thursday, October 17, 2024 at 1 PM EDT. Erica Hughberg of Tetrate will be joined by Dan Sun and Yuzhui Liu of Bloomberg and other contributors from Tetrate and the CNCF community for a conversation about the project and ways for others to get involved and contribute.
The session will include discussions about:
- Industry trends from traditional AI to generative AI
- What to think about when enabling the use of AI in an enterprise organization
- The role of the AI Platform – what it solves and what it doesn’t
- Why build an AI Gateway using Envoy and Envoy Gateway?
- How do I get started building my organization’s AI Platform?
Register for the online panel here.
In addition, attendees at KubeCon + CloudNativeCon North America 2024 (November 12-15) in Salt Lake City, UT, will have the opportunity to learn more about the project and how to get involved in it. A demo of the Envoy AI Gateway will also be available in the Tetrate booth (Booth Q2).
About Bloomberg
Bloomberg is a global leader in business and financial information, delivering trusted data, news, and insights that bring transparency, efficiency, and fairness to markets. The company helps connect influential communities across the global financial ecosystem via reliable technology solutions that enable our customers to make more informed decisions and foster better collaboration. For more information, visit Bloomberg.com/company or request a demo.
About Tetrate
Rooted in open source, Tetrate was founded to solve the application networking and security challenges created by modern computing so enterprises can innovate with speed and safety in hybrid and multi-cloud environments. As applications evolve into collections of decentralized microservices, monitoring and managing the network communications and security among those myriad services becomes challenging. This is why some of the largest financial institutions, governments and other enterprises rely on Tetrate to deliver modern application networking and security. Find out more at https://tetrate.io.
About the CNCF
Cloud native computing empowers organizations to build and run scalable applications with an open source software stack in public, private, and hybrid clouds. The Cloud Native Computing Foundation (CNCF) hosts critical components of the global technology infrastructure, including Kubernetes, Prometheus, and Envoy. CNCF brings together the industry’s top developers, end users, and vendors and runs the world’s largest open source developer conferences. Supported by more than 800 members, including the world’s largest cloud computing and software companies, as well as over 200 innovative startups, CNCF is part of the nonprofit Linux Foundation. For more information, please visit www.cncf.io.
The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page. Linux is a registered trademark of Linus Torvalds.
###
MEDIA CONTACTS
Cristin Connelly
Cathey.co for Tetrate
Chaim Haas
Bloomberg
212-617-7025
Media kit
Founder photos
Download ZIPAbout tetrate
Download ZIPFor more inquiries, contact info@tetrate.io