A Better Way?
The ingress controller is a critical part of any Kubernetes cluster. It’s probably the first thing you installed in your first cluster, but it’s also the first thing a lot of us forget about, as it hums away behind the scenes. You probably installed one of the common, basic controllers, like ingress-nginx, and configure it with Ingress resources. While this combination can do the basics, the state of the art has moved on a long way. If you’ve ever noticed the number of magic annotations you have to use on your Ingress resources to achieve pretty basic functionality, or had in-flight connections drop when you change config, you’ve probably wondered if there’s a better way.
Envoy Gateway and the New Gateway API
Envoy is a proxy designed from the ground-up to be cloud-native, hot-reloadable, observable and easily operable. It’s battle-tested all over the internet, including in some parts of Google’s edge. If this seems like the ideal proxy for a modern ingress controller, it is! And that project, Envoy Gateway, is now easier to migrate to than ever. In this post I’ll take you through how to migrate to Envoy Gateway from ingress-nginx—and, if you’re using another older ingress controller like HAProxy or Ambassador, we’ll be covering those soon on the Tetrate blog.
One of the major differences between Envoy Gateway (EG) and older ingress controllers is that it’s not configured by Ingress resources, but by the new Gateway API, which has several resources like Gateway and HTTPRoute. These newer, richer resources allow first-class configuration (i.e., using fields not annotations) of many more features, which modern ingress controllers like EG offer. With support for—and easy configurability of—things like global rate-limiting, OIDC authorization, WAF functionality, and more, a lot of people won’t need their additional “API Gateway” product. A move to Envoy Gateway is a move to a fully open-source, community-led, CNCF project, and below I’ll show you how.
Tetrate offers an enterprise-ready, 100% upstream distribution of Envoy Gateway, Tetrate Enterprise Gateway for Envoy (TEG). TEG is the easiest way to get started with Envoy Gateway for production use cases. Get access now ›
Migration from Ingress to Gateway
For the purposes of this blog, I’m using a simple cluster with the following ingress-nginx and example app running, configured to route traffic to the httpbin app:
NAMESPACE NAME READY STATUS RESTARTS AGE
default httpbin-648f469544-ns8w2 1/1 Running 0 62s
ingress-nginx ingress-nginx-controller-cf668668c-vjmcv 0/1 Running 0 19sThe first thing we need before we can use Envoy Gateway is config for our routes and other features, in the new Gateway API format. Thankfully this is easy, because there’s a new tool, ingress2gateway, which will read the existing config in our cluster and produce the new resource kinds.
ingress2gateway is still under active development, so there aren’t packages for all systems yet, and besides, it’s gaining features all the time. So, let’s install it from source. Because ingress2gateway is written in Go, this is pretty straight forwards:
go install github.com/kubernetes-sigs/ingress2gateway@latestNow we can run it as follows:
ingress2gateway print --providers ingress-nginxThe --providers option here is telling ingress2gateway to read in Ingress resources from the cluster. Specifically, Ingress resources used with ingress-nginx, i.e., having the ingress-nginx-specific annotations (because Ingress is so limited, each ingress controller that uses it has their own set of annotations for accessing additional functionality, meaning there’s effectively several variants of the Ingress resource).
If your Ingress resources aren’t in the namespace default, you can add a --namespace option.
This command will convert the Ingress resources it finds to the new Gateway API resources (HTTPRoute, etc.) and print them (they won’t be automatically applied to the cluster).
We must install EG before we can apply that config, because the new resource types like HTTPRoute aren’t built into core Kubernetes yet, and the EG installation process instals them too (their CRDs). Go ahead and install Envoy Gateway using one of the supported methods.
With EG installed, there’s one quick piece of low-level config we need to apply. This GatewayClass resource allows us to customise low-level things about the Envoy proxies that EG deploys for us, things like resource limits on the Pods. We’ll leave all that as default for the time being, but this resource still needs to exist. If you look closely at the Gateway resource that ingress2gateway printed, you can see it’s expecting there to be a GatewayClass called nginx – this allows settings to be applied to all the Envoy proxies used for ingress routes migrated from nginx, but not affect others. Apply the following file to your cluster:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: nginx
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controllerIf you do want to customize proxy-related settings, you can read the documentation for the GatewayClass resource.
Now we can apply our new configs, a quick way to do which would be:
ingress2gateway print --providers ingress-nginx | kubectl apply -f -Let’s test that to check it’s all working. We first need to find the IP of the load balancer for our new EG proxies. In the envoy-gateway-system namespace, there’ll be a Service of type: LoadBalancer the name of which will start with envoy-default-nginx-. Its ExternalIP is the one you want.
If you want to get this programmatically, you can use the following command:
kubectl get service -n envoy-gateway-system \
-l app.gateway.envoyproxy.io/name=envoy \
-l gateway.envoyproxy.io/owning-gateway-name=nginx \
-l gateway.envoyproxy.io/owning-gateway-namespace=default \
-o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}'Before you’re ready to move your DNS entries to point to this new load balancer, rather than ingress-nginx’s, we can use curl to connect to EG’s IP and still give it the host header it’s expecting:
curl --header 'Host: foo.example.com' 1.2.3.4You could now uninstall ingress-nginx if you wanted to, to save on resources, and not have the potential for any unintended routes in from outside the cluster. It won’t clash with EG if you leave it around, but some tools like ExternalDNS might get confused.
Next Steps
- Get started with Envoy Gateway ›
- Get access to Tetrate’s enterprise-ready distribution of Envoy Gateway ›
- Schedule a demo today ›
###
If you’re new to service mesh, Tetrate has a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
Are you using Kubernetes? Tetrate Enterprise Gateway for Envoy (TEG) is the easiest way to get started with Envoy Gateway for production use cases. Get the power of Envoy Proxy in an easy-to-consume package managed by the Kubernetes Gateway API. Learn more ›
Getting started with Istio? If you’re looking for the surest way to get to production with Istio, check out Tetrate Istio Subscription. Tetrate Istio Subscription has everything you need to run Istio and Envoy in highly regulated and mission-critical production environments. It includes Tetrate Istio Distro, a 100% upstream distribution of Istio and Envoy that is FIPS-verified and FedRAMP ready. For teams requiring open source Istio and Envoy without proprietary vendor dependencies, Tetrate offers the ONLY 100% upstream Istio enterprise support offering.
Need global visibility for Istio? TIS+ is a hosted Day 2 operations solution for Istio designed to simplify and enhance the workflows of platform and support teams. Key features include: a global service dashboard, multi-cluster visibility, service topology visualization, and workspace-based access control.
Get a Demo