Envoy Gateway, now at its 1.1 release, is the reference cloud-native load-balancer. It provides Layer 7 traffic routing and observability, i.e., it understands the HTTP protocol, and can direct traffic based on request paths.
A lot of us run on AWS. AWS also offers three primary load balancing options: the old and deprecated CLB – Classic Load Balancer (neé ELB); the Application Load Balancer – ALB; and the Network Load Balancer – NLB. Countless column-pixels have been dedicated to which you should choose, including articles from AWS themselves. The very short version is: ALBs work at Layer 7, duplicating many of Envoy Gateway’s features.
Tetrate offers an enterprise-ready, 100% upstream distribution of Envoy Gateway, Tetrate Enterprise Gateway for Envoy (TEG). TEG is the easiest way to get started with Envoy for production use cases. Get access now ›
You’ll always need some kind of AWS load-balancer to get traffic in from the internet (routed into a public subnet) to your worker nodes (hopefully situated in a private subnet). However, if you’re running Envoy Gateway, you don’t need the complexity, cost, and relatively high latency of an ALB; an NLB is optimal.
When you configure a Gateway resource, an AWS load-balancer is automatically created to expose your cluster’s ingress layer – Envoy Gateway or otherwise – to the outside world. I won’t go into the mechanisms powering this as it’s fairly complicated, but suffice to just understand that there’s a Kubernetes Service pointing at all the managed Envoy Proxy pods. The type: field of this Service is set to “LoadBalancer,” and a Kubernetes component called the cloud-controller spots that and calls out to AWS APIs to spin up an AWS load-balancer which sends traffic to all the Pods in that Kubernetes Service.
However, by default you’ll get a Classic Load Balancer – the worst! In this article I’ll take you through all the steps you need to use an NLB instead. The instructions I show are for EKS, but they should work on self-installed clusters with a few changes.
Setup
First let’s spin up an EKS cluster to use (if you want to use an existing cluster make sure you have the owners’ permission!). I’ll be using eksctl for this – if you don’t have it installed, instructions are on its site.
Verify eksctl is installed correctly:
$ eksctl version
0.187.0Note: Version 0.187.0 is the version I tested with. As of this writing, the default EKS version is Kubernetes 1.30, which again is the version these instructions are verified with.
Create a cluster:
$ eksctl create cluster --name envoy-gateway-demo –with-oidcNote: You’ll want to get a coffee while this happens.
Our cluster needs to use the AWS VPC CNI plugin – again a very technical detail, but for our purposes this determines the kind of network connections made between Pods in the cluster, and is necessary for the NLB to be able to connect to our managed Envoy Proxy pods. In an EKS cluster this is the default option, and the eksctl command above will have installed it; if you’re managing your own cluster you’ll need to use this CNI plugin, no e.g. Cilium.
Now our cluster is ready, we can install the various components into it. Annoyingly the order of the following steps is important, otherwise things go wrong, often in quite subtle ways, but if you follow these steps you’ll be fine!
First, the reason we’re here: Envoy Gateway 1.1:
$ helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.1.0 -n envoy-gateway-system --create-namespaceIf you have any problems installing EG, more detailed installation instructions are available.
Configure GatewayClass
Next we need to configure a GatewayClass resource so that we can associate our EG config with the right instance – it’s possible to have multiple copies of EG running, or EG plus a different ingress controller.
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
namespace: envoy-gateway-system
name: custom-proxy-configThis resource contains a reference to an EnvoyProxy-type resource called custom-proxy-config, which will alter the Kubernetes configuration of all the actual Envoy instances that EG deploys to handle our traffic. So next, let’s deploy that:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facingThis resource is a little more complicated, but essentially it’s giving a set of annotations to apply to the Kubernetes Service object which gets traffic to the Envoy proxies deployed by EG. This isn’t really any different from a Service you’d deploy to get traffic to your own apps, except remember this one will have its type field set to LoadBalancer.
Using NLB Instead of CLB
Most distributions of Kubernetes ship with some code that interacts with the cloud provider the cluster is running on. This is how it’s able to do things like create storage buckets for PersistentVolumeClaims, and how it’s able to make cloud load-balancers for Services of type LoadBalancer. However the code that’s built into Kubernetes to do this is a little old and only makes the CLBs I mentioned before. Next, we’ll deploy a newer replacement for that code, written by AWS, which will see our Service object and make an NLB. The aws-load-balancer-type: external annotation on the Service tells the built-in code to ignore it and not make a CLB.
This installation is a little complicated, as we’ll need to create a couple of AWS IAM resources. Luckily the policy documents already exist, and we can use eksctl to deploy them and wire them up to service accounts etc.
# This only needs to be run once per AWS account. On subsequent deployments you’ll get an already exists error.
$ aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://<(curl -s https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.1/docs/install/iam_policy.json)
$ export AWS_ACCOUNT_ID=<your account ID>
$ eksctl create iamserviceaccount \
--cluster=envoy-gateway-demo \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--role-name AmazonEKSLoadBalancerControllerRole \
--attach-policy-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy \
--approve
$ helm repo add eks https://aws.github.io/eks-charts
$ helm repo update eks
$ helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=envoy-gateway-demo \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controllerYou should now be able to see two Pods for the AWS load-balancer controller in the kube-system namespace. As of this writing, the helm chart we used instals version v2.8.1.
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-load-balancer-controller-5ffb88ccf8-4dx2f 1/1 Running 0 31s
aws-load-balancer-controller-5ffb88ccf8-kqpb9 1/1 Running 0 31s
…Configure Envoy Gateway
We can now configure EG as normal – I’ll give a quick overview here but more detailed guides are available. First, a Gateway resource, representing the actual Envoys that will be deployed:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: apps
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80And an HTTPRoute, sending requests for www.example.com/httpbin to an httpbin Pod.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: httpbin
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: apps
hostnames:
- "www.example.com"
rules:
- matches:
- path:
type: PathPrefix
value: /httpbin/
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /
backendRefs:
- group: ""
kind: Service
name: httpbin
port: 8000
weight: 1Finally, we deploy a copy of httpbin itself, in the default namespace where the HTTPRoute above is expecting it:
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yamlReview Load Balancers
Ever since we deployed the Gateway resource, the Envoy Gateway control plane will have started spinning up Envoy instances to actually proxy our requests, as well as deploying peripheral resources like the Service type=LoadBalancer. Moments after that was created, the NLB creation should have started, and with luck, will now be complete.
If your AWS CLI is configured for the correct account and region, you can see the NLB with:
$ aws elbv2 describe-load-balancersAlternatively, it’s visible in the AWS web console.
Nothing should be too surprising, but take a note of the DNS name that it got, as that’s what we’ll need to connect to in order to test it. If you want to do that programmatically, you can use this rather long command:
$ export NLB_DNS_NAME=$(kubectl get service -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=default -l gateway.envoyproxy.io/owning-gateway-name=apps -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')We can use curl to connect to the NLB, which in turn will forward all traffic to EG inside the EKS cluster. EG will then parse the HTTP protocol being spoken, and direct us to a Pod depending on the HTTP host and path used. Recall that our Gateway and HTTPRoute are expecting a hostname of www.example.com, but because we need to physically connect to the NLB, the curl command looks like this:
$ /usr/bin/curl --header "Host: www.example.com" ${NLB_DNS_NAME}/httpbin/headers
{
"headers": {
"Accept": "*/*",
"Host": "www.example.com",
"User-Agent": "curl/8.1.2",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000",
"X-Envoy-External-Address": "143.58.214.48",
"X-Envoy-Original-Path": "/httpbin/headers"
}
}
Hopefully this has all worked, and given you everything you need to use the latest greatest NLBs with your shiny new EG 0.5 deployment. From here you might want to read around all the config we’ve set, or make this into a more production-ready setup, e.g. by adding a CDN in front of it, adding TLS, or injecting the Coraza WAF into the EG proxies.
Cleanup
If you now wish to tear everything down, you can run:
$ kubectl delete gateway apps
$ eksctl delete cluster --name envoy-gateway-demoThe first command is important, to remove the NLB, Security Group, etc that EG dynamically creates. Having run that, the eksctl delete command should complete successfully, however this is always a little hit-and-miss, and manual cleanup may be necessary.
Further Resources
###
If you’re new to service mesh and Kubernetes security, we have a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.
If you’re looking for a fast way to get to production with Istio, check out Tetrate Istio Distribution (TID). TID is Tetrate’s hardened, fully upstream Istio distribution, with FIPS-verified builds and support available. It’s a great way to get started with Istio knowing you have a trusted distribution to begin with, have an expert team supporting you, and also have the option to get to FIPS compliance quickly if you need to.
Once you have Istio up and running, you will probably need simpler ways to manage and secure your services beyond what’s available in Istio, that’s where Tetrate Service Bridge comes in. You can learn more about how Tetrate Service Bridge makes service mesh more secure, manageable, and resilient here, or contact us for a quick demo.