Introducing GetEnvoy Extension Toolkit for WebAssembly-Based Envoy Extensions

Tetrate, the enterprise service mesh company, is introducing a new feature of its open source GetEnvoy project that makes it easier for developers to extend and customize the Envoy proxy.

A quick flashback

A year ago here at Tetrate we started the GetEnvoy project with a goal to facilitate adoption of Envoy— the edge and service proxy used as a service mesh data plane– by making it trivial to get it up and running.

To give you a better context: As exciting and powerful Envoy is, it is still a low-level networking component that is strongly focused on operation at scale, but it has been lacking in terms of user experience.

A user journey typically starts with the question, “How can I install this tool in my environment?” Envoy, historically, hasn’t had an answer for that.

To fill this gap, we started the GetEnvoy project and introduced the getenvoy CLI as its user-facing component.

A new challenge

The next common demand when it comes to Envoy is, “How can I extend it?”

Until now, if you wanted to extend or customize Envoy, you had to “cross the line” and, effectively, become an Envoy developer.

Luckily, this situation is about to change. Support for a new technology–  WebAssembly (Wasm)— is coming into Envoy. Wasm makes it possible to develop extensions in various programming languages and, even more importantly, be able to deploy them in a fully dynamic manner.

Meet the GetEnvoy Extension Toolkit

The purpose of the GetEnvoy Extension Toolkit is to help developers curious about the extensibility of Envoy to get up and running in seconds.

As a developer, you most likely want to

• be able to start from a working and representative example
• have effective development workflow set up from the beginning
• leverage best practices and avoid common pitfalls by default

GetEnvoy Extension Toolkit will help you with all that!

Let’s create Envoy HTTP Filter in Rust

Let’s give GetEnvoy Extension Toolkit a try by developing an Envoy HTTP Filter in Rust!

1. Pre-requirements

Install getenvoy CLI, e.g.

$ curl -L https://getenvoy.io/cli | bash -s -- -b /usr/local/bin

Install Docker

Check

Run

$ getenvoy --version

You should see output similar to

getenvoy version 0.2.0

Run

$ docker --version

You should see output similar to

Docker version 19.03.8, build afacb8b

2. Scaffold a new HTTP Filter extension

To walk through the interactive wizard, run:

$ getenvoy extension init

Alternatively, to skip the wizard, provide the arguments on the command line, e.g.:

$ getenvoy extension init \
    --category envoy.filters.http \
    --language rust \
    --name me.filters.http.my_http_filter \
    my_http_filter 

Check

Run:

$ tree -a my_http_filter

You should see output similar to

my_http_filter
├── .cargo
│   └── config
├── .getenvoy
│   └── extension
│       └── extension.yaml
├── .gitignore
├── Cargo.toml
├── README.md
├── src
│   ├── config.rs
│   ├── factory.rs
│   ├── filter.rs
│   ├── lib.rs
│   └── stats.rs
└── wasm
    └── module
        ├── Cargo.toml
        └── src
            └── lib.rs

3. Build the extension

Run

$ getenvoy extension build

You should see output similar to

Updating crates.io index
Downloaded envoy-sdk v0.1.0
...
Compiling envoy-sdk v0.1.0
...
Finished dev [unoptimized + debuginfo] target(s) in 23.57s
Copying *.wasm file to 'target/getenvoy/extension.wasm'

Check

Run

$ tree target/getenvoy/

You should see output similar to

target/getenvoy
└── extension.wasm

4. Run unit tests

Run

$ getenvoy extension test

Check

You should see output similar to

running 1 test
test tests::should_initialize ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

5. Run extension in Envoy

Let’s do this part the hard way. Rather than use a single command, let’s do every step by hand.

i. Download Envoy binary

You need to download the same version of Envoy the extension is being developed against.

Run

$ cat .getenvoy/extension/extension.yaml

You should see output similar to

…

# Runtime the extension is being developed against.

runtime:
  envoy:
    version: wasm:1.15

To download that version of Envoy, run

$ getenvoy fetch wasm:1.15

You should see output similar to

fetching wasm:1.15/darwin
[Fetching Envoy] 100%

ii. Create an example Envoy configuration

Run

$ getenvoy extension examples add

To check, run

$ tree .getenvoy/extension/examples

You should see output similar to

.getenvoy/extension/examples
└── default
  ├── README.md
  ├── envoy.tmpl.yaml
  ├── example.yaml
  └── extension.json

iii. Learn more about the example configuration by looking into the README.md file

iv. Quick peek into example Envoy config

Run

$  cat .getenvoy/extension/examples/default/envoy.tmpl.yaml

You should see output similar to

...
http_filters:
- name: envoy.filters.http.wasm
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    config:
      configuration: {{ .GetEnvoy.Extension.Config }}
      name: {{ .GetEnvoy.Extension.Name }}
      root_id: {{ .GetEnvoy.Extension.Name }}
      vm_config:
        vm_id: {{ .GetEnvoy.Extension.Name }}
        runtime: envoy.wasm.runtime.v8
        code: {{ .GetEnvoy.Extension.Code }}
- name: envoy.filters.http.router
...

Notice that the example Envoy config contains placeholders {{ … }} that will be resolved by getenvoy CLI. 

v. Start Envoy with the example configuration

Run

$ getenvoy extension run

You should see output similar to

info Envoy command: [$HOME/.getenvoy/builds/wasm/1.15/darwin/bin/envoy -c /tmp/getenvoy_extension_run732371719/envoy.tmpl.yaml]
...
[info][main] [external/envoy/source/server/server.cc:339] admin address: 127.0.0.1:9901
...
[info][config] [external/envoy/source/server/listener_manager_impl.cc:700] all dependencies initialized. starting workers
[info][main] [external/envoy/source/server/server.cc:575] starting main dispatch loop

At this point, Envoy is started and extension is ready for use.

Check

To test the HTTP Filter extension, run

$ curl -i https://0.0.0.0:10000

In the Envoy output, you should see lines similar to

my_http_filter: #2 new http exchange starts at 2020-07-01T18:22:51.623813+00:00 with config:
my_http_filter: #2 observing request headers
my_http_filter: #2 -> :authority: 0.0.0.0:10000
my_http_filter: #2 -> :path: /
my_http_filter: #2 -> :method: GET
my_http_filter: #2 -> user-agent: curl/7.64.1
my_http_filter: #2 -> accept: */*
my_http_filter: #2 -> x-forwarded-proto: http
my_http_filter: #2 -> x-request-id: 8902ca62-75a7-40e7-9b2e-cd7dc983b091
my_http_filter: #2 http exchange complete

Now that you know what is happening under the hood, next time you can start the extension simply with:

$ getenvoy extension run

6.  Add a new feature

Let’s add a new feature to the extension – inject an extra header into proxied HTTP responses.

First, let’s update extension config to hold the name of a header to inject (added lines are highlighted in bold):

src/config.rs

/// Configuration for a Sample HTTP Filter.
#[derive(Debug, Default, Deserialize)]
pub struct SampleHttpFilterConfig {
#[serde(default)]
pub response_header_name: String, // added code
}

Next, let’s add on_response_headers method to the SampleHttpFilter:

src/filter.rs

/// Called when HTTP response headers have been received.
///
/// Use `filter_ops` to access and mutate response headers.
fn on_response_headers(
    &mut self,
    _num_headers: usize,
    _end_of_stream: bool,
    filter_ops: &dyn http::ResponseHeadersOps,
) -> Result<http::FilterHeadersStatus> {
    if !self.config.response_header_name.is_empty() {
        filter_ops.set_response_header(
            &self.config.response_header_name,
            "injected by WebAssembly extension"
        )?;
    }
    Ok(http::FilterHeadersStatus::Continue)
}

Finally, let’s update extension configuration in the default example setup:

.getenvoy/extension/examples/default/extension.json

{
 "response_header_name": "my-header"
}

Check

To verify the changes, re-restart the example setup:

$ getenvoy extension run

And make a sample request:

$ curl -i localhost:10000

You should see output similar to:

HTTP/1.1 200 OK
content-length: 22
content-type: text/plain
date: Tue, 07 Jul 2020 18:36:23 GMT
server: envoy
x-envoy-upstream-service-time: 0
my-header: injected by WebAssembly extension
 
Hi from mock service!

Notice an extra header injected into the response.

7. Add a new metric

Observability is one of the Envoy’s strongest propositions.

Let’s update the extension to expose metrics about its new behavior. Specifically, let’s provide a counter with a number of HTTP responses the extra header has been injected to.

Let’s edit the source code as follows (added lines are highlighted in bold):

src/stats.rs

use envoy::host::stats::Counter;

/// Sample stats.
pub struct SampleHttpFilterStats {
   requests_total: Box<dyn Counter>,
   responses_injected_total: Box<dyn Counter>,              // added code
} 

impl SampleHttpFilterStats {
   pub fn new(
       requests_total: Box<dyn Counter>,
       responses_injected_total: Box<dyn Counter>,          // added code
   ) -> Self {
       SampleHttpFilterStats {
           requests_total,
           responses_injected_total,                        // added code
       }
   }

   pub fn requests_total(&self) -> &dyn Counter {
       &*self.requests_total
   }

   pub fn responses_injected_total(&self) -> &dyn Counter { // added code
       &*self.responses_injected_total
   }
}

src/factory.rs

/// Creates a new factory.
pub fn new(clock: &'a dyn Clock, stats: &dyn Stats) -> Result<Self> {
    let stats = SampleHttpFilterStats::new(
        stats.counter("examples.http_filter.requests_total")?,
        stats.counter("examples.http_filter.responses_injected_total")?, // added code
    );
    // Inject dependencies on Envoy host APIs
    Ok(SampleHttpFilterFactory {
        config: Rc::new(SampleHttpFilterConfig::default()),
        stats: Rc::new(stats),
        clock,
    })
}

src/filter.rs

/// Called when HTTP response headers have been received.
///
/// Use `filter_ops` to access and mutate response headers.
fn on_response_headers(
    &mut self,
    _num_headers: usize,
    _end_of_stream: bool,
    filter_ops: &dyn http::ResponseHeadersOps,
) -> Result<http::FilterHeadersStatus> {
    if !self.config.response_header_name.is_empty() {
        filter_ops.set_response_header(
            &self.config.response_header_name,
           "injected by WebAssembly extension",
        )?;
        self.stats.responses_injected_total().inc()?; // added code
    }
    Ok(http::FilterHeadersStatus::Continue)
}

Check

Let’s re-restart the example setup, make a sample request and inspect Envoy metrics:

$ getenvoy extension run

$ curl -i localhost:10000
$ curl -i localhost:10000

You should see output similar to

examples.http_filter.responses_injected_total: 2

That concludes our brief look into development flow using GetEnvoy Extension Toolkit.

Final words

So far, we’ve shown you how easy it is to get started developing your very own Envoy extension using GetEnvoy.

By combining the convenience of getenvoy CLI and guidance from Envoy Rust SDK you can be productive from day one.

And besides the HTTP Filter extension demoed above, you can also use the toolkit to develop Envoy extensions of other types, such as Network Filter and Access Logger.

What’s coming next

In the coming months, we will be adding a number of new features to GetEnvoy.

On one side, we will switch our focus to the experience of extension users, providing them with a way to easily discover and try extensions out.

On the other, we will continue improving the experience on the developer flow. Support for more programming languages and more extension types will follow.

Stay tuned for further updates from GetEnvoy!

And please, share with us your awesome Envoy extensions in Rust 🙂 

References

https://www.getenvoy.io/tutorials/getting-started-http-filter-rust/

https://www.getenvoy.io/reference/getenvoy_extension_toolkit_reference/

https://docs.rs/envoy-sdk/

https://github.com/tetratelabs/getenvoy

https://github.com/tetratelabs/envoy-wasm-rust-sdk

https://www.getenvoy.io/

Yaroslav Skopets is a Tetrate engineer and Envoy contributor focusing on advancing Wasm support for the Envoy proxy. Tetrate writer Tevah Platt edited this content. 

Tetrate, the enterprise service mesh company, is committed to open source and created the open source project GetEnvoy to make envoy adoption easy.