Setting up an envoy proxy can help websites scale, especially in a dockerized world of micro-services. As websites grow, routing traffic becomes increasingly complicated. Different subdomains, domains, and paths might need to be routed to different backend servers. TLS (https) certificates need to be managed, monitoring/observability becomes important, and things like WebSockets, GRPC, statelessness, and Kubernetes might play a role.
Check out the open-source Switchboard project on Github. You can pull it from
inzania/switchboard:lateston Docker.
Envoy Proxy + LetsEncrypt + Docker
Switchboard resembles a Kubernetes ingress controller, but is more powerful and more portable. For example, it manages SSL certificate generation and renewal while still achieving statelessness. The docker container may be configured with any combination of mounted config directories and environment variables. This makes it easy to set up via docker-compose, Kubernetes, or any system which can deploy containers.
Features
- TLS termination (https)
- GRPC
- Web Sockets
- HTTP->HTTPS redirection
- Domain name redirection
- Sharding (different subdomains for different environments)
- Automatic certificate generation via CertBot/LetsEncrypt
- Stateless (certificate backup via S3)
Envoy Kubernetes Ingress
This example assumes kube2iam for AWS authentication in order to achieve the S3 backup-and-restore of certbot-generated certifiactes. It also tweaks the default logging formats to structured JSON, making it well suited for a variety of ingestion pipelines. Finally, it provides samples of readiness and liveness checks.
I have this deployed on AWS in order to keep costs low (as an alternative to using ELBs):
The more you're willing to self-manage your cloud software, the cheaper the costs will be. These days, it's actually relatively easy to use tools like Docker and Kubernetes to deploy stable infrastructure. I even use Docker and Kubernetes at home on "spellbook" Raspberry Pis for home-automation purposes. In this post, I'll show that the same basic tools can be used to manage an AWS Kubernetes deployment for extremely cheap.
Configuration Examples
Switchboard aims to provide a simple DSL for defining ingress (what traffic is accepted?) and egress (where does that traffic go?) for the envoy proxy. See the Github project for complete usage documentation.
Ingress
http://www:mydomain.com@my-cluster
Simply route traffic directed athttp://www.mydomain.comtomy-cluster(egress)https://www?:mydomain.com/search@google.com
Redirect both of the following togoogle.com:https://www.mydomain.com/searchhttps://mydomain.com/search
https!://:mydomain.com/api!@api-clusterhttps!forceshttps/api!strips the/apiprefix from the path before routing to the egress cluster
wss?://api:mydomain.com@my-clusterwssandwswork identically tohttpsandhttp, except support web socket upgradingwss?andhttps?listen on both secure and insecure schemas.
Egress
my-cluster:http@localhost:5200: Route regular HTTP traffic formy-clustertolocalhost:5200my-cluster:grpc@localhost:5201: Route GRPC traffic formy-clustertolocalhost:5201
Enabling Environments
I use Switchboard for this blog. The same Kubernetes production AWS cluster is running several different Docker containers, each hosting different projects of mine.
At home, I have a Linux computer running an almost identical deployment as my staging environment. I use the kustomize tool to change the ingress so that it listens on different domains.
Finally, development takes place on my laptop. Kustomize is used again, this time to enable Switchboard’s shard feature. The result is to enable URLs like dev-www.technicallywizadry.com, where I can access the development version of this blog thanks to an edit to /etc/hosts.
Technical Design
The docker image is based upon envoyproxy/envoy:latest. The Dockerfile layers in certbot, aws, and a few others. The Python start script generates the necessary configuration files. It’s the simple work of a Python-newb, but the resulting configuration files are fairly well-tuned.
