Let me describe a common service mesh scenario…
You’ve deployed your application and it is happily consuming some external resources on the ‘net. For example, say that
reviews.default.svc.cluster.local is communicating with external service
redis-12.eu-n-3.example.com. But you need to switch to a new external service
redis-db-4.eu-n-1.example.com. You’re using a service mesh, right? The light bulb goes on — how about we just redirect all traffic from
redis-db-4.eu-n-1.example.com? That certainly will work; add or modify a few resources and voila, traffic is re-routed and with zero downtime!
Only now there’s a new problem — your system is looking less like the tidy cluster you started with and more like a bowl of spaghetti!
What if we used a neutral name for the database? How about
db.default.svc.cluster.local? We might start with the same mechanism for re-routing traffic: from
redis-12.eu-n-3.example.com. Then when we needed to make the above change we just need to update the configuration to route traffic from
redis-db-4.eu-n-1.example.com. Done and again with zero downtime!
This is Virtual Host Routing to a Logical DNS Name. Virtual Host Routing is traditionally a server-side concept — a server responding to requests for one or more virtual servers. With a service mesh, it’s fairly common to also apply this routing to the client side, redirecting traffic destined for one service to another service.
To give you a bit more context, a “logical name” is defined as a placeholder name that is mapped to a physical name when a request is made. An application might be configured to talk to its database at
db.default.svc.cluster.local which is then mapped to
redis-12.eu-n-3.example.com in one cluster and
redis-db-4.eu-n-1.example.com in another.
Common practice is to use configuration to supply the DNS names to an application (add a
DB_HOST environment variable set directly to
redis-db-4.eu-n-1.example.com). By setting the configuration to a physical server, it’s harder to redirect the traffic later.
What are some best practices for working with external services? Processes like restricting outbound traffic and TLS origination can have a significant impact. The best practices listed below are not required, but this post is written assuming these practices are being followed.
Restricting Outbound Traffic
The outbound traffic policy determines if external services must be declared. A common setting for this policy is
ALLOW_ANY — any application running in your cluster can communicate to any external service. We recommend that the outbound traffic policy is set to
REGISTRY_ONLY which requires that external services are defined explicitly. For security, the Aspen Mesh distribution of Istio has
REGISTRY_ONLY by default.
If you are using an Istio distribution or if you want to explicitly set the outbound traffic policy, restrict outbound traffic by adding the following to your values file when deploying the
If an application communicates directly over HTTPS to upstream services, the service mesh can’t inspect the traffic and it has no idea if requests are failing (it’s all just encrypted traffic to the service mesh). The proxy is just routing bits. By having the proxy do “TLS origination”, the service mesh sees both requests and responses and can even do some intelligent routing based on the content of the requests.
We’ll use the rest of this blog to step through how to configure your application to communicate over just HTTP (change
https://… configuration to just
How to Set Up Virtual Host Routing to a Logical DNS Name
A logical DNS name must still be resolvable. Otherwise the service mesh won’t attempt to route traffic to it. In the yaml below, we are defining a DNS name of
httpbin.default.svc.cluster.local so that we can route traffic to it.
- port: 443
- port: 80
A service entry indicates that we have services running in our cluster that need to communicate to the outside Internet. The actual host (physical name) is listed (
httpbin.org in this example). Note that because we have the proxy doing TLS origination (just plain http between the application and the proxy), port
443 lists a protocol of
HTTP (instead of
- number: 443
A virtual service defines a set of rules to apply when traffic is routed to a specific host. In this example when traffic is routed to the
/foo endpoint of
httpbin.default.svc.cluster.local, the following rules are applied:
- Rewrite the URI from
- Rewrite the HOST header from
- Re-route the traffic to
Note that just re-routing the traffic is not sufficient for the server to handle our requests. The
HOST header is how a server understands how to process a request.
A destination rule defines policies that are applied to traffic after routing has occurred. In this case we define policies for traffic going to port
httpbin.org. The above configuration is routing plain HTTP traffic to port
443. The following destination rule indicates that this traffic should be sent over HTTPS via TLS (the proxy will do TLS origination).
mode: SIMPLE # initiates HTTPS when accessing httpbin.org
Testing with a simple pod
That’s it! You can now deploy a service and configure it to talk to
http://httpbin.default.svc.cluster.local/foo and traffic will get re-routed to https://httpbin.org/get. Let’s test it out…
- name: test-container
command: ["/bin/sleep", "3650d"]
$ kubectl apply -f pod.yaml
The above pod just sleeps for 10 years. Not very interesting by itself but it also provides the
curl command that we can use for testing.
2. Curl the logical name:
$ kubectl exec -c test-container test-pod -it -- \
curl -v http://httpbin.default.svc.cluster.local/foo
Here is the expected output (the response body was removed for brevity):
* Trying 100.66.72.128...
* TCP_NODELAY set
* Connected to httpbin.virtual-host-routing.svc.cluster.local (100.66.72.128) port 80 (#0)
> GET /foo HTTP/1.1
> Host: httpbin.virtual-host-routing.svc.cluster.local
> User-Agent: curl/7.60.0
> Accept: */*
< HTTP/1.1 200 OK
< date: Mon, 23 Sep 2019 22:04:08 GMT
< content-type: application/json
< content-length: 916
< x-amzn-requestid: 1743ed99-df5b-41c2-aa46-9662e10be674
< cache-control: public, max-age=86400
< x-envoy-upstream-service-time: 219
< server: envoy
* Connection #0 to host httpbin.virtual-host-routing.svc.cluster.local left intact
And with that, you should be set!
Virtual Host Routing to a Logical DNS Name can be a useful tool, allowing a server to communicate with external services without needing to specify the physical DNS name of the external service. And a service mesh makes it easy, enhancing your capabilities and keeping things rational (no offense to spaghetti lovers!).
If you enjoyed learning about (and trying out!) this topic, subscribe to our blog to get updates when new articles are posted.