Istio uses the Envoy sidecar proxy to handle traffic within the service mesh. The following article describes how to use an external proxy, F5 BIG-IP, to integrate with an Istio service mesh without having to use Envoy for the external proxy. This can provide a method to extend the service mesh to services where it is not possible to deploy an Envoy proxy.
This method could be used to secure a legacy database to only allow authorized connections from a legacy app that is running in Istio, but not allow any other applications to connect.
Securing Legacy Protocols
A common problem that customers face when deploying a service mesh is how to restrict access to an external service to a limited set of services in the mesh. When all services can run on any nodes it is not possible to restrict access by IP address (“good container” comes from the same IP as “malicious container”).
One method of securing the connection is to isolate an egress gateway to a dedicated node and restrict traffic to the database from those nodes. This is described in Istio’s documentation:
Istio cannot securely enforce that all egress traffic actually flows through the egress gateways. Istio only enables such flow through its sidecar proxies. If attackers bypass the sidecar proxy, they could directly access external services without traversing the egress gateway. Thus, the attackers escape Istio’s control and monitoring. The cluster administrator or the cloud provider must ensure that no traffic leaves the mesh bypassing the egress gateway.
— https://istio.io/docs/examples/advanced-gateways/egress-gateway/#additional-security-considerations (2019-03-25)
Another method would be to use mesh expansion to install Envoy onto the VM that is hosting your database. In this scenario the Envoy proxy on the database server would validate requests prior to forwarding them to the database.
The third method that we will cover will be to deploy a BIG-IP to act as an egress device that is external to the service mesh. This is a hybrid of mesh expansion and multicluster mesh.
Mesh Expansion Without Envoy
Under the covers Envoy is using mutual TLS to secure communication between proxies. To participate in the mesh, the proxy must use certificates that are trusted by Istio; this is how VM mesh expansion and multicluster service mesh are configured with Envoy. To use an alternate proxy we need to have the ability to use certificates that are trusted by Istio.
Example of Extending Without Envoy
A proof-of-concept of extending the mesh can be taken with the following example. We will create an “echo” service that is TCP based that will live outside of the service mesh. The goal will be to restrict access to only allow authorized “good containers” to connect to the “echo” service via the BIG-IP. The steps involved.
- Retrieve/Create certificates trusted by Istio
- Configure external proxy (BIG-IP) to use trusted certificates and only trust Istio certificates
- Add policy to external proxy to only allow “good containers” to connect
- Register BIG-IP device as a member of the Istio service mesh
- Verify that “good container” can connect to “echo” and “bad container” cannot
First we install a set of certificates on the BIG-IP that Envoy will trust and configure the BIG-IP to only allow connections from Istio. The certs could either be pulled directly from Kubernetes (similar to setting up mesh expansion) or generated by a common CA that is trusted by Istio (similar to multicluster service mesh).
Once the certs are retrieved/generated we install them onto the proxy, BIG-IP, and configure the device to only trust client side certificates that are generated by Istio.
To enable a policy to validate the identity of the “good container” we will inspect the X509 Subject Alternative Name fields of the client certificate to inspect the spiffe name that contains the identity of the container.
Once the external proxy is configured we can register the device using “istioctl register” (similar to mesh expansion).
To verify that our test scenario is working we will have two namespaces “default” and “trusted”. Connections from “trusted” will be allowed and “default” will be reject. From each namespace we create a pod and run the command “nc bigip.default.svc.cluster.local 9000”. Looking at our BIG-IP logs we can verify that our policy (iRule) worked:
Mar 25 18:56:39 ip-10-1-1-7 info tmm5: Rule /Common/log_cert <CLIENTSSL_CLIENTCERT>: allowing: spiffe://cluster.local/ns/trusted/sa/sleep
Mar 25 18:57:00 ip-10-1-1-7 info tmm2: Rule /Common/log_cert <CLIENTSSL_CLIENTCERT>: rejecting spiffe://cluster.local/ns/default/sa/default
Connection from our “good container”
/ # nc bigip.default.svc.cluster.local 9000 hi HI
Connection from our “bad container”
# nc bigip.default.svc.cluster.local 9000
In the case of the “bad container” we are unable to connect. The “nc”, netcat, command is simulating a very basic TCP client. A more realistic example would be connecting to an external database that contains sensitive data. In the “good” example we are echo’ing back the capitalized input (“hi” becomes “HI”).
Just One Example
In this article we looked at expanding a service mesh without Envoy. This was focused on egress TCP traffic, but it could be expanded to:
- Using BIG-IP as an SNI proxy instead of NGINX
- Securing inbound traffic using mTLS and/or JWT tokens
- Using BIG-IP as an ingress gateway
- Using ServiceEntry/DestinationRules instead of registered service
If you want to see the process in action, check out this short video walkthrough.
Let me know in the comments whether you’re interested in any of these use-cases or come-up with your own. Thank you!