Every time an mTLS problem arises, it has the potential to cause a deployment or service outage. 

Mutual TLS (mTLS) is a method of ensuring traffic is authenticated and encrypted between Kubernetes services. In a highly distributed cloud-native system, using mTLS can drastically increase your security posture by eliminating impersonators, bad actors, and traffic snooping. However, this comes with additional complexity, and troubleshooting issues can be challenging.   

Implementing mTLS is often done with a sidecar service mesh like Istio. Istio supports mTLS out of the box, but in our experience the default configuration is not enough. 

Simple configuration changes are often all that is needed to solve the most taxing of issues.  

The Aspen Mesh team has experience helping our customers deploy mutual Transport Layer Security (TLS) across a wide range of industry verticals and situations. 

8 ways to mitigate risk when deploying mutual TLS:

1. Enforce STRICT mode as a default

Why it’s important: Ensures end-to-end security for all devices and services. 

PERMISSIVE mode allows a service to accept both plain text traffic and mutual TLS traffic at the same time. This feature greatly improves the mutual TLS onboarding experience; however, this mode is not as secure as STRICT mode. When STRICT is set, the service only accepts mutual TLS traffic. For services with transport authentication enabled by an authentication policy, the peers section has an additional key (mode), which you can use to define which traffic a service can accept from its peers. This mode key can take two values: PERMISSIVE or STRICT. If PERMISSIVE is set, the service can accept both plain-text traffic as well as encrypted. If STRICT is set, the service only accepts mutual TLS traffic. 

In Istio, sidecar proxies for each service are needed to establish mutual TLS communication. However, during a service onboarding process there may be cases where the operator cannot install sidecar proxies for all clients and services at the same time. In these situations, it is ideal to enable communication between non-Istio client services and Istio target services. By enabling the PERMISSIVE mode in the authentication policy for a target service, non-Istio client services can continue to send plain-text traffic to the target service until the onboarding process is complete, when STRICT can be enabled. 

 

2. Enable mutual TLS global by default

Why it’s important: Defines scope of mutual TLS settings. 

There are three levels of granularity through which mutual TLS settings can be defined. For each service, Istio applies the narrowest matching policy. The order is: service-specific, namespace-wide, mesh-wide. It is best to enforce mTLS globally, and then change to PERMISSIVE at the service level only when necessary.   


3. Don’t rely on Perimeter defenses to secure core applications and services

Why it’s important: Greater security and compliance with industry regulations. Perimeter defenses are insufficient; secure critical core services and traffic with end-to-end security. 

Don’t rely on Perimeter defenses to secure your core applications and services. While defense-in-depth has been a valid approach to security, it does not stop an attacker from viewing unencrypted data after the security bubble has been pierced. Only services that are encrypted end-to-end can provide the level of security needed to protect confidential information such as financial transactions or medical records. 

 

4. Visibility for service communication

Why it’s important: Eases misconfigurations and aids in troubleshooting. 

A visibility tool is paramount to get a picture of what’s happening at the service level. For example, if one service has no sidecar installed, or a service with a sidecar has a service-specific policy set and there is something wrong with communication between services, the cause may be one service has mutual TLS enabled where another service may not support mutual TLS. A visibility tool that runs in real time can help to find problems quickly as they arise, dynamically visualizing services and their communications to ease misconfigurations and aid in troubleshooting. 


5. Enable an external Certificate Authority

 Why it’s important: More easily integrate with external systems.

Although the Istio Certificate Authority (CA) generates a self-signed root certificate/key, and uses them to sign all workload certificates, external parties normally will not trust a digital certificate signed by an Internal CA. Also, the certificate management overhead of an Internal CA is higher than that of external CA. External parties normally trust a digital certificate signed by a trusted External CA easing the ability to work with external services over using an Internal CA. 

 

6. Optimize System Design – Design for mTLS and service mesh from the start

Why it’s important: Retrofitting system-wide mTLS after deployment of the platform and services is difficult and error prone. Designing for mTLS from the beginning will increase your success rate and ensure that all communication channels are secured properly.  Attempting to implement mTLS after deploying the platform and applications will require reconfiguration of the service mesh, installing and configuring a root CA, redeploying all applications, and regression testing of the applications and external integrations. 

 

7. Mitigate the mutual TLS performance hit

Why it’s important: Mitigate to restore ideal system performance to prevent services operating at less than optimum levels 

After enabling mutual TLS, depending on the service or workload, a performance hit may be experienced which can affect services. A performance hit up to 10% is possible, but largely depends on the services and workload of the deployment. To mitigate any resultant performance hit, Envoy can be tuned, and the underlying deployment hardware re-sized to restore system performance. 

 

8. Decrease certificate rotation time

Why it’s important: Enhances security which helps to defeat hackers. 

Istio has a default Certificate rotation time that rotates new Certificates, and encryption keys are issued. However, the longer this rotation interval, the more opportunity an attacker has to defeat the encryption. Set the Certificate rotation time to the minimum rotation time needed for your deployment / services. This helps defeat hackers by shortening the window of time they have to execute “brute force” attacks. 

 

How can Aspen Mesh help you on your Istio journey?

Aspen Mesh provides the knowledge, leadership & expertise needed to successfully help you deploy mTLS. We have helped customers deploy in the Enterprise, Telco, Financial services, and Healthcare verticals. We have deployed mTLS in complex High Availability and Disaster Recovery environments and provided upfront training before the deployment starts, and expert support after the deployment is complete. We can: 

  • Advocate for your needs within the open-source community 
  • Provide training and expert support 
  • Mitigate your deployment risk with our service offerings, which include: 
  • A 360° quantitative and qualitative evaluation of your Istio environment to identify problems. 
  • Expertise in the design of a scalable service mesh environment. 
  • Comprehensive security review and detailed recommendations. 
  • Upgrade Istio in your environment, then ensure it’s running smoothly. 
  • Existing Istio environment benchmarking and tuning for maximum performance. 

 

Deployment success means picking the right partner

mTLS can be challenging, integration with external workloads is problematic, and debugging information can be a blackhole. Visibility is essential when something goes wrong in any deployment, but more so in a mTLS deployment. We provide 24/7 and international support delivered by our expert Istio engineers from F5, a trusted enterprise vendor you can rely on. You can be assured our engineers are with you every step of the way, no matter the problem – big or small.  

Aspen Mesh can help fully manage and monitor your mTLS deployment, setting you up for success in the long term. Our team of experts have the knowledge and expertise to guide you every step of the way. See our Professional Services including in-depth Istio Health Assessment, Architecture Design and Security Essentials and Custom Projects. Get in touch to start the conversation.