Aspen Mesh proud to sponsor IstioCon 2022

Presentations on Istio Security, Dual Stack Support & TLS Orig. at IstioCon 2022

Virtual IstioCon starts Monday, April 25th. It's the biggest gathering of the Istio community and a great place to learn and share ideas. Aspen Mesh is a longtime Istio open source contributor--we are a top five Istio Contributor of Pull Requests. We are proud to sponsor IstioCon 2022 for the third year.

We're excited three members of the Aspen Mesh team are presenters this year.

Aspen Mesh presentations at IstioCon 2022

Tuesday, April 26, 10:30-10:40 a.m. EST - A Beginner's Guide to Following Istio's Security Best Practices, Jacob Delgado, Senior Software Engineer
Following the Istio Security Best Practices page is a daunting task for newcomers to Istio. Even experienced operators have difficulty discerning where to begin. In this talk I will present an easy way for beginners to adopt Istio and settings/configuration I recommend based on my experience.  

Tuesday, April 26, 11 a.m. EST - Istio Upgrades, Jacob Delgado, Senior Software Engineer & Sam Naser, Software Engineer at Google
Are upgrades getting easier? How easy is easy enough? Are helm and revision based upgrades catching on? What is still painful? How often do you upgrade? How often would you like to? Are patches easier than minor upgrades?

Wednesday, April 27, 10:40-10:50 a.m. EST - TLS Origination Best Practices, Kenan O'Neal, Software Engineer
Quick dive for beginners on TLS origination to improve security. This talk will focus on settings that may not be expected for new users with a focus on validating settings. I will touch on what settings Istio uses by default and how to configure Destination Rules to correctly check certificates.  

Thursday, April 28, 10:50-11:00 a.m. EST - Dual Stack Cluster Setup, Josh Tischer, Lead DevOps Engineer
Dual Stack support is very limited in today’s cloud ecosystem. Learn how to run/test Istio on a Dual Stack cluster in AWS on both OpenShift 4.8+ and KubeAdmin. OpenShift 4.7+ is one of the few options that officially support Dual Stack mode for bare metal clusters and Azure. I am excited to share Aspen Mesh’s experience and empower your team with another option for Dual Stack support. 

You can explore the full list of IstioCon sessions here and be sure to register for IstioCon today

About Aspen Mesh
Enterprise-class Istio service mesh is all we do at Aspen Mesh by F5. We offer Professional Services and 24/7 Support plans to ensure you have OS Istio experts available when you need them. 

Explore our Service Mesh Knowledge Hub for the latest resources on how OS Istio drives performance from your microservices architecture. Deep dive white paper topics include: Multi-cluster deployment to enable hybrid and multi-cloud architectures, security, mTLS, compliance and more.  


mTLS Authentication for Microservices Security is Critical to Digital Transformation

For an enterprise, mentions of “Cloud Native” and “Digital Transformation” are two ways of saying that a service mesh deployment is on the cards. Once deployed, the service mesh forms the backbone of business operations and is usually paramount to business continuity. As an enterprise starts to implement its Digital Transformation plan and migrates from a monolithic application environment to a cloud native application environment, security becomes an immediate concern. From a business operations and revenue generation perspective, it is important to understand the benefits and deployment pitfalls of mTLS to ensure business-as-normal operation. 

Here are some common issues we run into all the time

  • Loss of Regulatory Compliance: Several industries such as healthcare and financial services require compliance with mandated specifications, with many other industries complying with agreed best practices. If compliance is lost, then business operations may be affected if regulatory compliance is lost. A solution to this headache is to enforce mutual TLS STRICT mode as default. This will help to regain regulatory compliance by ensuring end-to-end security for all devices and services
  • Loss of Brand Reputation: Customer or internal proprietary information may be made public via exposure from a security breach. Customers may choose to do business with a more secure supplier consequently. A customer or supplier is unlikely to look favorably upon their chosen supplier or vendor if highlighted on the evening news for example. By enabling mutual TLS GLOBAL setting as default will help to protect your reputation by securing customer and proprietary information. 
  • Loss of Business Agility: With dynamic cloud-based applications, upgrades to business logic, services and offerings can seem endless. Frequent security enhancements are also essential. Upgrades come with their own headaches and can slow a business from rolling out new services due to various glitches. It’s best not to rely solely on perimeter defenses, regain agility by securing critical core applications and services with end-to-end security. 
  • Loss of Business Confidence: The service mesh enables many end-users. If one supplier or service has a loss of confidence in the integrity of the service mesh, this can affect everyone -- and other services. The ability to visualize service communication can help to restore confidence by reducing misconfigurations and simplifying troubleshooting. 

 

In our experience at Aspen Mesh, deploying mutual TLS has many benefits, but is easy to misconfigure – which can lead to severe disruptions to business continuity when deploying new microservices or upgrading a service mesh. Download our new white paper, Istio mTLS: 8 Ways to Ensure End-to-End Security, to learn about several more mutual TLS concerns that could spell disaster for your business! 

 

- Andy 


Istio for multi-cluster deployment: Q & A with an expert

Q&A with Brian Jimerson, Solutions Architect and Istio and multi-cloud deployment expert

I recently had a sit down with one of our Aspen Mesh expert Istio engineers to answer some questions that I hear from customers as they start their multi-cluster Istio journey. If your organization already has a series of disparate single-cluster Istio deployments, real benefits can be achieved by connecting them together to create a multi-cluster service mesh. 

Here are some of the highlights of my conversation with Brian Jimerson, one of our seasoned solution engineers whose experience runs deep optimizing Fortune 2000 enterprise architectures. I wore the hat of a customer exploring the ways moving to a multi-cluster environment might impact performance.

Q: I have stringent SLOs and SLAs for my cloud applications. How can I work to meet these?

Brian: Having multiple Kubernetes clusters in different fault zones can help with disaster recovery scenarios. Using a multi-cluster service mesh configuration can simplify cutover for an event.

Q: As an international company, I  have data privacy requirements in multiple countries. How do I ensure that data is stored in a private manner?

Brian: Using Aspen Mesh's locality-based routing can ensure that customers are using a cluster in their own country, while reducing latency. This can help ensure that private data does not leave the user's country.

Q: I need the ability to quickly scale workloads and clusters based on events. How can I do this in a way that's transparent to customers?

Brian: Using a multi-cluster service mesh can help to scale clusters out and in without interruption to users. A multi-cluster service mesh acts like a single cluster service mesh to services, so communication patterns remain unchanged.

Q: I have compliance requirements for workloads, but they also need to be accessed by other applications. How do I do this?

Brian: Using a multi-cluster service mesh, you can operate compliance workloads in a hardened cluster, and securely control traffic to those workloads from another cluster.

 

I encourage you to read the full paper, Advantages of Going Multi-cluster Using Istio -- and Best Practices to Get There. The move to a multi-cluster environment is complex and there are many things to consider. Working with Brian and our Aspen Mesh Solutions team, I've written a a deep-dive paper that spells out both the advantages and things to look out for when evaluating a move to multi-cluster. There are some disadvantages you have to weigh, and I detail those along with the expected performance improvements that come with deploying a multi-cluster service mesh. In this paper I lay out the benefits and important considerations when considering migrating to a multi-cluster that leverages Istio.

Enterprise-class Istio service mesh is all we do at Aspen Mesh by F5. We offer Professional Services and 24/7 Support plans to give you the ability to tap Istio expertise to optimize your microservices environment with industry best practices, then have peace of mind that we're backing you up. Get in touch if you seek a trusted advisor to help you navigate your OS Istio -- we've designed Istio solutions for some of the largest organizations in the world. I encourage you to get in touch if you would like to talk with Brian, a Solutions Architect who can answer any questions you have about how to chart a toward a cloud native environment. If you want to learn more about our full suite of Services delivered by our team of Istio experts (whether you have OS Istio in pre-prod or production), reach out.

 

-Andy


Recent security vulnerabilities require Zero-Trust Security tactics for your microservices environment

Despite significant technological advancements, security is still hard. A single phishing email, missed patch, or misconfiguration can let the bad guys in to wreak havoc or steal data. For companies moving to the cloud and the cloud-native architecture of microservices and containerized applications, it’s even harder. Now, in addition to the perimeter and the network itself, the myriad connections between microservice containers must also be protected. 

With microservices, the surface area of your network vulnerable to attack increases exponentially, putting data at greater risk. Moreover, network-related problems like access control, load balancing, and monitoring that had to be solved only once for a monolithic application now must be solved separately for each service within a cluster, as well as between clusters. 

Zero Trust Security Methodology and Networking Principles

Zero-Trust dates to the 1990’s as a method for “Perimeter-less” security. The main concept behind the methodology is “never trust, always verify” even if the network was previously verified.

  • Networks should always be considered hostile 
  • Network locality is not sufficient for deciding trust in a network 
  • Every device, user, and request should be authenticated and authorized 
  • Network policies must be dynamic and calculated from as many sources of data as possible
     

Today, it’s essential to apply a Zero-Trust approach to network security and to service mesh technology. In our white paper just completed, Zero-Trust Security for your Microservices Architecture, we outline what it takes to implement the key tenets of Zero-Trust security using a service mesh to secure a microservices environment. In the paper we provide the steps to mitigate cyberattacks to protect containerized applications. 

What's covered in our white paper, Zero-Trust Security for your Microservices Architecture:
  1. Zero-Trust authentication methodology for a service mesh
  2. mTLS encryption: Achieve non-repudiation for requests without requiring any changes or support from the applications. Identity, certificates, and authorization to ensure “every device, user, and request is authenticated and authorized” -- a Zero Trust principle
  3. Learn the built-in methods Istio uses to combat security vulnerabilities
  4. Ingress and Egress security control within a service mesh


Lastly, in the paper we touch on Aspen Mesh’s approach to Zero-Trust security, including how we configure mTLS, secure ingress, monitor egress, prevent RBAC (Role Based Access Control) misconfigurations and apply policy and configuration best practices.

Aspen Mesh has deep expertise in Istio and understands how to get the most out of it - our Services and 24/7 Service Mesh Support are unmatched in the industry.

- Andy


Get a complimentary health check of your os istio graphic

Get a Health Check Report of your Istio to see if everything's configured and optimized.

How do you know your Open Source Istio is operating at its full potential? At Aspen Mesh, we focus on optimizing Istio-based service mesh for our customers (service mesh is all we do).

We talk to companies every day about their OS Istio, and the most common question we get is, “How do we know we’ve got everything in our Istio implementation working correctly?” Whether you’re in a pre-production environment, have Istio deployed in a portion of your network, or network-wide, there's often a fear something’s not configured correctly or there's a potential problem lurking that you don’t have the insight to head-off. Just as importantly, we're asked if there is enhanced Istio functionality to leverage that can drive better performance.

At Aspen Mesh the first thing we do for a new customer is a 360-degree health check of their Istio implementation. It’s a lot like a 100-point diagnostic inspection for your car – a way to identify what’s working fine, where there are potential problems, and get recommendations from an expert about what’s critical to address immediately.

That got us thinking, we should give everyone this level of insight into their Istio implementation.

Aspen Mesh Now Offers a Complimentary OS Istio Health Check Report. This evaluation provides insight across all key areas, identifies critical issues, directs you to best practices, and recommends next steps. You receive an assessment of your Istio by our Istio experts. This is the same evaluation we conduct for every new Aspen Mesh customer running Istio.

A few things that are covered in the Report:

  • Platform: Ensure a stable foundation for smooth version upgrades.
  • Security: ID security risks & apply best practices.
  • Ingress/Egress: Know you’re following best practices.
  • Application Policy inspection
  • Recommendations about where to optimize your performance.
  • Steps to take to go live with confidence.

You Receive your Report After it is Complete

Our Istio expert will review the report with you and recommend remediation steps for critical items discovered – and answer any questions you have. There's no obligation and the Report typically takes about 2 business days. After the review, we give you with a copy of your report. If you want to learn how we work to tackle any Istio problem you have and optimize an Istio environment, we can also share how to take advantage of Aspen Mesh's array of customized Services and Aspen Mesh 24/7 white glove Expert Support for OS Istio.

Where we get the data about your Istio to build your Report
The Aspen Mesh Istio Inspection Report analyzes your Istio system for common misconfigurations and vulnerabilities.

The Report is done in 3 easy steps:

  1. You run the Aspen Mesh Data Collector tool on a workstation with your Kubernetes context configured. This generates a compressed file with the data collected from your Istio installation.
  2. You upload the compressed data file to the Aspen Mesh site.
  3. Aspen Mesh engineers analyze the data collected and build your customer report that details all of our findings.

The Aspen Mesh Data Collector collects the following data:

  • Kubernetes, Istio, and Envoy versions
  • Node topology (number of nodes, node size)
  • Objects installed in your cluster (Kubernetes and Istio objects)
  • Kubernetes events

Note that the Aspen Mesh Data Collector does not collect any potentially sensitive data such as secrets, certificates, or logs. All data that is collected is securely stored and accessed only by Aspen Mesh. Get in touch if you have questions about the process --  I can send you a link to our Data Collector tool and share how we gather and analyze your data to provide a comprehensive assessment. Just send me a note and I'm happy to connect.

-Steven Cheng, Sr. Solutions Engineer at Aspen Mesh


you've got kubernetes promotional graphic

Get App-focused Security from an Enterprise-class Service Mesh | On-demand Webinar

In our webinar you can now view on demand, You’ve Got Kubernetes. Now you Need App-focused Security using Istio, we teamed with Mirantis, an industry leader in enterprise-ready Kubernetes deployment and management, to talk about security, Kubernetes, service mesh, istio and more. If you have Kubernetes, you’re off to a great start with a great platform for security based on Microsegmentation and Network Policy. But firewalls and perimeters aren’t enough -- even in their modern, in-cluster form.  

As enterprises embark on the cloud journey, modernizing applications with microservices and containers running on Kubernetes is key to application portability, code reuse and automation. But along with these advantages come significant security and operational challenges due to security threats at various layers of the stack. While Kubernetes platform providers like Mirantis manage security at the infrastructure, orchestration and container level, the challenge at application services level remains a concern. This is where a service mesh comes in. 

Companies with a hyper focus on security – like those in healthcare, finance, government, and highly regulated industries – demand the highest level of security possible to thwart cyberthreats, data breaches and non-compliance issues. You can up level your security by adding a service mesh that’s able to secure thousands of connections between microservices containers inside of a single cluster or across the globe. Today Istio is the gold standard for enterprise-class service mesh for building Zero Trust Security. But I’m not the first to say that implementing open source Istio has its challenges -- and can cause a lot of headaches when Istio deployment and management is added to a DevOps team’s workload without some forethought.  

Aspen Mesh delivers an Istio-based, security hardened enterprise-class service mesh that’s easy to manage. Our Istio solution reduces friction between the experts in your organization because it understands your apps -- and it seamlessly integrates into your SecOps approach & certificate authority architecture. 

It’s not just about what knobs and config you adjust to get mTLS in one cluster – in our webinar we covered the architectural implications and lessons learned that’ll help you fit service mesh into your up-leveled Kubernetes security journey. It was a lively discussion with a lot of questions from attendees. Click the link below to watch the live webinar recording.

-Andrew

 

Click to watch webinar now:

On Demand Webinar | You’ve Got Kubernetes. Now you Need App-focused Security using Istio.

 The webinar gets technical as we delve into: 

  • How Istio controls North-South and East-West traffic, and how it relates to application-level traffic. 
  • How Istio secures communication between microservices. 
  • How to simplify operations and prevent security holes as the number of microservices in production grows. 
  • What is involved in hardening Istio into an enterprise-class service mesh. 
  • How mTLS provides zero-trust based approach to security. 
  • How Aspen Mesh uses crypto to give each container its own identity (using a framework called SPIFFE). Then when containers talk to each other through the service mesh, they prove who they are cryptographically. 
  • Secure ingress and egress, and Cloud Native packet capture. 

Improve your application with service mesh

Improving Your Application with Service Mesh

Engineering + Technology = Uptime 

Have you come across the term “application value” lately? Software-first organizations are using it as a new form of currency. Businesses delivering a product or service to its customers through an application understand the growing importance of their application’s security, reliability and feature velocity. And, as applications that people use become increasingly important to enterprises, so do engineering teams and the right tools 

The Right People for the Job: Efficient Engineering Teams 

Access to engineering talent is now more important to some companies than access to capital. 61% of executives consider this a potential threat to their business. With the average developer spending more than 17 hours each week dealing with maintenance issues, such as debugging and refactoring, plus approximately four hours a week on “bad code” (representing nearly $85 billion worldwide in opportunity cost lost annually), the necessity of driving business value with applications increases. And who is it that can help to solve these puzzles? The right engineering team, in combination with the right technologies and tools. Regarding the piece of the puzzle that can solved by your engineering team, enterprises have two options as customer demands on applications increase:  

  1. Increase the size and cost of engineering teams, or  
  2. Increase your engineering efficiency.  

Couple the need to increase the efficiency of your engineering team with the challenges around growing revenue in increasingly competitive and low margin businessesand the importance of driving value through applications is top of mind for any business. One way to help make your team more efficient is by providing the right technologies and tools. 

The Right Technology for the Job: Microservices and Service Mesh 

Using microservices architectures allows enterprises to more quickly deliver new features to customers, keeping them happy and providing them with more value over timeIn addition, with microservices, businesses can more easily keep pace with the competition in their space through better application scalability, resiliency and agility. Of course, as with any shift in technology, there can be new challenges.  

One challenge our customers sometimes face is difficulty with debugging or resolving problems within these microservices environments. It can be challenging to fix issues fast, especially when there are cascading failures that can cause your users to have a bad experience on your applicationThat’s where a service mesh can help. 

Service mesh provides ways to see, identify, trace and log when errors occurred and pinpoint their sources. It brings all of your data together into a single source of truth, removing error-prone processes, and enabling you to get fast, reliable information around downtime, failures and outages. More uptime means happy users and more revenue, and the agility with stability that you need for a competitive edge. 

Increasing Your Application Value  

Service mesh allows engineering teams to address many issues, but especially these three critical areas: 

  • Proactive issue detection, quick incident response, and workflows that accelerate fixing issues 
  • A unified source of multi-dimensional insights into application and infrastructure health and performance that provides context about the entire software system 
  • Line of sight into weak points in environments, enabling engineering teams to build more resilient systems in the future  

If you or your team are running Kubernetes-based applications at scale and are seeing the advantages, but know you can get more value out of them by increasing your engineering efficiency and uptime for your application's’ users, it’s probably time to check out a service mesh. 


How Service Mesh Helps Application Management Aspen Mesh

How Service Mesh Helps Application Management

Manage Microservices More Efficiently

Microservice-based applications come with some serious upside, but keeping track of every single service is a challenge — especially for the platform teams that can't narrow their vision to a single microservice. If you’re operating or developing in a microservices architecture, there’s a good chance part of your days are spent wondering what your services are up to. It's frustrating to move from service to service and have to relearn everything; how it's configured, what telemetry you'll have, how it manages certs and keys.

With the adoption of microservices, problems can also emerge due to the sheer number of services that exist in large systems. For a monolith, issues like security, load balancing, monitoring and rate limiting only had to be solved once, but for microservices, those issues now have to be handled separately for each service. 

The good news though? A service mesh helps address many of these challenges so engineering teams – and businesses – can deliver applications more quickly and securely.

 

Take Your Digital Transformation Further

There are many things to think about as companies embrace digital transformation by migrating from legacy or monolithic systems to microservices. Starting with, well, microservices. It’s easy to understand why microservice-based applications are becoming more and more common. Through microservice architectures, enterprises are seeing: 

  • Improved scalability
  • Increased development velocity
  • Easier debugging
  • Rapid alignment between development and user requirements 

As companies build or convert to more modern applications, they’re leveraging microservices to drive differentiation and market leadership. As a side effect, they realize that they are increasing complexity and decentralizing ownership and control. These new challenges require new solutions to effectively monitor, manage and control microservice-based applications at runtime.

Keep in mind that Kubernetes has become the defacto method for enterprises to orchestrate containers. Kubernetes is a superb tool when it comes to deploying, scheduling and running containerized applications through a basic approach to networking that doesn't provide rich service-to-service communication.

That’s where service mesh comes in. A service mesh like Aspen Mesh adds observability, security and policy capabilities to Kubernetes. A service mesh helps to ensure resiliency and uptime – it provides solutions that enable engineering teams to more effectively monitor, control and secure the modern application at runtime. Companies are adopting service mesh as a way to enhance Kubernetes, as it provides a toolbox of features that address various microservices challenges that modern enterprises are facing.

 

Get the Most Out of Your Service Mesh

Fill out the form below to get the free eBook "Getting the Most Out of Your Service Mesh" to keep learning about what a service mesh can do to help, and how to get the most out of it.




How to Capture Packets that Don't Exist

How to Capture Packets that Don’t Exist

One of my favorite networking tools is Wireshark; it shows you a packet-by-packet view of what’s going on in your network. Wireshark’s packet capture view is the lowest level and most extensive you can get before you have to bust out the oscilloscope. This practice is well-established in the pre-Kubernetes world, but it has some challenges if you’re moving to a Cloud Native environment. If you are using or moving to Cloud Native, you’re going to want to use packet-level tools and techniques in any environment, so that’s why we built Aspen Mesh Packet Inspector. It’s designed to address these challenges across various environments, so you can more easily see what’s going on in your network without the complexity.  

Let me explain the challenges facing our users moving into a Kubernetes world. It’s important to note that there are two parts to a troubleshooting session based on packet capture: actually capturing the packets, and then loading them into your favorite tool. Aspen Mesh Packet Inspector enables users to capture packets even in Kubernetes. That’s the part you need to address to power all the existing tools you probably already have.  Leveraging the existing tools is important as our customers have invested heavily in them.  And not just monetarily – their reputation for reliable apps, services and networks depend on the reliability and usefulness of the tools, procedures and experience powered by a packet view. 

What’s so hard about capturing these packets in modern app architectures on Kubernetes? The two biggest challenges are that these packets may never be actual packets that go through a switch, and even if they were, they’d be encrypted and useless. 

Outside of the Kubernetes world, there are many different approaches to capture packets. You can capture packets right on your PC to debug a local issue. For serious network debugging, you’re usually capturing packets directly on networking hardware, like a monitor port on a switch, or dedicated packet taps or brokers. But in Kubernetes, some traffic will never hit a dedicated switch or tap. Kubernetes is used to schedule multiple containers onto the same physical or virtual machine. If one container wants to talk to another container that happens to be on the same machine, then the packets exchanged between them are virtual – they're just bytes in RAM that the operating system shuffles between containers. 

There’s no guarantee that the two containers that you care about will be scheduled onto the same machine, and there’s no guarantee that they won’t beIn fact, if you know two containers are going to want to talk to each other a lot, it’s a good idea to encourage scheduling on the same node for performance: these virtual packets don’t consume any capacity on your switch and advanced techniques can accelerate container-to-container traffic inside a machine. 

Customers that stake their reputation on reliability don’t like mixing “critical tool” and “no guarantee”.  They need to capture traffic right at the edge of the container. That’s what Aspen Mesh Packet Inspector does. It’s built into Carrier-Grade Aspen Mesh, a service mesh purpose built for these critical applications. 

There’s still a problem though – if you are building apps on Kubernetes, you should be encrypting traffic between pods. It’s a best practice that is also required by various standards including those behind 5G.  In the past, capture tools have relied on access to the encryption key to show the decrypted info. New encryption like TLS1.3 has a feature called “forward secrecy” that impedes this. Forward secrecy means every connection is protected with its own temporary key that was securely created by the client and the server – if your tool wasn’t in-the-middle when this key was generated, it’s too late. Access to the server’s encryption key later won't work. 

One approach is to force a broker or tap into the middle for all connections. But that means you need a powerful (i.e. expensive) broker, and it’s a single-point-of-failure. Worse, it’s a security single-point-of-failure: everything in the network has to trust it to get in the middle of all conversations. 

Our users already have something better suited – an Aspen Mesh sidecar (built on Envoy). They’re already using a sidecar next to each container to offload encryption using strong techniques like mutual TLS with forward secrecy. Each sidecar has only one security identity for the particular app container it is protecting, so sidecars can safely authenticate each other without any trusted-box-in-the-middle games. 

That’s the second key part of Aspen Mesh Packet Inspector – because Aspen Mesh is where the plaintext-to-encrypted operation happens (right before leaving the Kubernetes pod), we can record the plaintext. We capture the plaintext and slice it into virtual packets (in a standard “pcap” format). When we feed it to a capture system like a packet broker, we use mutual TLS to protect the captured data.  Our users combine this with a secure packet broker, and get to see the plaintext that was safely and securely transported all the way from the container edge to their screen. 

If you’re a service provider operating Kubernetes at scale, packet tapping capabilities are critical for you to be able to operate the networks effectively, securely and within regulatory and compliance standards. Aspen Mesh Packet Inspector provides the missing link in Kubernetes, providing full packet visibility for troubleshooting and meeting lawful intercept requirements.  


Aspen Mesh 1.5.10-am1 & 1.6.12-am2 Available for Envoy Security Updates

We recently released Aspen Mesh 1.5.10 as well as 1.6.12-am2, which both address important security updates related to an HTTP header security vulnerability that was recently reported in Envoy. We highly recommend our customers to update to these patched versions by following the instructions here.

At Aspen Mesh, we’re dedicated to helping keep upstream open source and cloud-native applications (including Istio and Envoy) as secure and healthy as possible. When this vulnerability was discovered, we were eager to quickly contribute to a fix in order to keep Istio’s (and Aspen Mesh’s) end-users secure.

Envoy HTTP Header Security Vulnerability

Envoy recently reported an incorrect handling of duplicate HTTP headers (CVE-2020-25017). In this CVE, it was reported that Envoy was only considering the first value when multiple values were presented for a non-inline header. The logic would then ignore all following values for a non-iline header when executing matching logic. An attacker could use this exploit as a request policy bypass by including additional non-inline headers with the same key, but with different values, provided that only the first header value matched the routing rule. As a result, client requests could be routed incorrectly and Envoy would pass different headers to the application workloads than what it has in its own internal representation leading to inconsistency. 

In this blog, we will cover test cases and policy examples which our users can deploy to validate if their current version is affected and how this CVE can be exploited by attackers in their environment. 

Test Cases Based on Istio Policies

In this Envoy CVE, there was a vulnerability allowing attackers to set multiple values of a non-inline HTTP headers, such as x-foo:bar and x-foo:baz whereby the affected Envoy components would only observe the first value, x-foo:bar, in matchers, but both x-foo:bar and x-foo:baz would be forwarded to the application workload. Upstreams may take both values into consideration, resulting in an inconsistency between Envoy’s request matching and the upstream view of the request. You can find the complete list of inline HTTP headers in Envoy here for reference.

The Aspen Mesh team worked to develop a solution to trigger and verify the fix for the security vulnerability by working through several test cases. For these test cases, we developed a VirtualService policy resource that specified incoming requests that matched an x-foo header with only a value baz. This would be routed to the httpbin service in the baz namespace. Any other request that did not match that rule would be routed to the httpbin service in the default namespace.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-headers
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - match:
    - headers:
        x-foo:
          exact: baz
    route:
    - destination:
        host: httpbin.baz.svc.cluster.local
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local

 

With this policy deployed, we generated client traffic with different combinations of non-inline headers to trigger this CVE, and the results are summarized in this table below:

Test Case 1.5.7-am5 destination service 1.5.10-am1 destination service
curl httpbin.default:8000/get -H "x-foo: bar" -H "x-foo: baz" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: bar" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" httpbin.baz.svc.cluster.local httpbin.baz.svc.cluster.local
curl httpbin.default:8000/get  httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" -H "x-foo: bar" httpbin.baz.svc.cluster.local httpbin.default.svc.cluster.local

 

We can see in Aspen Mesh version 1.5.7-am5 that the request is routed incorrectly to the httpbin in the baz namespace thereby bypassing configured policy, but in 1.5.10-am1, the same request is routed to thehttpbin service in the default namespace, as we expected. This behavior change, along with the verification of the upstream commits that fixed this issue from upstream Envoy, leads us to believe that our 1.5.10-am1 release contains the required CVE fix. We ran a similar experiment on the affected Aspen Mesh version 1.6.5-am1 and the patched version 1.6.12-am2 to validate that CVE is addressed correctly.

The official remediation fixes these inconsistencies, providing a more secure Aspen Mesh installation with uniform policy enforcement. The newest Aspen Mesh binaries are available for download here.