Aspen Mesh 1.5.10-am1 & 1.6.12-am2 Available for Envoy Security Updates

We recently released Aspen Mesh 1.5.10 as well as 1.6.12-am2, which both address important security updates related to an HTTP header security vulnerability that was recently reported in Envoy. We highly recommend our customers to update to these patched versions by following the instructions here.

At Aspen Mesh, we’re dedicated to helping keep upstream open source and cloud-native applications (including Istio and Envoy) as secure and healthy as possible. When this vulnerability was discovered, we were eager to quickly contribute to a fix in order to keep Istio’s (and Aspen Mesh’s) end-users secure.

Envoy HTTP Header Security Vulnerability

Envoy recently reported an incorrect handling of duplicate HTTP headers (CVE-2020-25017). In this CVE, it was reported that Envoy was only considering the first value when multiple values were presented for a non-inline header. The logic would then ignore all following values for a non-iline header when executing matching logic. An attacker could use this exploit as a request policy bypass by including additional non-inline headers with the same key, but with different values, provided that only the first header value matched the routing rule. As a result, client requests could be routed incorrectly and Envoy would pass different headers to the application workloads than what it has in its own internal representation leading to inconsistency. 

In this blog, we will cover test cases and policy examples which our users can deploy to validate if their current version is affected and how this CVE can be exploited by attackers in their environment. 

Test Cases Based on Istio Policies

In this Envoy CVE, there was a vulnerability allowing attackers to set multiple values of a non-inline HTTP headers, such as x-foo:bar and x-foo:baz whereby the affected Envoy components would only observe the first value, x-foo:bar, in matchers, but both x-foo:bar and x-foo:baz would be forwarded to the application workload. Upstreams may take both values into consideration, resulting in an inconsistency between Envoy’s request matching and the upstream view of the request. You can find the complete list of inline HTTP headers in Envoy here for reference.

The Aspen Mesh team worked to develop a solution to trigger and verify the fix for the security vulnerability by working through several test cases. For these test cases, we developed a VirtualService policy resource that specified incoming requests that matched an x-foo header with only a value baz. This would be routed to the httpbin service in the baz namespace. Any other request that did not match that rule would be routed to the httpbin service in the default namespace.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-headers
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - match:
    - headers:
        x-foo:
          exact: baz
    route:
    - destination:
        host: httpbin.baz.svc.cluster.local
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local

 

With this policy deployed, we generated client traffic with different combinations of non-inline headers to trigger this CVE, and the results are summarized in this table below:

Test Case 1.5.7-am5 destination service 1.5.10-am1 destination service
curl httpbin.default:8000/get -H "x-foo: bar" -H "x-foo: baz" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: bar" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" httpbin.baz.svc.cluster.local httpbin.baz.svc.cluster.local
curl httpbin.default:8000/get  httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" -H "x-foo: bar" httpbin.baz.svc.cluster.local httpbin.default.svc.cluster.local

 

We can see in Aspen Mesh version 1.5.7-am5 that the request is routed incorrectly to the httpbin in the baz namespace thereby bypassing configured policy, but in 1.5.10-am1, the same request is routed to thehttpbin service in the default namespace, as we expected. This behavior change, along with the verification of the upstream commits that fixed this issue from upstream Envoy, leads us to believe that our 1.5.10-am1 release contains the required CVE fix. We ran a similar experiment on the affected Aspen Mesh version 1.6.5-am1 and the patched version 1.6.12-am2 to validate that CVE is addressed correctly.

The official remediation fixes these inconsistencies, providing a more secure Aspen Mesh installation with uniform policy enforcement. The newest Aspen Mesh binaries are available for download here.  


Enterprise Service Mesh

From Middleware to Containers: Infrastructure is Finally Cool

As someone fresh out of school just starting my software engineering career, I want to solve interesting problems. Who doesn’t? A computer science degree gave me the opportunity see a spectrum of different engineering opportunities, which led me to decide that working on infrastructure would be the most impactful area, and with the rise of cloud native technologies, actually a compelling space to work in. There is a difference between developing new functionality and developing to solve existing problems. More often than not, the solutions that address existing challenges in an industry are the ones the are used the most and last the longest. This is what excites me about working on infrastructure, the ability to build something that millions of people will rely on to run their applications. On the surface it doesn’t appear to be the most exciting work, but you can be sure that your time and effort is being put to good use.

You want to see your contributions make an impact somehow, whether that’s writing webapps, iPhone applications, business tools, etc. - the things that people actually use day-to-day. Infrastructure may not be as visible or as tangible as these kinds of technologies, but it’s gratifying to know that it’s the underlying piece that makes it all work. As much as I want to be able to say that I contribute to something that all of my non-tech friends can easily understand (like the front-end of Netflix), I think it’s even more interesting to make them think about the things that happen behind the scenes. We all expect our favorite apps, websites, etc. to be able to respond quickly to our requests no matter how many people are using them at the same time, but on the backend this is not something that is easy to handle and properly test for. What about security? We also expect that when we are trusting software with our information that it isn’t being easily intercepted or leaked along the way. Scalability and security are just two of many kinds of problems that software infrastructure incorporates, and in the end we are relying on them to actually make the front-end software usable. The advantage these days is that infrastructure software has become an incredibly interesting space to be in. Tools like Docker, Kubernetes and Istio are fascinating technologies with vibrant communities around them.

One of the cool, heavily used Kubernetes-related projects that I’m a fan of is Envoy. I can’t help but think about how some version of Envoy is being used every time I order a Lyft to make sure I actually get a ride. Infrastructure doesn’t seem as intriguing at first because as important it is, it’s running in the background and easily forgotten. Everyone needs it, but in the end, who wants to build it? The answer to that question is definitely changing as the infrastructure landscape evolves. Kubernetes, the OS of the cloud, has become a project that everyone wants a hand in. You don’t hear about people itching to make contributions to the Linux kernel, but you hear about Kubernetes and containers everywhere.

Coming up with solutions to solve the problems that we’re running into today has become more attractive to junior developers especially. We’re watching as more and more people are using technology every day, and like I mentioned before, we want our contributions to be impactful. How are we going to handle all of this traffic in a smooth and scalable way? Enter: distributed systems. Microservices are critical to constructing applications that can handle huge transaction volumes at scale. Enterprise applications run by companies like Lyft, Twitter and Google would fall apart with even normal rates of traffic without their distributed architectures. Working on these infrastructural pieces is challenging, and provides the impact that we, junior developers, are looking for.

Another thing that makes this work enticing to junior developers is that it involves an open source community. The way that the tech community has decided to solve some of these bigger, infrastructure-related problems has largely been through open source, which is both intimidating and inviting to those who are new to the tech industry. There is an open group of people talking about the technology and a community willing to help, but at the same time it’s daunting to contribute to these bigger open source projects when you’re just starting out. I will say, however, that the benefits of being able to leverage so many technologies and the community support make it a lot of fun to be a part of.

To recap, here are some of my favorite things about working on infrastructure:

  • We can solve some really hard problems with good infrastructure!
  • If it’s done right, you can build something that can be easily customized to solve problems of various sizes and for all kinds of use cases.
  • All of the cool things and services we consume daily rely on it. Talk about actually seeing your hard work being put to good use!
  • Whether you’re doing proprietary work or not, you are being introduced to open source and the community that comes with it.

I’ll admit, developing infrastructure, despite all of the interesting bits, is still not the most glamorous work. It’s the underlying technology that most people take for granted in their everyday use of technology, and is often less shiny than a beautifully designed UI and other components that sit on top of it. But once you dig in, it’s exciting to see what an impact you can make with it and cloud-native technologies and communities make it a fun space to work in. What I will say though is that it’s a great way to start out your career in tech, and it’s a fun, challenging, and very rewarding place to be.