How to Capture Packets that Don't Exist

How to Capture Packets that Don’t Exist

One of my favorite networking tools is Wireshark; it shows you a packet-by-packet view of what’s going on in your network. Wireshark’s packet capture view is the lowest level and most extensive you can get before you have to bust out the oscilloscope. This practice is well-established in the pre-Kubernetes world, but it has some challenges if you’re moving to a Cloud Native environment. If you are using or moving to Cloud Native, you’re going to want to use packet-level tools and techniques in any environment, so that’s why we built Aspen Mesh Packet Inspector. It’s designed to address these challenges across various environments, so you can more easily see what’s going on in your network without the complexity.  

Let me explain the challenges facing our users moving into a Kubernetes world. It’s important to note that there are two parts to a troubleshooting session based on packet capture: actually capturing the packets, and then loading them into your favorite tool. Aspen Mesh Packet Inspector enables users to capture packets even in Kubernetes. That’s the part you need to address to power all the existing tools you probably already have.  Leveraging the existing tools is important as our customers have invested heavily in them.  And not just monetarily – their reputation for reliable apps, services and networks depend on the reliability and usefulness of the tools, procedures and experience powered by a packet view. 

What’s so hard about capturing these packets in modern app architectures on Kubernetes? The two biggest challenges are that these packets may never be actual packets that go through a switch, and even if they were, they’d be encrypted and useless. 

Outside of the Kubernetes world, there are many different approaches to capture packets. You can capture packets right on your PC to debug a local issue. For serious network debugging, you’re usually capturing packets directly on networking hardware, like a monitor port on a switch, or dedicated packet taps or brokers. But in Kubernetes, some traffic will never hit a dedicated switch or tap. Kubernetes is used to schedule multiple containers onto the same physical or virtual machine. If one container wants to talk to another container that happens to be on the same machine, then the packets exchanged between them are virtual – they're just bytes in RAM that the operating system shuffles between containers. 

There’s no guarantee that the two containers that you care about will be scheduled onto the same machine, and there’s no guarantee that they won’t beIn fact, if you know two containers are going to want to talk to each other a lot, it’s a good idea to encourage scheduling on the same node for performance: these virtual packets don’t consume any capacity on your switch and advanced techniques can accelerate container-to-container traffic inside a machine. 

Customers that stake their reputation on reliability don’t like mixing “critical tool” and “no guarantee”.  They need to capture traffic right at the edge of the container. That’s what Aspen Mesh Packet Inspector does. It’s built into Carrier-Grade Aspen Mesh, a service mesh purpose built for these critical applications. 

There’s still a problem though – if you are building apps on Kubernetes, you should be encrypting traffic between pods. It’s a best practice that is also required by various standards including those behind 5G.  In the past, capture tools have relied on access to the encryption key to show the decrypted info. New encryption like TLS1.3 has a feature called “forward secrecy” that impedes this. Forward secrecy means every connection is protected with its own temporary key that was securely created by the client and the server – if your tool wasn’t in-the-middle when this key was generated, it’s too late. Access to the server’s encryption key later won't work. 

One approach is to force a broker or tap into the middle for all connections. But that means you need a powerful (i.e. expensive) broker, and it’s a single-point-of-failure. Worse, it’s a security single-point-of-failure: everything in the network has to trust it to get in the middle of all conversations. 

Our users already have something better suited – an Aspen Mesh sidecar (built on Envoy). They’re already using a sidecar next to each container to offload encryption using strong techniques like mutual TLS with forward secrecy. Each sidecar has only one security identity for the particular app container it is protecting, so sidecars can safely authenticate each other without any trusted-box-in-the-middle games. 

That’s the second key part of Aspen Mesh Packet Inspector – because Aspen Mesh is where the plaintext-to-encrypted operation happens (right before leaving the Kubernetes pod), we can record the plaintext. We capture the plaintext and slice it into virtual packets (in a standard “pcap” format). When we feed it to a capture system like a packet broker, we use mutual TLS to protect the captured data.  Our users combine this with a secure packet broker, and get to see the plaintext that was safely and securely transported all the way from the container edge to their screen. 

If you’re a service provider operating Kubernetes at scale, packet tapping capabilities are critical for you to be able to operate the networks effectively, securely and within regulatory and compliance standards. Aspen Mesh Packet Inspector provides the missing link in Kubernetes, providing full packet visibility for troubleshooting and meeting lawful intercept requirements.  


Aspen Mesh 1.5.10-am1 & 1.6.12-am2 Available for Envoy Security Updates

We recently released Aspen Mesh 1.5.10 as well as 1.6.12-am2, which both address important security updates related to an HTTP header security vulnerability that was recently reported in Envoy. We highly recommend our customers to update to these patched versions by following the instructions here.

At Aspen Mesh, we’re dedicated to helping keep upstream open source and cloud-native applications (including Istio and Envoy) as secure and healthy as possible. When this vulnerability was discovered, we were eager to quickly contribute to a fix in order to keep Istio’s (and Aspen Mesh’s) end-users secure.

Envoy HTTP Header Security Vulnerability

Envoy recently reported an incorrect handling of duplicate HTTP headers (CVE-2020-25017). In this CVE, it was reported that Envoy was only considering the first value when multiple values were presented for a non-inline header. The logic would then ignore all following values for a non-iline header when executing matching logic. An attacker could use this exploit as a request policy bypass by including additional non-inline headers with the same key, but with different values, provided that only the first header value matched the routing rule. As a result, client requests could be routed incorrectly and Envoy would pass different headers to the application workloads than what it has in its own internal representation leading to inconsistency. 

In this blog, we will cover test cases and policy examples which our users can deploy to validate if their current version is affected and how this CVE can be exploited by attackers in their environment. 

Test Cases Based on Istio Policies

In this Envoy CVE, there was a vulnerability allowing attackers to set multiple values of a non-inline HTTP headers, such as x-foo:bar and x-foo:baz whereby the affected Envoy components would only observe the first value, x-foo:bar, in matchers, but both x-foo:bar and x-foo:baz would be forwarded to the application workload. Upstreams may take both values into consideration, resulting in an inconsistency between Envoy’s request matching and the upstream view of the request. You can find the complete list of inline HTTP headers in Envoy here for reference.

The Aspen Mesh team worked to develop a solution to trigger and verify the fix for the security vulnerability by working through several test cases. For these test cases, we developed a VirtualService policy resource that specified incoming requests that matched an x-foo header with only a value baz. This would be routed to the httpbin service in the baz namespace. Any other request that did not match that rule would be routed to the httpbin service in the default namespace.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-headers
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - match:
    - headers:
        x-foo:
          exact: baz
    route:
    - destination:
        host: httpbin.baz.svc.cluster.local
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local

 

With this policy deployed, we generated client traffic with different combinations of non-inline headers to trigger this CVE, and the results are summarized in this table below:

Test Case 1.5.7-am5 destination service 1.5.10-am1 destination service
curl httpbin.default:8000/get -H "x-foo: bar" -H "x-foo: baz" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: bar" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" httpbin.baz.svc.cluster.local httpbin.baz.svc.cluster.local
curl httpbin.default:8000/get  httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" -H "x-foo: bar" httpbin.baz.svc.cluster.local httpbin.default.svc.cluster.local

 

We can see in Aspen Mesh version 1.5.7-am5 that the request is routed incorrectly to the httpbin in the baz namespace thereby bypassing configured policy, but in 1.5.10-am1, the same request is routed to thehttpbin service in the default namespace, as we expected. This behavior change, along with the verification of the upstream commits that fixed this issue from upstream Envoy, leads us to believe that our 1.5.10-am1 release contains the required CVE fix. We ran a similar experiment on the affected Aspen Mesh version 1.6.5-am1 and the patched version 1.6.12-am2 to validate that CVE is addressed correctly.

The official remediation fixes these inconsistencies, providing a more secure Aspen Mesh installation with uniform policy enforcement. The newest Aspen Mesh binaries are available for download here.