Secure Application Deployment with Istio on Kubernetes

Learn How To Securely Deploy Your Application with Istio in this New liveProject from Manning

We are excited to announce that a new liveProject from Manning Publications is now available. A liveProject is a project for you to complete where you can tackle a real-world scenario learning new skills as you go. You’ll develop your own solution and get just enough help to succeed.

In “Secure Application Deployment with Istio on Kubernetes” created by Neeraj Poddar, our Co-Founder and Chief Architect, you will get an overview of Istio’s service mesh, learn how to expose your application to users securely via HTTPS, learn how to protect sensitive user information by utilizing Istio Auth policies, understand application behavior by using Istio telemetry, and efficiently debug your service mesh. 

In addition, you’ll have the opportunity to:

  1. Get hands-on experience of safely and securely exposing an ecommerce microservices-based store using Istio  
  2. Iteratively build in new security features step-by-step, mastering Istio as you go  
  3. Handle deploying your application with Istio and Kubernetes, encrypting traffic to your application via self-managed TLS certificate, understanding application behavior by using Istio telemetry 
  4. Finish up with an efficient debug of your app using the Istio service mesh 
     

Istio service mesh gives you complete visibility over your large scale microservices applications, making it easy to enforce security, manage traffic, spot and debug errors, and improve user experience. 

By the time you’re done with this liveProject, you’ll have the skills to ensure the security of your app, and start making the move away from error-prone manual management of microservices.  

Head over to Manning’s website to get access to this project. 

If you’re looking for even more power for your service mesh, check out what Aspen Mesh adds to Istio. From policy frameworks, an easy to use UI, analytics and alerting as well as first-class support, Aspen Mesh can help your service mesh accomplish even more. 


istiocon 9 trends

Top 9 Takeaways from IstioCon 2021

At the beginning of last year, we predicted the top three developments around service mesh in 2020 would be:

  1. A quickly growing need for service mesh
  2. Istio will be hard to beat
  3. Core service mesh use cases will emerge that will be used as models for the next wave of adopters

And we were right about all three, as evidenced by what we learned at IstioCon.

As a new community-led event, IstioCon 2021 provided the first organized opportunity for Istio’s community members to gather together on a large, worldwide scale, to present, learn and discuss the many features and benefits of the Istio service mesh. And this event was a resounding success.

With over 4,000 attendees — in its first year, and as a virtual event — IstioCon attendance exceeded expectations by multiples. The event showcased the lessons learned from running Istio in production, first-hand experiences from the Istio community, and featured maintainers from across the Istio ecosystem including Lin Sun, John Howard, Christian Posta, Neeraj Poddar, and more. With sessions presented across five days in English, as well as keynotes and sessions in Chinese, this was indeed a worldwide effort. It is well-known that the Istio community reaches far and wide, but it was fantastic to see that so many people interested in, considering, and even using Istio in production at scale were ready to show up and share.

But apart from the outstanding response of the Istio community, we were particularly excited to dig into what people are really using this service mesh for and how they’re interacting with it. So, we’ve pulled together the below curated list of top Istio trends, hot topics, and our top three list of sessions you don’t want to miss.

Top 3 Istio Service Mesh Trends to Watch

After watching each session (so you don’t have to!), we’ve distilled the top three service mesh and Istio industry takeaways that came out of IstioCon that you should keep on your radar.

1. Istio is production-ready. No longer just a shiny new object, this nascent technology has transformed over the past few years from a new infrastructure technology into the microservices management technology that people are using, now, in production and at scale at real companies. We saw insightful user story presentations from T-Mobile, Airbnb, eBay, Salesforce, FICO, and more.

2. Istio is more versatile than you thought. Did you know that Istio is being used right now by users and companies to manage everything from user-facing applications like Airbnb to behind-the-scenes infrastructure like running 5G?

3. Istio and Kubernetes have a lot in common. There are lots of similarities between Istio and Kubernetes in terms of how these technologies have developed, and how they are being adopted. It’s well known that Kubernetes is “the defacto standard for cloud native applications.” Istio is being called ”the most popular service mesh” according to the CNCF annual user survey. But more than this, the two are growing closer together in terms of the technologies themselves. We look forward to the growth of both technologies.

Top 3 Hot Topics

In addition to higher level industry trends, there were many other hot topics that surfaced as part of this conference. From security to Wasm, multicluster, integrations, policies, ORAS, and more, there is a lot going on in the service mesh marketplace that many folks may not have realized. Here are the three hot topics we’d like you to know about:

1. Mulitcluster. You can configure a single mesh to include multiple clusters. Using a multicluster deployment within a single mesh affords capabilities beyond that of a single cluster deployment, including fault isolation and fail over, location-aware routing, various control plane models, and team or project isolation. It was indeed a hot topic at IstioCon, with an entire workshop devoted to Istio Multicluster, plus two additional individual sessions and a dedicated office-hours session about multicluster.

2. Wasm. WebAssembly (Wasm) is a sandboxing technology that can be used to extend the Istio proxy (Envoy). The Proxy-Wasm sandbox API replaces Mixer as the primary extension mechanism in Istio. Over the past year, Wasm has come further to the forefront in terms of interest, as seen here by garnering two sessions plus its own office-hours session.

3. Security. Let’s face it, we’re all concerned about security, and with good reason. Istio has decided to face security challenges head on, and while not exactly a new topic, it’s one worth reiterating. The Istio Product Security Working Group had a session, plus we saw two more sessions featuring security as a headliner, and a dedicated office-hours session. 

Side note: Aspen Mesh had a tie with one another hot topic; debugging Istio. If you get a chance, check out the three recorded sessions on debugging as well.

Top 3 Sessions You Will Want to Watch On-demand

Not everyone has time to watch a conference for five days in a row. And that’s ok. There are about 77 sessions we wish you could watch, but we’ve also identified the top three we think you’ll get the most out of. Check these out:

1. Using Istio to Build the Next Generation 5G Platform. As the most-watched session at this event, we have to start here. In this session, Aspen Mesh’s Co-founder and Chief Architect Neeraj Poddar and David Lenrow, Senior Principal Cloud Security Architect at Verizon, covered what 5G is and why it matters, architecture options with Istio, platform requirements, security, and more.

2. User story from Salesforce - The Salesforce Service Mesh: Our Istio Journey. In this session, Salesforce Software Architect Pratima Nambiar talked us through their background around why they needed a service mesh, their initial implementation, Istio’s value, progressive adoption of Istio, and features they are watching and expect to adopt. 

3. User story from eBay - Istio at Scale: How eBay is Building a Massive Multitenant Service Mesh Using Istio. In this session, Sudheendra Murthy covered eBay’s story, from their applications deployment to service mesh journey, scale testing, and future direction.

What’s Next for Istio?

We were excited to be part of this year’s IstioCon, and it was wonderful to see the Istio community come together for this new event. As our team members have been key contributors to the Istio project over the past few years, we’ve had a front row seat at the growth of the project itself along with the community.

To learn more about what the Istio project has coming up on the horizon, check out this project roadmap session. We’re looking forward to the continued growth of this open source technology, so that more companies — and people — can benefit from what it has to offer.


IstioCon 2021

Aspen Mesh to Sponsor IstioCon 2021

Aspen Mesh is excited to join the Istio community as a Platinum sponsor of the very first conference dedicated to Istio; IstioCon 2021.  

IstioCon is a new, community-led event showcasing the lessons learned from running Istio in production, hands-on experiences from the Istio community, and featuring maintainers from across the Istio ecosystem.  

The conference will offer a mix of keynotes, technical talks, lightning talks, workshops and roadmap sessions. 

We're excited for this new Istio community event and we hope you can join us. You'll find several Aspen Meshers scheduled to give presentations, so be sure to stop by:

  • "Using Istio to Build the Next Generation 5G Platform" with Neeraj Poddar (Aspen Mesh) and David Lenrow (Verizon)
  • "Simple Certificate Management (Pilot-Agent Environmental Variables)" with Jacob Delgado (Aspen Mesh)
  • "Istio Project Roadmap" with Neeraj Poddar (Aspen Mesh) and Louis Ryan (Google)
  • "Istio Product Security Working Group - What It Is and Why It's Important" with Jacob Delgado (Aspen Mesh) and Brian Avery (Red Hat)

Find more information about IstioCon here. 

IstioCon 2021


Aspen Mesh 1.5.10-am1 & 1.6.12-am2 Available for Envoy Security Updates

We recently released Aspen Mesh 1.5.10 as well as 1.6.12-am2, which both address important security updates related to an HTTP header security vulnerability that was recently reported in Envoy. We highly recommend our customers to update to these patched versions by following the instructions here.

At Aspen Mesh, we’re dedicated to helping keep upstream open source and cloud-native applications (including Istio and Envoy) as secure and healthy as possible. When this vulnerability was discovered, we were eager to quickly contribute to a fix in order to keep Istio’s (and Aspen Mesh’s) end-users secure.

Envoy HTTP Header Security Vulnerability

Envoy recently reported an incorrect handling of duplicate HTTP headers (CVE-2020-25017). In this CVE, it was reported that Envoy was only considering the first value when multiple values were presented for a non-inline header. The logic would then ignore all following values for a non-iline header when executing matching logic. An attacker could use this exploit as a request policy bypass by including additional non-inline headers with the same key, but with different values, provided that only the first header value matched the routing rule. As a result, client requests could be routed incorrectly and Envoy would pass different headers to the application workloads than what it has in its own internal representation leading to inconsistency. 

In this blog, we will cover test cases and policy examples which our users can deploy to validate if their current version is affected and how this CVE can be exploited by attackers in their environment. 

Test Cases Based on Istio Policies

In this Envoy CVE, there was a vulnerability allowing attackers to set multiple values of a non-inline HTTP headers, such as x-foo:bar and x-foo:baz whereby the affected Envoy components would only observe the first value, x-foo:bar, in matchers, but both x-foo:bar and x-foo:baz would be forwarded to the application workload. Upstreams may take both values into consideration, resulting in an inconsistency between Envoy’s request matching and the upstream view of the request. You can find the complete list of inline HTTP headers in Envoy here for reference.

The Aspen Mesh team worked to develop a solution to trigger and verify the fix for the security vulnerability by working through several test cases. For these test cases, we developed a VirtualService policy resource that specified incoming requests that matched an x-foo header with only a value baz. This would be routed to the httpbin service in the baz namespace. Any other request that did not match that rule would be routed to the httpbin service in the default namespace.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-headers
spec:
  hosts:
  - httpbin.default.svc.cluster.local
  http:
  - match:
    - headers:
        x-foo:
          exact: baz
    route:
    - destination:
        host: httpbin.baz.svc.cluster.local
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local

 

With this policy deployed, we generated client traffic with different combinations of non-inline headers to trigger this CVE, and the results are summarized in this table below:

Test Case 1.5.7-am5 destination service 1.5.10-am1 destination service
curl httpbin.default:8000/get -H "x-foo: bar" -H "x-foo: baz" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: bar" httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" httpbin.baz.svc.cluster.local httpbin.baz.svc.cluster.local
curl httpbin.default:8000/get  httpbin.default.svc.cluster.local httpbin.default.svc.cluster.local
curl httpbin.default:8000/get -H "x-foo: baz" -H "x-foo: bar" httpbin.baz.svc.cluster.local httpbin.default.svc.cluster.local

 

We can see in Aspen Mesh version 1.5.7-am5 that the request is routed incorrectly to the httpbin in the baz namespace thereby bypassing configured policy, but in 1.5.10-am1, the same request is routed to thehttpbin service in the default namespace, as we expected. This behavior change, along with the verification of the upstream commits that fixed this issue from upstream Envoy, leads us to believe that our 1.5.10-am1 release contains the required CVE fix. We ran a similar experiment on the affected Aspen Mesh version 1.6.5-am1 and the patched version 1.6.12-am2 to validate that CVE is addressed correctly.

The official remediation fixes these inconsistencies, providing a more secure Aspen Mesh installation with uniform policy enforcement. The newest Aspen Mesh binaries are available for download here.  


Helping Istio Sail

Around three years ago, we recognized the power of service mesh as a technology pattern to help organizations manage the next generation of software delivery challenges, and that led to us founding Aspen Mesh. And we know that efficiently managing high-scale Kubernetes applications requires the power of Istio. Having been a part of the Istio project since 0.2, we have seen countless users and customers benefit from the observability, security and traffic management capabilities that a service mesh like Istio provides. 

Our engineering team has worked closely with the Istio community over the past three years to help drive the stability and add new features that make Istio increasingly easy for users to adopt. We believe in the value that open source software — and an open community  —  brings to the enterprise service mesh space, and we are driven to help lead that effort through setting sound technical direction. By contributing expertise, code and design ideas like Virtual Service delegation and enforcing RBAC policies for Istio resources, we have focused much of our work on making Istio more enterprise-ready. One of these contributions includes Istio Vet, which was created and open sourced in early days of Aspen Mesh as a way to enhance Istio's user experience and multi resource configuration validation. Istio Vet proved to be very valuable to users, so we decided to work closely with the Istio community to create istioctl analyze in order to add key configuration analysis capabilities to Istio. It’s very exciting to see that many of these ideas have now been implemented in the community and part of the available feature set for broader consumption. 

As the Istio project and its users mature, there is a greater need for open communication around product security and CVEs. Recognizing this, we were fortunate to be able to help create the early disclosure process and lead the product security working group which ensures the overall project is secure and not vulnerable to known exploits. 

We believe that an open source community thrives when users and vendors are considered as equal partners. We feel privileged that we have been able to provide feedback from our customers that has helped accelerate Istio’s evolution. In addition, it has been a privilege to share our networking expertise from our F5’s heritage with the Istio project as maintainers and leads of important functional areas and key working groups.

The technical leadership of Istio is a meritocracy and we are honored that our sustained efforts have been recognized with - my appointment to the Technical Oversight Committee

As a TOC member, I am excited to work with other Istio community leaders to focus the roadmap on solving customer problems while keeping our user experience top of mind. The next, most critical challenges to solve are day two problems and beyond where we need to ensure smoother  upgrades, enhanced security and scalability of the system. We envision use cases emerging from industries like Telco and FinServ which will  push the envelope of technical capabilities beyond what many can imagine.

It has been amazing to see the user growth and the maturity of the Istio project over the last three years. We firmly believe that a more diverse leadership and an open governance in Istio will further help to advance the project and increase participation from developers across the globe.

The fun is just getting started and I am honored to be an integral part of Istio’s journey! 


Announcing Aspen Mesh 1.5

Announcing Aspen Mesh 1.5

We’re excited to announce the release of Aspen Mesh 1.5 which is based on Istio’s latest LTS release 1.5 (specific tag 1.5.2). Istio 1.5 is the most feature-rich and stable release since Istio 1.1 and contains many enhancements and improvements that we are excited about. With this release, Istio continues to increase stability, performance and usability. At Aspen Mesh, we’re proud of the work that the Istio community has made and are committed to helping it become even better. We continue to represent users’ interests and improve the istio project as maintainers and working group leads for key areas, including Networking and Product Security. It has been amazing to see the growth of the Istio community and we look forward to being a part of its inclusive and diverse culture.

Given the changes in the release, we understand that it can be challenging for users to focus on their key issues and discover how the new features and capabilities relate to their needs while adding business value. We hope this blog will make it easier for our users to digest the sheer volume of changes in Istio 1.5 and understand the reasons why we have, at times, chosen to disable or not enable certain capabilities. We evaluate features and capabilities within Istio from release to release in terms of stability, readiness and backwards compatibility, and we’re eager to enable all capabilities once they are ready for enterprise consumption.

Key updates in Istio 1.5 include: 

  • The networking APIs were moved from v1alpha1 to v1beta1 showing that they are getting closer to stabilizing; 
  • Enabling Envoy to support filters written in WebAssembly (WASM); 
  • The istio operator continues to make major strides in installing and upgrading Istio, as well as istioctl; 
  • A new API model to secure mutual TLS (mTLS) and JWT use in your service mesh.

The number of features added or changed by Istio is long and exciting, so let’s dig deeper into the changes, rationale behind them and how it will impact you as an Aspen Mesh customer.


New Security APIs

Istio 1.5 introduces new security APIs (PeerAuthentication, RequestAuthentication) that work in conjunction with Authorization Policy resources to create a stronger security posture for your applications. We understand that customers have widely adopted Authentication Policies, and the pain of incrementally adopting mTLS as well as debugging issues encountered. We believe that the new set of APIs are a step forward in the evolution of application security. With the older APIs being deprecated in Istio 1.5— and with the intent to remove these APIs in Istio 1.6—it is imperative that customers understand the work required to ease this transition.

To ease migration from the v1alpha1 APIs to the v1beta1 APIs Aspen Mesh has updated our dashboard to reflect mTLS status using the new APIs, we have updated our Secure Ingress API to use the new security APIs and we have updated our configuration generation tool to help you incrementally adopt the new APIs.


Challenges with Older APIs

While Authentication Policies worked for most of the use cases, there were architectural issues with the API choices.

Authentication Policies used to be specified at the service level, but were applied at the workload level. This often caused confusion and potential policy escapes as all other Istio resources are specified and applied at the workload level. The new API addresses this concern and is consistent with the Istio configuration model.

Authentication Policies used to handle both end user and peer authentication causing confusion. The new API separates them out as they are different concerns that solve different sets of problems. This logical separation should make it easier to secure your application as the possibility of misconfiguration is reduced. For example, when configuring mTLS if there is a conflict between multiple PeerAuthentication resources, the strongest PeerAuthentication (by security enforcement) is applied. Previously conflict resolution was not defined and behavior wasn’t deterministic.

Authentication Policies used to validate end user JWT credentials and reject any request that triggered the rule and had no or invalid/expired JWT tokens. The new API separates the Authentication and Authorization pieces into two separate composable layers. This has the benefit of expressing all allow/deny policies in a single place, i.e. Authorization resources. The downside of this approach is that if a RequestAuthentication policy is applied, but no JWT token is presented, then the request is allowed unless there's a deny Authorization policy. However, separating authentication/identity and authorization follows traditional models of securing systems, with respect to access control. If you are currently using Authentication policies to reject requests without any JWT tokens, you will need to add a deny Authorization policy for achieving parity. While Istio has greatly simplified their security model, our Secure Ingress API provides a higher level abstraction for the typical use case. Ease of adoption is one of our core tenets and we believe that Secure Ingress does just that. Feel free to reach out to us with questions about how to do this.


Migration Towards New APIs

You don’t have to worry about resources defined by the existing Authentication Policy API as they will continue to work in Istio 1.5. Aspen Mesh supports the direction of the Istio community as it trends towards using workload selectors throughout its APIs. By adopting these new APIs we are hopeful that it will alleviate some application security concerns. 

With the removal of the v1alpha1 APIs in istio 1.6 (ETA June 2020) we are focused on helping our customers migrate towards using the new APIs. Our Secure Ingress Policy has already been ported to support the new APIs, as well as our tool to help you incrementally adopt mTLS. Note that our Secure Ingress controller preserves backward compatibility by creating both RequestAuthentication and Authorization policies as needed to ensure paths protected by JWT tokens cannot be accessed without tokens.

Users upgrading to Aspen Mesh 1.5.2 will see their MeshPolicy replaced with a mesh-wide PeerAuthentication resource. This was intentional, although upstream Istio chose not to remove a previously installed MeshPolicy, so that users can quickly migrate their cluster towards using the new security APIs. With that in mind, istioctl now includes analyzers to help you find and manage your MeshPolicy/Policy resources, giving you a deprecation message with instructions to migrate away from your existing policies.


Extending Envoy Through WASM

WASM support, as an idea, can be traced back to October 2018. A proxy ABI specification for WASM was also drafted with the idea that all proxies, and not just Envoy, can adopt WASM helping to further cement the Istio community as thought leaders in the service mesh industry. The Istio community also had major contributions to Envoy through the envoy-wasm repository to enable WASM support and is the fork of Envoy used in Istio 1.5. There are plans to have their hard work incorporated into the mainline Envoy repository.

This is a major feature as it will enable users to extend Envoy in new and exciting ways.

We applaud the Istio community for its work to deliver this capability. Previously writing a filter for Envoy required extensive knowledge of not only Envoy C++ code, but also how to build it, integrate it and deploy Envoy.

WebAssembly support in Envoy will allow users to create, share and extend Envoy to meet their needs. Support for creating Envoy extensions already exists with SDKs for C++, Rust, and AssemblyScript, and there are plans to increase the number of supported languages shortly. Istio is also continuing to build its supporting infrastructure around these Envoy features to easily integrate them into the Istio platform. There is also on-going work to move already existing Istio filters to WASM; in fact, Telemetry V2 is a WASM extension. Istio is eating its own dog food and the results speak for themselves. Expect future blogs around WASM capabilities in Istio as we learn more about its feature set and capabilities.

For more information, please see Istio’s WASM announcement


istiod and Helm Based Installs

Support for Helm v3-based installs continues to be a top priority for Aspen Mesh. Aspen Mesh Helm installation process is a seamless operation for your organization. Aspen Mesh 1.5 does not support the istiod monolith yet, but continues to ship with Istio’s microservice platform.

We are actively working with the Istio community to enable installations and upgrades from a microservice platform to the monolithic platform a first-class experience with minimal downtime in istio 1.6. This will ensure that our customers will have a stable and qualified release when they migrate to istiod.

As Istio has evolved, a decision was made to simplify Istio and transform its microservice platform to that of a monolith for operational simplification and overhead reduction. The upgrade path from microservices control plane to monolith (istiod) requires a fresh install and possible downtime. We want our customers to have a smooth upgrade and experience minimal downtime disruption.


Moving Towards a Simplified Operational System (istiod)

Benefits of a monolithic approach to end users, include a reduction in installation complexity and fewer services deployed which makes it easier for users to understand what is being deployed in their Kubernetes environment, especially as the intra-communication within the controlplane and the inter-communication between the control plane and data plane continues to grow in complexity. 

With the migration towards istiod, one of the key features that we anticipate will mature in istio 1.6 is the istio in-cluster Operator. In-cluster operators have quickly gained traction within the Kubernetes community and the Istio community has made a significant effort towards managing the lifecycle of Istio. A key feature that we are eager about is the ability to have multiple control planes, which would enable such things as canaries and potentially in-place upgrades. As the in-cluster operator continues to evolve we will continue to evaluate it and when we feel it is ready for enterprise use you should expect a more detailed blog to follow.


Other Features on the Horizon

As always, Istio and Aspen Mesh are constantly moving forward. Here is a preview of some new features we see on the horizon.


Mixer-less Telemetry

Many users have inquired about mixer-less telemetry, hence referred to as Telemetry V2. During testing, Aspen Mesh found an issue with TCP blackhole telemetry, others found an issue with HTTP blackhole telemetry and a few other minor issues were found. Aspen Mesh is committed to helping the Istio community test and fix issues found in Telemetry V2, but until feature parity is reached with Mixer telemetry we will continue to ship with Mixer, aka Telemetry V1. This is perhaps our most requested feature and we are eager to enable it. The work surrounding this has been a Herculean effort by the Istio community and we are eager to enable it in a future Aspen Mesh release.


Intelligent Inbound and Outbound Protocol Detection

While upstream Istio has previously enabled outbound intelligent protocol detection, Aspen Mesh has decided to disable it; the reasoning for this was detailed in our Protocol sniffing in production blog. For Istio 1.5, inbound intelligent protocol detection was also enabled. While this has the potential to be a powerful feature, Aspen Mesh believes that deterministic behavior is best defined by our end users. There have been a handful of security exploits related to protocol sniffing and we will continue to monitor its stability and performance implications and accordingly enable them if warranted in future releases.

For this and prior releases, we recommend our users continue to prefix the port name with the appropriate supported protocol as suggested here. You can also use our vetter to discover any missing port prefixes in your service mesh.


Next Steps for You

The Aspen Mesh 1.5 binaries are available for download here. If you have any questions about your upgrade path, feel free to reach out to us at support@aspenmesh.io.

If you’re new to Aspen Mesh, you can download a free 30-day trial here


Aspen Mesh - Service Mesh Security and Complinace

Announcing Aspen Mesh 1.4.6 Security Update

Aspen Mesh is announcing the release of 1.4.6 which addresses important Istio security updates. Below are the details of the security fixes taken from Istio 1.4.6 security update.

Security Update: 

ISTIO-SECURITY-2020-003: Two Uncontrolled Resource Consumption and Two Incorrect Access Control Vulnerabilities in Envoy.

  • CVE-2020-8659: The Envoy proxy may consume excessive memory when proxying HTTP/1.1 requests or responses with many small (i.e. 1 byte) chunks. Envoy allocates a separate buffer fragment for each incoming or outgoing chunk with the size rounded to the nearest 4Kb and does not release empty chunks after committing data. Processing requests or responses with a lot of small chunks may result in extremely high memory overhead if the peer is slow or unable to read proxied data. The memory overhead could be two to three orders of magnitude more than configured buffer limits.
  • CVE-2020-8660: The Envoy proxy contains a TLS inspector that can be bypassed (not recognized as a TLS client) by a client using only TLS 1.3. Because TLS extensions (SNI, ALPN) are not inspected, those connections may be matched to a wrong filter chain, possibly bypassing some security restrictions.
  • CVE-2020-8661: The Envoy proxy may consume excessive amounts of memory when responding to pipelined HTTP/1.1 requests. In the case of illegally formed requests, Envoy sends an internally generated 400 error, which is sent to the Network::Connection buffer. If the client reads these responses slowly, it is possible to build up a large number of responses, and consume functionally unlimited memory. This bypasses Envoy’s overload manager, which will itself send an internally generated response when Envoy approaches configured memory thresholds, exacerbating the problem.
  • CVE-2020-8664: For the SDS TLS validation context in the Envoy proxy, the update callback is called only when the secret is received for the first time or when its value changes. This leads to a race condition where other resources referencing the same secret (e.g,. trusted CA) remain unconfigured until the secret’s value changes, creating a potentially sizable window where a complete bypass of security checks from the static (“default”) section can occur.
    • This vulnerability only affects the SDS implementation of Istio’s certificate rotation mechanism for Istio 1.4.5 and earlier which is only when SDS and mutual TLS are enabled. SDS is off by default and must be explicitly enabled by the operator in all versions of Istio prior to Istio 1.5. Istio’s default secret distribution implementation based on Kubernetes secret mounts is not affected by this vulnerability.

Bug Fix:

  • Fixed issue preventing Kubernetes nodes from being restarted

Minor Enhancements:

  • Allow private image registries to be specified more easily
  • Better checking of problematic names in Secure Ingress
  • Better mTLS status reporting
  • Improved classification of vetter notes

If you're already using Aspen Mesh, you can get the updates here.

 

New call-to-action
Aspen Mesh - Service Mesh Security and Complinace

Aspen Mesh 1.4.4 & 1.3.8 Security Update

Aspen Mesh is announcing the release of 1.4.4 and 1.3.8 (based on upstream Istio 1.4.4 & 1.3.8), both addressing a critical security update. All our customers are strongly encouraged to upgrade to these versions immediately based on your currently deployed Aspen Mesh version.

The Aspen Mesh team reported this CVE to the Istio community as per Istio CVE reporting guidelines, and our team was able to further contribute to the community--and for all Istio users--by fixing this issue in upstream Istio. As this CVE is extremely easy to exploit and the risk score was deemed very high (9.0), we wanted to respond  with urgency by getting a patch in upstream Istio and out to our customers as quickly as possible. This is just one of the many ways we are able to provide value to our customers as a trusted partner by ensuring that the pieces from Istio (and Aspen Mesh) are secure and stable.

Below are details about the CVE-2020-8595, steps to verify whether you’re currently vulnerable, and how to upgrade to the patched releases.

CVE Description

A bug in Istio's Authentication Policy exact path matching logic allows unauthorized access to resources without a valid JWT token. This bug affects all versions of Istio (and Aspen Mesh) that support JWT Authentication Policy with path based trigger rules (all 1.3 & 1.4 releases). The logic for the exact path match in the Istio JWT filter includes query strings or fragments instead of stripping them off before matching. This means attackers can bypass the JWT validation by appending “?” or “#” characters after the protected paths.

Example Vulnerable Istio Configuration

Configure JWT Authentication Policy triggers on an exact HTTP path match "/productpage" like this. In this example, the Authentication Policy is applied at the ingress gateway service so that any requests with the exact path matching “/productpage” requires a valid JWT token. In the absence of a valid JWT token, the request is denied and never forwarded to the productpage service. However, due to this CVE, any request made to the ingress gateway with path "/productpage?" without any valid JWT token is not denied but sent along to the productpage service, thereby allowing access to a protected resource without a valid JWT token. 

Validation

Since this vulnerability is in the Envoy filter added by Istio, you can also check the proxy image you’re using in the cluster locally to see if you’re currently vulnerable. Download and run this script to verify if the proxy image you’re using is vulnerable. Thanks to Francois Pesce from the Google Istio team in helping us create this test script.

Upgrading to This Version

Please follow upgrade instructions in our documentation to upgrade to these versions. Since this vulnerability affects the sidecar proxies and gateways (ingress and egress), it is important to follow the post-upgrade tasks here and rolling upgrade all of your sidecar proxies and gateways.


Announcing Aspen Mesh 1.4 Service Mesh

Announcing Aspen Mesh 1.4

We’re excited to announce the release of Aspen Mesh 1.4 which is based on Istio’s latest LTS release 1.4 (specific tag 1.4.3). The Aspen Mesh 1.4.3 release builds upon our latest 1.3 release along with our new Secure Ingress API, that we will be writing about in detail in a blog soon to follow, alongside the new capabilities of Istio 1.4.

Our Open Source Commitment

Aspen Mesh has been heavily involved with the Istio community for this release. A core value of Aspen Mesh is to help drive adoption of Istio. Our Istio Vet tool was the the first tool created to help analyze configuration issues and it is great to have this capability within the core open source project by using istioctl analyze. Aspen Mesh is committed to helping our customers and the larger Istio community as we will continue to add analyzers in the future based on issues that arise.

For 1.4, we are also pleased that we could help the community create istio-client-go, allowing the open source community to programmatically use and extend the Istio APIs in their software. We have deprecated the widely adopted Aspen Mesh version of our istio-client-go in favor of the the community’s version.

Automatic mTLS

Automatic mTLS is a feature that Aspen Mesh is eager about enabling in the future, but for now is disabled in upstream Istio as it is in our supported release. We are strong advocates of using mTLS for service-to-service communication, but users adopting Istio will sometimes encounter issues and will need to debug mTLS in their service mesh.

Mixer-less Telemetry

Mixer-less telemetry moves from an experimental feature into alpha in Istio 1.4. Aspen Mesh is working with the open source community to ensure feature parity with Mixer during this state of transition, however, we continue to support Istio Mixer at this time. Please see our announcement of Aspen Mesh 1.3 to see the importance of this feature.

Istio Authorization Policies

With policy being a key reason why our customers are adopting Istio, the new Authorization Policy is a step towards simplifying Policy. Istio continues to improve their APIs to better achieve a simplified model of service mesh configuration. Stay tuned for future blogs as Aspen Mesh is a strong proponent of this model.

istioctl

Along with the new analysis capabilities referenced above, istioctl also has the capability to help users with installation and upgrade along with a new experimental capability to wait for configuration to be passed to all proxies within your service mesh. At this time, we believe Helm provides the enterprise with the best way to manage deploying Istio into production, but we are always exploring and evaluating new approaches to managing your Istio deployment.

Analyzers included in Istio 1.4 help ensure that Gateways are configured properly, features that are being deprecated in Istio that are being used are given a warning message, and that all pods containing sidecars have the same, expected version of Envoy running. These are only some of the analyzers included and more information can be found here. Some of the work from Aspen Mesh’s Istio Vet was able to be ported to istioctl with the community’s help. We intend to continue porting any capabilities found in istio-vet into istioctl.

The Aspen Mesh 1.4 binaries are available for download here


Aspen Mesh - Service Mesh Security and Complinace

Aspen Mesh 1.3.6 Security Update

Aspen Mesh is announcing the release of 1.3.6 which addresses important Istio security updates. Below are the details of the security fixes taken from Istio 1.3.6 security update.

Security Update: 

ISTIO-SECURITY-2019-007: A heap overflow and improper input validation have been discovered in Envoy.

  • CVE-2019-18801: Fix a vulnerability affecting Envoy’s processing of large HTTP/2 request headers. A successful exploitation of this vulnerability could lead to a denial of service, escalation of privileges, or information disclosure.
  • CVE-2019-18802: Fix a vulnerability resulting from whitespace after HTTP/1 header values which could allow an attacker to bypass Istio’s policy checks, potentially resulting in information disclosure or escalation of privileges.

Bug Fix:

  • Fixed an issue where a duplicate listener was generated for a proxy’s IP address when using a headless TCP service. (Issue 17748)
  • Fixed an issue with the destination_service label in HTTP related metrics incorrectly falling back to request.host which can cause a metric cardinality explosion for ingress traffic. (Issue 18818)

Minor Enhancements:

  • Added support for Citadel to periodically check the root certificate remaining lifetime and rotate expiring root certificates. (Issue 17059)
  • Added PILOT_BLOCK_HTTP_ON_443 boolean environment variable to Pilot. If enabled, this flag prevents HTTP services from running on port 443 in order to prevent conflicts with external HTTP services. This is disabled by default. (Issue 16458)

Additionally, the Aspen Mesh 1.3.5 release contains bug fixes and minor enhancements from Istio release 1.3.5.  

The Aspen Mesh 1.3.6 binaries are available for download here.