Get The Fully Supported Service Mesh

Aspen Mesh provides the observability, security and scalability you need to operate service mesh in the enterprise, all in an easy-to-use package

 

Submit Your Resume

Upload your resume. (5 MB max - .pdf, .doc, or .docx)

August 9, 2019

Recommended Remediation for Kubernetes CVE-2019-11247

 

A Kubernetes vulnerability CVE-2019-11247 was announced this week.  While this is a vulnerability in Kubernetes itself, and not Istio, it may affect Aspen Mesh users. This blog will help you understand possible consequences and how to remediate.

  • You should mitigate this vulnerability by updating to Kubernetes 1.13.9, 1.14.5 or 1.15.2 as soon as possible.
  • This vulnerability affects the interaction of Roles (a Kubernetes RBAC resource) and CustomResources (Istio uses CustomResources for things like VirtualServices, Gateways, DestinationRules and more).
  • Only certain definitions of Roles are vulnerable.
  • Aspen Mesh’s installation does not define any such Roles, but you might have defined them for other software in your cluster.
  • More explanation and recovery details below.

Explanation

If you have a Role defined anywhere in your cluster with a “*” for resources or apiGroups, then anything that can use that Role could escalate to modify many CustomResources.

This Kubernetes issue and this helpful blog have extensive details and an example walkthrough that we’ll summarize here.  Kubernetes Roles define sets of permissions for resources in one particular namespace (like “default”).  They are not supposed to define permissions for resources in other namespaces or resources that live globally (outside of any namespace); that’s what ClusterRoles are for.

Here’s an example Role from Aspen Mesh that says “The IngressGateway should be able to get, watch or list any secrets in the istio-system namespace”, which it needs to bootstrap secret discovery to get keys for TLS or mTLS:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: istio-ingressgateway-sds
  namespace: istio-system
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

This Role does not grant permissions to secrets in any other namespace, or grant permissions to anything aside from secrets.   You’d need additional Roles and RoleBindings to add those. It doesn’t grant permissions to get or modify cluster-wide resources.  You’d need ClusterRoles and ClusterRoleBindings to add those.

The vulnerability is that if the Role grants access to CustomResources in one namespace, it accidentally grants access to the same kinds of CustomResources that exist at global scope.  This is not exploitable in many cases because if you have a namespace-scoped CustomResource called “Foo”, you can’t also have a global CustomResource called “Foo”, so there are no global-scope Foos to attack.  Unfortunately, if your role allows access to a resource “*” or apiGroup “*”, then the “*” matches both namespace-scoped Foo and globally-scoped Bar in vulnerable versions of Kubernetes. This Role could be used to attack Bar.

If you’re really scrutinizing the above example, note that the apiGroup “” is different than the apiGroup “*”: the empty “” refers to core Kubernetes resources like Secrets, while “*” is a wildcard meaning any.

Aspen Mesh and Istio define three globally-scoped CustomResources: ClusterRbacConfig, MeshPolicy, ClusterIssuer.  If you had a vulnerable Role defined and an attacker could assume that Role, then they could have modified those three CustomResources.  Aspen Mesh does not provide any vulnerable Roles so a cluster would need to have those Roles defined for some other purpose.

Recovery

First and as soon as possible, you should upgrade Kubernetes to a non-vulnerable version: 1.13.9, 1.14.5 or 1.15.2.

When you define Roles and ClusterRoles, you should follow the Principle of Least Privilege and avoid granting access to “*” – always prefer listing the specific resources required.

You can examine your cluster for vulnerable Roles.  If any exist, or have existed, those could have been exploited by an attacker with knowledge of this vulnerability to modify Aspen Mesh configuration.  To mitigate this, recreate any CustomResources after upgrading to a non-vulnerable version of Kubernetes.

This snippet will print out any vulnerable Roles currently configured, but cannot tell you if any may have existed in the past.  It relies on the jq tool:

kubectl get role --all-namespaces -o json |jq '.items[] | select(.rules[].resources | index("*"))'
kubectl get role --all-namespaces -o json |jq '.items[] | select(.rules[].apiGroups | index("*"))'

This is not a vulnerability in Aspen Mesh, so there is no need to upgrade Aspen Mesh.

Leave a Reply

Your email address will not be published. Required fields are marked *