Get The Fully Supported Service Mesh

Aspen Mesh provides the observability, security and scalability you need to operate service mesh in the enterprise, all in an easy-to-use package

 

Submit Your Resume

Upload your resume. (5 MB max - .pdf, .doc, or .docx)

August 21, 2019

Why Is Policy Hard?

 

Aspen Mesh spends a lot of time talking to users about policy, even if we don’t always start out calling it that. A common pattern we see with clients is:

  1. Concept: “Maybe I want this service mesh thing”
  2. Install: “Ok, I’ve got Aspen Mesh installed, now what?”
  3. Observe: “Ahhh! Now I see how my microservices are communicating.  Hmmm, what’s that? That pod shouldn’t be talking to that database!”
  4. Act: “Hey mesh, make sure that pod never talks to that database”

The Act phase is interesting, and there’s more to it than might be obvious at first glance. I’ll propose that in this blog, we work through some thought experiments to delve into how service mesh can help you act on insights from the mesh.

First, put yourself in the shoes of the developer that just found out their test pod is accidentally talking to the staging database. (Ok, you’re working from home today so you don’t have to put on shoes; the cat likes sleeping on your shoeless feet better anyways.) You want to control the behavior of a narrow set of software for which you’re the expert; you have local scope and focus.

Next, put on the shoes of a person responsible for operating many applications; people we talk to often have titles that include Platform, SRE, Ops, Infra. Each day they’re diving into different applications so being able to rapidly understand applications is key. A consistent way of mapping across applications, datacenters, clouds, etc. is critical. Your goal is to reduce “snowflake architecture” in favor of familiarity to make it easier when you do have to context switch.

Now let’s change into the shoes of your org’s Compliance Officer. You’re on the line for documenting and proving that your platform is continually meeting compliance standards. You don’t want to be the head of the “Department of No”, but what’s most important to you is staying out of the headlines. A great day at work for you is when you’ve got clarity on what’s going on across lots of apps, databases, external partners, every source of data your org touches AND you can make educated tradeoffs to help the business move fast with the right risk profile. You know it’s ridiculous to be involved in every app change, so you need separation-of-concerns.

I’d argue that all of these people have policy concerns. They want to be able to specify their goals at a suitably high level and leave the rote and repetitive portions to an automated system.  The challenging part is there’s only one underlying system (“the kubernetes cluster”) that has to respond to each of these disparate personas.

So, to me policy is about transforming a bunch of high-level behavioral prescriptions into much lower-level versions through progressive stages. Useful real-world policy systems do this in a way that is transparent and understandable to all users, and minimizes the time humans spend coordinating. Here’s an example “day-in-the-life” of a policy:

At the top is the highest level goal: “Devs should test new code without fear”. Computers are hopeless to implement this. At the bottom is a rule suitable for a computer like a firewall to implement.

The layers in the middle are where a bad policy framework can really hurt. Some personas (the hypothetical Devs) want to instantly jump to the bottom – they’re the “4.3.2.1” in the above example. Other personas (the hypothetical Compliance Officer) is way up top, going down a few layers but not getting to the bottom on a day-to-day basis.

I think the best policy frameworks help each persona:

  • Quickly find the details for the layer they care about right now.
  • Help them understand where did this come from? (connect to higher layers)
  • Help them understand is this doing what I want? (trace to lower layers)
  • Know where do I go to change this? (edit/create policy)

As an example, let’s look at iptables, one of the firewalling/packet mangling frameworks for Linux.  This is at that bottom layer in my example stack – very low-level packet processing that I might look at if I’m an app developer and my app’s traffic isn’t doing what I’d expect.  Here’s an example dump:


root@kafka-0:/# iptables -n -L -v --line-numbers -t nat
Chain PREROUTING (policy ACCEPT 594K packets, 36M bytes)
num   pkts bytes target     prot opt in out   source destination
1     594K 36M ISTIO_INBOUND  tcp -- * * 0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT 594K packets, 36M bytes)
num   pkts bytes target     prot opt in out   source destination

Chain OUTPUT (policy ACCEPT 125K packets, 7724K bytes)
num   pkts bytes target     prot opt in out   source destination
1      12M 715M ISTIO_OUTPUT  tcp -- * * 0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 12M packets, 715M bytes)
num   pkts bytes target     prot opt in out   source destination

Chain ISTIO_INBOUND (1 references)
num   pkts bytes target     prot opt in out   source destination
1        0 0 RETURN     tcp -- * *     0.0.0.0/0 0.0.0.0/0            tcp dpt:22
2     594K 36M RETURN     tcp -- * *   0.0.0.0/0 0.0.0.0/0            tcp dpt:15020
3        2 120 ISTIO_IN_REDIRECT  tcp -- * * 0.0.0.0/0            0.0.0.0/0

Chain ISTIO_IN_REDIRECT (1 references)
num   pkts bytes target     prot opt in out   source destination
1        2 120 REDIRECT   tcp -- * *     0.0.0.0/0 0.0.0.0/0            redir ports 15006

Chain ISTIO_OUTPUT (1 references)
num   pkts bytes target     prot opt in out   source destination
1      12M 708M ISTIO_REDIRECT  all -- * lo 0.0.0.0/0           !127.0.0.1
2        7 420 RETURN     all -- * *     0.0.0.0/0 0.0.0.0/0            owner UID match 1337
3        0 0 RETURN     all -- * *     0.0.0.0/0 0.0.0.0/0            owner GID match 1337
4     119K 7122K RETURN     all -- * *   0.0.0.0/0 127.0.0.1
5        4 240 ISTIO_REDIRECT  all -- * * 0.0.0.0/0            0.0.0.0/0

Chain ISTIO_REDIRECT (2 references)
num   pkts bytes target     prot opt in out   source destination
1      12M 708M REDIRECT   tcp -- * *   0.0.0.0/0 0.0.0.0/0            redir ports 15001


This allows me to quickly understand a lot of details about what is happening at this layer. Each rule specification is on the right-hand side and is relatively intelligible to the personas that operate at this layer. On the left, I get “pkts” and “bytes” – this is a count of how many packets have triggered each rule, helping me answer “Is this doing what I want it to?”. There’s even more information here if I’m really struggling: I can log the individual packets that are triggering a rule, or mark them in a way that I can capture them with tcpdump.  

Finally, furthest on the left in the “num” column is a line number, which is necessary if I want to modify or delete rules or add new ones before/after; this is a little bit of help for “Where do I go to change this?”. I say a little bit because in most systems that I’m familiar with, including the one I grabbed that dump from, iptables rules are produced by some program or higher-level system; they’re not written by a human. So if I just added a rule, it would only apply until that higher-level system intervened and changed the rules (in my case, until a new Pod was created, which can happen at any time). I need help navigating up a few layers to find the right place to effect the change.

iptables lets you organize groups of rules into your own chains, in this case the name of the chain (ISTIO_***) is a hint that Istio produced this and so I’ve got a hint on what higher layer to examine.

For a much different example, how about the Kubernetes CI Robot (from Prow)? If you’ve ever made a PR to Kubernetes or many other CNCF projects, you likely interacted with this robot. It’s an implementer of policy; in this case the policies around changing source code for Kubernetes.  One of the policies it manages is compliance with the Contributor’s License Agreement; contributors agree to grant some Intellectual Property rights surrounding their contributions. If k8s-ci-robot can’t confirm that everything is alright, it will add a comment to your PR:

This is much different than firewall policy, but I say it’s still policy and I think the same principles apply. Let’s explore. If you had to diagram the policy around this, it would start at the top with the legal principle that Kubernetes wants to make sure all the software under its umbrella has free and clear IP terms. Stepping down a layer, the Kubernetes project decided to satisfy that requirement by requiring a CLA for any contributions. So on until we get to the bottom layer, the code that implements the CLA check.

As an aside, the code that implements the CLA check is actually split into two halves: first there’s a CI job that actually checks the commits in the PR against a database of signed CLAs, and then there’s code that takes the result of that job and posts helpful information for users to resolve any issues. That’s not visible or important at that top layer of abstraction (the CNCF lawyers shouldn’t care).

This policy structure is easy to navigate. If your CLA check fails, the comment from the robot has great links. If you’re an individual contributor you can likely skip up a layer, sign the CLA and move on. If you’re contributing on behalf of a company, the links will take you to the document you need to send to your company’s lawyers so they can sign on behalf of the company.

So those are two examples of policy. You probably encounter many other ones every day from corporate travel policy to policies (written, unwritten or communicated via email missives) around dirty dishes.

It’s easy to focus on the technical capabilities of the lowest levels of your system. But I’d recommend that you don’t lose focus on the operability of your system. It’s important that it be transparent and easy to understand. Both the iptables and k8s-ci-robot are transparent. The k8s-ci-robot has an additional feature: it knows you’re probably wondering “Where did this come from?” and it answers that question for you. This helps you and your organization navigate the layers of policy. 

When implementing service mesh to add observability, resilience and security to your Kubernetes clusters, it’s important to consider how to set up policy in a way that can be navigated by your entire team. With that end in mind, Aspen Mesh is building a policy framework for Istio that makes it easy to implement policy and understand how it will affect application behavior.

Did you like this blog? Subscribe to get email updates when new Aspen Mesh blogs go live.

Leave a Reply

Your email address will not be published. Required fields are marked *