Using D3 in React: A Pattern for Using Data Visualization at Scale

Data visualization is an important part of what we do at Aspen Mesh. When you implement a service mesh, it provides a huge trove of data about your services. Everything you need to know about how your services are communicating is available, but separating the signal from the noise is essential. Data visualization is a powerful tool to distill complex data sets into simple, actionable visuals. To build these visualizations, we use React and D3. React is great for managing a large application and organizing code into discrete components to keep you sane. D3 is magical for visuals on large data sets. Unfortunately, much of the usefulness of each library is lost and bugs are easy to come by when they are not conscientiously put together. In the following article, I will detail the pattern that I work with to build straight forward D3 based visualization components that fit easily within a large scale React application.

Evaluating React + D3 Patterns

The difficulty in putting both libraries together is each has its own way of manipulating the DOM. React through JSX and its Virtual DOM and D3 through .append(). The simplest way to combine them is to let them do their own thing in isolation and act as black boxes to each other. I did not like this approach because it felt like jamming a separate D3 application inside of our existing React app. The code was structured differently, it had to be tested differently and it was difficult to use existing React components and event handlers. I kept researching and playing around with the code until I came on a pattern that addresses those issues. It enables React to track everything in the Virtual DOM and still allows D3 to do what it does best.

Key Aspects:

  • Allow React to handle entering and exiting elements so it can keep track of everything in the Virtual DOM.
  • Code structured and testsed the same way as the rest of the React app.
  • Utilize React lifecycle methods and key attribute to emulate D3 data joins.
  • Manipulate and update element attributes through D3 by selecting the React ref object.
  • D3 for all the tough math. Scales, axes, transitions.

To illustrate the pattern, I will build out a bar graph that accepts an updating data set and transitions between them. The chart has an x axis based on date, and a y axis based on a numerical value. Each data points looks like this:

interface Data {
  id: number;
  date: string;
  value: number;
}

I'll focus on the core components, but to run it and see it all working together, check out the git repo.

SVG Component

The root element to our chart is an SVG. The SVG is rendered through JSX, and subsequent chart elements, such as the axes and the bar elements, are passed in as child components. The SVG is responsible for setting the size and margin and dictating that to its child elements. It also creates scales based on the data and available size and passing the scales down. The SVG component can handle resizing as well as binding D3 panning and zooming functionality. I won't illustrate zooming and panning here, but if you're interested, check out this component. The basic SVG component looks like this.

interface SVGProps {
  svgHeight: number;
  svgWidth: number;
  data: Data[];
}


export default class Svg extends React.Component<SVGProps> {
  render() {
    const { svgHeight, svgWidth, data } = this.props;

    const margin = { top: 20, right: 20, bottom: 30, left: 40 };
    const width = svgWidth - margin.left - margin.right;
    const height = svgHeight - margin.top - margin.bottom;

    const xScale = d3
      .scaleBand()
      .range([0, width])
      .padding(0.1);

    const yScale = d3.scaleLinear().range([height, 0]);

    xScale.domain(data.map(d => d.date));
    yScale.domain([0, d3.max(data, d => d.value) || 0]);

    const axisBottomProps = {
      height,
      scale: xScale
    };
    const axisLeftProps = { scale: yScale };

    const barProps = {
      height,
      width,
      xScale,
      yScale,
      data
    };

    return (
      <svg height={svgHeight} width={svgWidth}>
        <g transform={`translate(${margin.left},${margin.top})`}>
          <AxisBottom {...axisBottomProps} />
          <AxisLeft {...axisLeftProps} />
          <Bars {...barProps} />
        </g>
      </svg>
    );
  }
}

Axes

There are two axis components for the left and bottom axes, and they receive the corresponding D3 scale object as a prop. The React ref object is key to linking up D3 and React. React will render the element, and so keep track of it in the Virtual DOM, and will then pass the ref to D3 so it can manage all the complex attribute math. On componentDidMount call, D3 selects the ref object and then calls the corresponding axis function on it, building the axis. On componentDidUpdate, the axis is redrawn with the updated scale after a D3 transition() to give it a smooth animation. The bottom axis component looks as follows:

interface AxisProps {
  scale: d3.ScaleLinear<any, any>;
}

export default class Axis extends React.Component<AxisProps> {
  ref: React.RefObject<SVGGElement>;

  constructor(props: AxisProps) {
    super(props);
    this.ref = React.createRef();
  }

  componentDidMount() {
    if (this.ref.current) {
      d3.select(this.ref.current).call(d3.axisLeft(this.props.scale));
    }
  }

  componentDidUpdate() {
    if (this.ref.current) {
      d3.select(this.ref.current)
        .transition()
        .call(d3.axisLeft(this.props.scale));
    }
  }

  render() {
    return <g ref={this.ref} />;
  }
}

Rendering with Bars with React Data Joins

The Bars element illustrates how to emulate D3's data join functionality through React lifecycle methods and its key attribute. Data joins allow us to map DOM elements to specific data points and to recognize when those data points enter our set, exit, or are updated by a change in the data set. It is a powerful way to visually represent data constancy between changing data sets. It also allows us to update our chart and only redraw elements that change instead of redrawing the entire graph. Using D3 data joins, with the .enter() or .exit() methods, requires us to append elements through D3 outside of React's Virtual DOM and generally ruins everything. To get around this limitation, we can instead mimic D3 data joins through React's lifecycle methods and its own diffing algorithm. The functions that would be run on .enter() can be executed inside of componentDidMount, updates in componentDidUpdate, and .exit() in componentWillUnmount. Running transitions in componentWillUnmount requires using React Transitions to delay the element from being removed from the DOM until the transition has run. The necessary element for React to map an element to a data point, in this case a bar to a number and a date, is the component's key attribute. By making the key attribute a unique value for each data point, React can recognize through its diffing algorithm if that element needs to be added in, removed, or just updated based on the data point it represents. The key attribute works exactly the same as the key function that would be passed to D3's .data() function.

In this example, two components are created to render the bars on the chart. The first component, Bars, will map over each data point and a render a corresponding Bar component. It binds each data point to the Bar component through the datum prop and assigns a unique key attribute, in this case, the data points unique id.

interface BarsProps {
  data: Data[];
  height: number;
  width: number;
  xScale: d3.ScaleBand<any>;
  yScale: d3.ScaleLinear<any, any>;
}

class Bars extends React.Component<BarsProps> {
  render() {
    const { data, height, width, xScale, yScale } = this.props;
    const barProps = {
      height,
      width,
      xScale,
      yScale
    };
    const bars = data.map(datum => {
      return <Bar key={datum.id} {...barProps} datum={datum} />;
    });
    return <g className="bars">{bars}</g>;
  }
}

The Bar component renders a <rect /> element and passes the ref object to D3 in its lifecycle methods. The lifecycle methods then operate on the element's attributes in familiar D3 dot notation.

interface BarProps {
  datum: Data;
  height: number;
  width: number;
  xScale: d3.ScaleBand<any>;
  yScale: d3.ScaleLinear<any, any>;
}

class Bar extends React.Component<BarProps> {
  ref: React.RefObject<SVGRectElement>;

  constructor(props: BarProps) {
    super(props);
    this.ref = React.createRef();
  }

  componentDidMount() {
    const { height, datum, yScale, xScale } = this.props;

    d3.select(this.ref.current)
      .attr("x", xScale(datum.date) || 0)
      .attr("y", yScale(datum.value) || 0)
      .attr("fill", "green")
      .attr("height", 0)
      .transition()
      .attr("height", height - yScale(datum.value));
  }

  componentDidUpdate() {
    const { datum, xScale, yScale, height } = this.props;
    d3.select(this.ref.current)
      .attr("fill", "blue")
      .transition()
      .attr("x", xScale(datum.date) || 0)
      .attr("y", yScale(datum.value) || 0)
      .attr("height", height - yScale(datum.value));
  }

  render() {
    const { xScale } = this.props;
    const attributes = {
      width: xScale.bandwidth()
    };
    return <rect data-testid="bar" {...attributes} ref={this.ref} />;
  }
}

Testing

By rendering everything through the React Virtual DOM, we can run tests on it with the same setup as we would test our other components. This test setup checks that each data point is represented as a bar in the SVG. Two data points are given intially, and then the component is rerendered with only one of the data points. We test that there are two green bars from the initial mount. Then we test that the update is applied correctly and we only have a single blue bar.

import React from "react";
import "jest-dom/extend-expect";
import { render } from "react-testing-library";
import Svg from "../Svg";

it("renders a bar for each data point", () => {
  const svgHeight = 500;
  const svgWidth = 500;
  const data = [
    { id: 1, date: "9/19/2018", value: 1 },
    { id: 2, date: "11/23/2018", value: 33 }
  ];

  const barProps = {
    svgHeight,
    svgWidth,
    data
  };

  const barProps2 = {
    ...barProps,
    data: [data[0]]
  };

  const { rerender, getAllByTestId, getByTestId } = render(
    <Svg {...barProps} />
  );
  expect(getAllByTestId("bar").length).toBe(2);
  expect(getByTestId("bar")).toHaveAttribute("fill", "green");

  rerender(<Svg {...barProps2} />);

  expect(getAllByTestId("bar").length).toBe(1);
  expect(getByTestId("bar")).toHaveAttribute("fill", "blue");
});

I like this pattern a lot. It fits really nicely into the existing production React app and it allows for recognizable code patterns by encouraging building components for each element in a D3 visualization. It's a smaller learning curve for React developers to building large amounts of D3 and we can use existing display components and event systems within D3 managed visualizations. By allowing D3 to manage the attributes, we can still use advanced features like transitions animations, panning and zooming.

Creating a UI for a service mesh requires managing a lot of complex data and then representing that data in intuitive ways. By combining React and D3 judiciously, we can allow React to do what it does best and manage large application state and then let D3 shine by creating excellent visualizations.

If you want to check out what the final product looks like, check out the Aspen Mesh beta. It's free and easy to sign up for.


Observability, or "Knowing What Your Microservices Are Doing"

Microservicin’ ain’t easy, but it’s necessary. Breaking your monolith down into smaller pieces is a must in a cloud native world, but it doesn’t automatically make everything easier. Some things actually become more difficult. An obvious area where it adds complexity is communications between services; observability into service to service communications can be hard to achieve, but is critical to building an optimized and resilient architecture.

The idea of monitoring has been around for a while, but observability has become increasingly important in a cloud native landscape. Monitoring aims to give an idea of the overall health of a system, while observability aims to provide insights into the behavior of systems. Observability is about data exposure and easy access to information which is critical when you need a way to see when communications fail, do not occur as expected or occur when they shouldn’t. The way services interact with each other at runtime needs to be monitored, managed and controlled. This begins with observability and the ability to understand the behavior of your microservice architecture.

A primary microservices challenges is trying to understand how individual pieces of the overall system are interacting. A single transaction can flow through many independently deployed microservices or pods, and discovering where performance bottlenecks have occurred provides valuable information.

It depends who you ask, but many considering or implementing a service mesh say that the number one feature they are looking for is observability. There are many other features a mesh provides, but those are for another blog. Here, I’m going to cover the top observability features provided by a service mesh.

Tracing

An overwhelmingly important things to know about your microservices architecture is specifically which microservices are involved in an end-user transaction. If many teams are deploying their dozens of microservices, all independently of one another, it’s difficult to understand the dependencies across your services. Service mesh provides uniformity which means tracing is programming-language agnostic, addressing inconsistencies in a polyglot world where different teams, each with its own microservice, can be using different programming languages and frameworks.

Distributed tracing is great for debugging and understanding your application’s behavior. The key to making sense of all the tracing data is being able to correlate spans from different microservices which are related to a single client request. To achieve this, all microservices in your application should propagate tracing headers. If you’re using a service mesh like Aspen Mesh, which is built on Istio, the ingress and sidecar proxies automatically add the appropriate tracing headers and reports the spans to a tracing collector backend. Istio provides distributed tracing out of the box making it easy to integrate tracing into your system. Propagating tracing headers in an application can provide nice hierarchical traces that graph the relationship between your microservices. This makes it easy to understand what is happening when your services interact and if there are any problems.

Metrics

A service mesh can gather telemetry data from across the mesh and produce consistent metrics for every hop. Deploying your service traffic through the mesh means you automatically collect metrics that are fine-grained and provide high level application information since they are reported for every service proxy. Telemetry is automatically collected from any service pod providing network and L7 protocol metrics. Service mesh metrics provide a consistent view by generating uniform metrics throughout. You don’t have to worry about reconciling different types of metrics emitted by various runtime agents, or add arbitrary agents to gather metrics for legacy apps. It’s also no longer necessary to rely on the development process to properly instrument the application to generate metrics. The service mesh sees all the traffic, even into and out of legacy “black box” services, and generates metrics for all of it.

Valuable metrics that a service mesh gathers and standardizes include:

  • Success Rates
  • Request Volume
  • Request Duration
  • Request Size
  • Request and Error Counts
  • Latency
  • HTTP Error Codes

These metrics make it simpler to understand what is going on across your architecture and how to optimize performance.

Most failures in the microservices space occur during the interactions between services, so a view into those transactions helps teams better manage architectures to avoid failures. Observability provided by a service mesh makes it much easier to see what is happening when your services interact with each other, making it easier to build a more efficient, resilient and secure microservice architecture.


Top 3 Reasons to Manage Microservices with Service Mesh


Building microservices is easy, operating a microservice architecture is hard. Many companies are successfully using tools like Kubernetes for deploys, but they still face runtime challenges. This is where the service mesh comes in. It greatly simplifies the managing of containerized applications and makes it easier to monitor and secure microservice-based applications. So what are the top 3 reasons to use a supported service mesh? Here’s my take.

Security

Since service mesh operates on a data plane, it’s possible to apply common security across the mesh which provides much greater security than multilayer environments like Kubernetes. A service mesh secures inter-service communications so you can know what a service is talking to and if that communication can be trusted.

Observability

Most failures in the microservices space occur during the interactions between services, so a view into those transactions helps teams better manage architectures to avoid failures. A service mesh provides a view into what is happening when your services interact with each other. The mesh also greatly improves tracing capabilities and provides the ability to add tracing without touching all of your applications.

Simplicity

A service mesh is not a new technology, rather a bundling together of several existing technologies in a package that makes managing the infrastructure layer much simpler. There are existing solutions that cover some of what a mesh does, take for example DNS. It’s a good way to do service discovery when you don’t care about the source trying to discover the service. If all you need in service discovery is to find the service and connect to it, DNS is sufficient, but it doesn’t give you fast retries or health monitoring. When you want to ask more advanced questions, you need a service mesh. You can cobble things together to address much of what a service mesh addresses, but why would you want to if you could just interact with a service mesh that provides a one-time, reusable packaging?

There are certainly many more advantages to managing microservices with a service mesh, but I think the above 3 are major selling points where organizations that are looking to scale their microservice architecture would find the greatest benefit. No doubt there will also be expanded capabilities in the future such as analytics dashboards that provide easy to consume insights from the huge amount of data in a service mesh. I’d love to hear other ideas you might have on top reasons to use service mesh, hit me up @zjory.