Pipes and Filters

Purpose

Pipes and filters allow classes to narrow their responsibility to just a small segment of the problem and pass what is left onto the next class. This chain of objects can be very modular such that they can be plugged in anywhere in the stream dynamically as needed.

The filters are the stages and the pipes are the connections between them. The constraint that makes the whole thing work is that every stage speaks the same data type at both ends: it takes one of these and produces one of these. That single shared type is both the price of the pattern and the reason it pays. Once it exists, any stage can follow any other, the order becomes a decision you make at run time, and a new stage is a class rather than an edit.

Design

Use the output of one class to make the input of another in a chain. Beyond that the only real design choice is who drives. In a push pipeline each stage hands its result to the next as soon as it has one. In a pull pipeline each stage asks the previous one for the next item, which is what a chain of lazy iterators is. Between those sits the buffered form, where stages run concurrently with a queue between them, and that is where the interesting problems live: a fast producer in front of a slow consumer needs some way of being told to wait, which is what backpressure means.

Example

Autobot Robotics has some vision processing software. In this software, an image goes through several layers of preprocessing before it is handed to the final decoder.

It did not start that way. The first version was one decode routine of a few hundred lines with flags on it, and the flags multiplied because different plants needed different treatment. A site lit by sodium lamps needs the lighting correction applied before edge detection or the edges come out of the wrong places. A site with printed color codes on the parts needs color normalization last, because normalizing early throws away the contrast the edge pass depends on. Every combination was a boolean, no combination was tested, and the routine had reached the point where changing it for one customer meant guessing about the others.

Splitting it into stages turned the order into data. Each preprocessing step became a class that takes a bitmap and stores the bitmap it produced, and the client assembles them in whichever order that site needs. The stage order is now three lines you can read, and the sodium lamp site differs from the others by one swapped line rather than by a flag threaded through four hundred.

class Decoder
{
    public Bitmap map;
    public Decoder();
    public Decoder(Bitmap map);
}

class LightingDecoder : Decoder
{
    public LightingDecoder(Bitmap map){
        // do processing on map and store it in this.map
    }
}

class ColorDecoder : Decoder
{
    public ColorDecoder(Bitmap map) {
        // do processing on map and store it in this.map
    }
}

class EdgeDecoder : Decoder
{
    public EdgeDecoder(Bitmap map){
        // do processing on map and store it in this.map
    }
}

class Client
{
    Bitmap inMap = new Bitmap();
    public Bitmap process(Bitmap map){
        Decoder result;
        result = new LightingDecoder(inMap);
        result = new EdgeDecoder(result.map);
        result = new ColorDecoder(result.map);
        return result.map;
    }
}

Two things about that code are worth saying out loud. Using the constructor as the processing step is the part I would change now: a stage with an apply(Bitmap) method can be held in a list, configured from a file and reordered without recompiling, and constructors cannot. And process() takes a bitmap and then quietly uses the field instead, which is a slip in the original that I have left alone rather than tidy away.

Seen in the Wild

The canonical case is the Unix shell pipeline, where each program reads standard input and writes standard output and the shell supplies the pipes, which is why grep has never needed to know that sort exists. LLVM's optimiser is a pipeline of passes over a single intermediate representation, and the pass order is a configurable list. GStreamer builds audio and video processing as a graph of elements connected by pads, with buffering and backpressure handled by the framework. Node.js exposes the idea directly as stream.pipe(), and .NET's LINQ and Java's streams are lazy pull pipelines in which each stage requests items from the one before it.

Comparisons

• Decorator is the closest relative and the one worth being careful about. A decorator wraps one object and preserves its interface so the client cannot tell; a filter transforms data and hands it to something else entirely. Decorators compose around an object, filters compose along a stream.
• Chain of Responsibility has the same shape and the opposite expectation. In a chain exactly one handler is supposed to deal with the request; in a pipeline every stage contributes.
• Iterator is what a lazy pull pipeline is built out of. A chain of map and filter over an enumerable is this pattern with iterators as the pipes.
• Strategy is what an individual stage usually is: one swappable algorithm behind a fixed interface.
• Adapter is what you write when two stages do not agree on the type flowing between them, and counting the adapters in a pipeline is a decent measure of how badly the shared type was chosen.

Why It Exists

Criticisms

This could be considered an enterprise, architectural, or a structural design pattern depending on the scale of the project. The same idea describes three classes chained in a function, a set of Unix processes connected by kernel pipes, and a topology of services connected by a message broker, and those have almost nothing in common operationally. That is why nobody agrees where it belongs in a catalog, and it is a fair complaint about the pattern rather than about the people classifying it.

The shared data type is the real weakness. It has to suit every stage, so it drifts towards a lowest common denominator, and then stage four needs something that stage one knew and somebody adds a field to the payload to carry it. Do that three times and the payload is a bag of everything, the stages are quietly coupled through it, and the interchangeability that justified the design is gone while the structure that cost you the copies remains.

Error handling is genuinely awkward, and every pipeline framework has a different unconvincing answer. A function that fails has a stack trace that says where. A pipeline that fails has an item somewhere in the middle, some stages applied and others not, and the questions of which stage failed, what to do with the half-processed item, and whether it can be resumed are all yours to answer. Add concurrency and you inherit backpressure, ordering and partial failure as well. And once a chain is assembled from configuration, the sequence that actually ran exists nowhere in the source, which is the same debugging problem as a deep Decorator stack.