Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Sequences and Laziness

Transducers

Learn Clojure transducers: composable, reusable transformations that avoid intermediate sequences. Covers map, filter, comp, transduce, into, eduction and early exit.

A normal sequence pipeline such as (->> xs (filter odd?) (map inc) (take 5)) creates an intermediate lazy sequence between every step. That is convenient, but it allocates, and the pipeline is welded to sequences: you cannot reuse the same steps on a channel, a stream or a reduce. Transducers separate "what to do to each item" from "where the items come from and where the results go".

A transducer is a function that takes a reducing function and returns a new reducing function. You do not write that by hand in everyday code: you call map, filter, take and friends with no collection argument, and you get a transducer back.

Getting your first transducer

CLOJURE
(def xf (comp (filter odd?)
              (map inc)
              (take 3)))

(into [] xf (range 20))        ;=> [2 4 6]
(transduce xf + 0 (range 20))  ;=> 12
(sequence xf (range 20))       ;=> (2 4 6)

Three points to notice:

  1. (filter odd?), (map inc) and (take 3) are one-argument calls. With a collection they would be ordinary lazy sequence functions; without, they are transducers.
  2. comp composes them, and the data flows left to right (opposite of normal function composition, which reads right to left). Here odd numbers are filtered first, then incremented, then three are taken.
  3. The same xf runs in several contexts. into pours results into a collection, transduce reduces them, sequence produces a lazy sequence.

No intermediate collection is ever built: each item travels through all the steps before the next item starts.

The transducing contexts

Function Result
(into to xf from) from transformed and conj'd into to
(transduce xf f init coll) a reduction, like reduce with a transformation first
(sequence xf coll) a lazy sequence
(eduction xf coll) a reusable, re-runnable "recipe" you can reduce or iterate
(chan 10 xf) in core.async a channel that transforms values as they pass through

Transducers do not know which one they are in. That is the whole point: write the transformation once, and use it on a vector, a file's lines or an asynchronous channel without changing it.

CLOJURE
(def clean
  (comp (map clojure.string/trim)
        (remove clojure.string/blank?)
        (map clojure.string/lower-case)
        (distinct)))

(into [] clean ["  Hello" "" "WORLD " "hello"])
;=> ["hello" "world"]

Stateful transducers and early exit

Some transducers keep state: take, partition-all, dedupe, distinct and map-indexed hold counters or previously seen values. Clojure creates fresh state each time the transducer is run, so xf is safe to share between runs and threads.

take also supports early termination: once it has enough, it signals the reduction to stop. The source is not walked any further, which matters for infinite or expensive sources.

CLOJURE
(transduce (take 3) conj [] (range))   ;=> [0 1 2]   (range is infinite)

Inside a custom reduction you can end early yourself with reduced:

CLOJURE
(reduce (fn [acc x] (if (> acc 10) (reduced acc) (+ acc x)))
        0 (range))
;=> 15

Writing a transducer by hand

Occasionally you need one that the library lacks. The shape is: take a reducing function rf, return a function with three arities, init [], completion [result] and step [result input].

CLOJURE
(defn every-nth [n]
  (fn [rf]
    (let [i (volatile! -1)]
      (fn
        ([] (rf))
        ([result] (rf result))
        ([result input]
         (if (zero? (mod (vswap! i inc) n))
           (rf result input)
           result))))))

(into [] (every-nth 3) (range 10))   ;=> [0 3 6 9]

volatile! is a cheap mutable cell, safe here because the state is private to one run. The completion arity is where buffering transducers (like partition-all) flush the last group. Always call (rf result) there so downstream steps can finish too.

When to use them

  • Use transducers for hot paths, large inputs, multi-step pipelines you want to reuse in more than one place, and with channels.
  • Use plain sequence functions when the pipeline is short, runs once, or you want laziness and ->> readability. The two styles interoperate, and a transducer is never required.

A rough rule: if profiling shows a multi-stage pipeline allocating a lot, switch it to a transducer.

Try it: a transducer in JavaScript

The runnable stand-in below shows the core trick: each step wraps a reducing function, and comp stacks them so one item flows through everything before the next starts.

JS Transducers from scratch

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Try it: same steps, different destinations

The transformation stays the same while the destination changes: an array, a sum, a Set and a string.

JS One transformation, four results

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you get a transducer from map?

    The one-argument arities of map, filter, take and others return transducers.
  2. 2. In (comp (filter odd?) (map inc)), which step sees each item first?

    Transducers composed with comp process each item left to right, so filter runs before map.
  3. 3. Which is NOT a benefit of transducers?

    They often reduce allocation, but short pipelines may not benefit. Measure before converting.
  4. 4. What must the completion arity of a custom transducer do?

    Completion lets buffering steps flush, and each step must pass completion to the next.

Summary

  • A transducer is a transformation of a reducing function; call map, filter, take etc. without a collection to get one.
  • comp joins them; data flows left to right, one item at a time, with no intermediate collections.
  • Run them with into, transduce, sequence, eduction or on a core.async channel.
  • Stateful transducers get fresh state per run, and reduced or take stop early.
  • Use them for hot or reusable pipelines; plain lazy sequences are fine elsewhere. See the everyday sequence functions.

Next: Sequence Functions in Practice