Paper2Video Why? My papers Videos Figures How it works? Source Alina Devkota

Paper2Video

Papers you can watch!

Paper2Video reads a machine-learning paper or a lecture deck and produces a narrated explainer video in which animated technical figures are drawn in step with the sentence being spoken. The same source file also renders as a PowerPoint deck with the narration in the speaker notes.

Everything visual is computed and deterministic. No generative video model is involved: the figures are matplotlib drawings whose numbers can be checked against the paper.

"type": "illustration",
"figure": "attention_lookup",
"heading": "What attention computes",
"caption": "A soft lookup over every position.",
"opts": {
  "tokens": ["The", "animal", "didn't", "cross",
              "the", "street", "it"],
  "query": 6,
  "scores": [0.4, 3.2, 0.3, 0.9, 0.4, 1.6, 0.5]
},
"narration": [
  "Every position emits a query, a key and a value.",
  "Dot the query against every key.",
  "Softmax turns those scores into weights."
]

One scene of a storyboard. A paper is a sequence of these: each names a figure, supplies its content as data, and carries narration lines. Each line advances the figure by one reveal.

The scene above, rendered. The softmax weights come from the scores in the storyboard, so the 0.65 on animal can be checked by hand.

Why?

Reading a paper well takes hours. Most people triaging a literature list want the argument, the mechanism and the caveats in about ten minutes, and the existing options are poor.

The gap is a medium that is visual, narrated, cheap enough to make per paper, and accurate enough to trust. Video generation models fail the last requirement: they produce evocative footage with no relationship to the paper's claims. A deterministic renderer driven by structured data does not.

My papers

Explainers for the papers I wrote as first or second author, made with the same pipeline. Each one starts from the paper's storyboard and is reviewed by the person who knows the work best.

Papers and lectures I picked

Each of these started as one PDF and one storyboard. Nothing is hand-animated; every frame comes out of the same figure library. The automatic storyboards were a starting point, and every one shown here was reviewed and edited by hand. Each ends with a caveat scene naming what the paper does not establish, in proportion. Captions are built in.

One vocabulary, reused

A primitive is a generic drawing function with no paper-specific strings or numbers: pipeline draws a box-and-arrow flow for any architecture, curve_family plots any family of curves with an optional Pareto frontier, vector_decompose splits a vector out of any subspace. A preset is a primitive plus a bag of options, pure data. A new paper normally contributes presets only.

Two rules keep the library from becoming a junk drawer. Name the shape, not the paper: stacked_blocks, never transformer_encoder. And every visible string comes from the options. If a label is typed into the function body, it belongs in a preset.

The library is young. A hundred primitives were enough for the videos on this page, but most papers still call for shapes it does not have yet, so growing it is the main work ahead. Every new primitive makes the next paper cheaper to storyboard.

One frame per kind of figure, cropped from the videos. Hover for the source; click to watch that moment.

The contract for a primitive

def my_figure(R, ax, u, d, o):
    """One-line description, shown by --figures."""
    x, y, w, h = _box(o)                  # the drawing rect
    a = _fade(u, R.rt(0, 0.4, 0.6), 0.6)  # reveal on beat 0
    R.txt(ax, x, y + h, o.get("title", ""), 20, R.c["ink"], a)

R.rt(k, …) means "the start of narration beat k". The renderer resolves it against the measured audio, so an element appears exactly when its sentence begins.

How a video gets made?

Five stages, one storyboard file in the middle. The storyboard is the review surface: a person reads it, fixes the claims, adjusts the emphasis and re-renders.

Pipeline diagram. 1 Extract: Poppler turns the PDF into section text and page renders. 2 Plan: the Claude API, a local LLM or a hand-written outline produces a storyboard JSON, constrained by the figure catalogue. 3 Narrate: narration lines go through a TTS engine to audio with timing, one clip per line. 4 Render: the figure primitives library and the render engine draw frames synchronized with the audio. 5 Deliver: one storyboard yields an MP4 video and PPTX slides. A human review loop feeds back into rendering. No generative video models.
Click to open at full size.

  1. Extract

    Poppler pulls section text, full-page renders and embedded figures from the PDF. A scene can crop a lecture deck's own diagram straight from a page.

  2. Plan

    Claude drafts the storyboard against a fixed schema. The prompt carries the live figure catalogue, so the model can only name figures that exist, and anything the validator rejects is sent back for repair.

  3. Narrate

    Each line becomes one audio clip, cached by a hash of text, voice and rate. Scene length is measured from the rendered audio, so a reveal lands on the sentence that explains it.

  4. Render

    Every frame is drawn from scratch in matplotlib by the figure primitives library and piped to ffmpeg, with soft subtitles.

  5. Deliver

    One storyboard gives an MP4 with narration and a PowerPoint deck with native shapes where an emitter exists and the narration in the speaker notes.

What actually runs for a normal render is a text-to-speech model and nothing else. The LLM runs only during planning, and no generative video model is involved at any stage.

What it is for?