Dolly
A native Mac screen recorder whose polish engine also runs headless: a recording plus a log of what happened becomes a finished film
- Role
- Sole designer and builder
- Status
- In development
- Period
- Nov 2026
Swift · Metal · AVFoundation · VideoToolbox · ScreenCaptureKit · SwiftUI · AppKit · ProRes
Problem
A polished screen recording, the kind Screen Studio makes, needs a cursor that glides, a camera that zooms onto what matters, and the screen framed on a background. Those are apps for a person at a Mac. My own automated demo and verification video pipelines record unattended with Playwright, and each carried its own polish code: a cursor that slid in straight lines between action points, and zooms written as an ffmpeg crop expression.
The open-source notscreenstudio (MIT) showed a free recorder of this kind is buildable. I read its code first and listed 11 shortcomings. Its cursor trails its own clicks by about 0.17 s, so the ripple lands where the cursor is not yet. Zoom and pan run on separate springs and are clamped at the frame edge, so moves swoop and stop dead. Every auto zoom is a constant 2x. And it expects a dense 120 Hz cursor log, which Playwright never produces.
What it does
Dolly takes a recording plus a log of what happened in it (clicks, hovers, key presses, zoom requests, cuts) and renders a polished film:
- A cursor that passes through every click and moves like a hand, even from a log of action points only.
- One planned camera path: zoom and pan move together, zoom depth fits the clicked element, and a cap stops it magnifying source pixels.
- Typing detection that zooms onto the caret and can speed typing up, plus an optional shortcut pill.
- Separate motion blur for cursor, zoom and pan.
The engine is a command-line tool, dolly render, that runs from Terminal or a launchd job with no app open, prints JSON progress and exits with documented codes. The Mac app records screen, camera, microphone and input, keeps each recording as a project with undo for every edit, and exports through the same engine. The key log never stores a typed character, only the class of each key.
Architecture
Contract. Four small JSON files: events, edits, style (every field defaulted, so {} is valid) and overlays. Events carry their own coordinate space, so Dolly knows nothing about Playwright; each pipeline converts its own log.
Motion. The whole log is known before rendering, so planning is offline. The cursor is synthesised between exact anchors with a minimum-jerk speed profile on a slightly bowed path. The camera is a zoom level plus a position inside its feasible range, driven by one eased parameter, so every frame is in bounds by construction.
Export and preview. Decode, render and encode overlap instead of running one frame at a time. The app's preview draws the same frames export writes, and tests compare them by hash.
Key decisions
-
Rewrite, not fork. notscreenstudio's motion and input model needed replacing either way, so I wrote Dolly from scratch and credited the ideas I borrowed: motion read from a precomputed table, a single-pass Metal renderer, parity tests. No code was copied.
-
Engine first. The first milestone was the engine, CLI and an adapter for my verification pipeline; the app came second. No product I surveyed takes a sparse automation log.
-
Command Line Tools only. No Xcode on the build Mac. Shaders compile at run time, and the app uses only the SwiftUI a probe compiled, with state in
@Observablemodels because@Stateneeds a missing macro plugin. -
Determinism is the test. Two renders must produce identical pre-encode frame hashes, and every merge checks a real recording's full hash list. Hashes come before encoding because encoder output varies run to run.
-
Constant quality, not bit rate. At 0.25 bits per pixel the first key frame fell to 38-39.5 dB. Quality 0.85 gives a worst of 40.6-42.4 dB on three real recordings at about half the size.
-
ProRes 4444 capture by default. On 1 px coloured text, every 4:2:2 format measured 23.2-23.5 chroma error, H.264 37.8, and 4444 0.07. It costs 7.7, 10.6 and 12.8 GB per minute at 4K, 5K and 6K on busy content, so 422 LT is the "smaller files" preset.
Numbers
The export criterion is a 60 s recording in at most 20 s at 1080p60 and 60 s at 4K60. Measured on the development Mac at load 3.5-4.3, not a quiet machine:
| Export of 60 s | Bound | Measured |
|---|---|---|
| 1080p60 | 20 s | 7.24 s |
| 4K60 | 60 s | 24.38 s |
| 4K60 from a ProRes 422 source | 60 s | 24.38 s |
| 1080p60, motion blur 0.5 | 20 s | 7.12 s |
| 4K60, motion blur 0.5 | 60 s | 24.32 s |
That is a 2.5-2.8x margin. In motion tests the cursor's worst distance from a click is 0.0 output px over 32 tracks at 1080p and 4K, against a 2 px bound, and it arrives 0.161-0.263 s before each click over 40 moves. 184 of 184 clicked elements stay in view with zoom capped at 1.0 output px per source px. Preview frames matched export 23 of 23, worst PSNR 40.39 dB. In my verification pipeline a walkthrough render took 3:00 against 4:19 for the old path. A 40-minute render peaks at 100.8 MB, down from 2.83 GB.
What broke
- The encoder crashed under contention. With every hardware H.264 session busy, the software fallback rejected the quality setting and killed the render. Dolly now retries on a bit rate, and passed 110 bursts of 6 concurrent renders.
- Trimmed audio played 21.7 ms late. A stream copy cuts AAC on packet boundaries, leaving an empty edit. The first fix stalled the writer; trimmed audio is now re-encoded.
- The first review found data loss. An output path naming a folder replaced it, one equal to the input overwrote the recording, and audio more than 1 s ahead of video deadlocked the exporter. A second review could not break the fixes.
- App and CLI disagreed. The CLI reads colours as 8-bit hex, the app planned from memory, so a ripple alpha of 0.55 became 140/255 and 30 of 300 frames differed. The app now plans from the written files.
- The first real-hardware session failed. macOS kept permission grants keyed to earlier ad-hoc-signed test builds, so the signed app read as denied, and two windows opened out of view. The fix is a draft PR, not yet verified.
Results
Started on 16 November 2026. M1 (engine, CLI, pipeline adapter) passed all 9 criteria after two independent reviews and a fresh verification rerun. M2a (typing, colour, blur, edits) passed all 10. Dolly is now the default renderer in my verification video pipeline, falling back to the old render if it is missing or fails. M2b, the app, is built and its exports match the CLI frame for frame in headless tests, but recording on real hardware is not yet accepted, and the work is on hold until 25 November.
The repository is private. The app is signed with a self-signed local certificate, not Developer ID, and is not notarized. The site takes waitlist sign-ups.
What I'd change
- Real hardware on day one. I built the full app before one session with real permission grants, and signing problems surfaced last.
- The 4K preview is too close to its bound. Seeking with layers measured 149.5 ms against a 150 ms bound, under load, so it is indicative only.
- The pitch-preserving time stretch is not bit-deterministic. Lengths vary by 7-8 samples run to run, so one audio test flakes under load.