Parrot — Record and Replay X Application Interactions

Parrot lets you record mouse clicks, keyboard input, and navigation paths through any X11 GUI application, then replay those interactions automatically. It is designed for benchmarking and testing desktop applications in a reproducible, sandboxed environment.

Everything runs inside Docker, so there are no security implications for the host system and results are fully reproducible across machines.

How It Works

1. Record — Interact with any X application through a browser-based VNC viewer. Parrot captures every mouse movement, click, and keystroke. It also takes care of setting the screen and application size correctly.
2. Replay — Play back the recorded session against the same application. Parrot finds the correct window, replays each action with correct timing, and optionally asserts on screenshots.
3. Benchmark — Plug the replay into the Green Metrics Tool via a usage_scenario.yml to measure energy and performance.

Example Applications

Application	Path
Firefox	applications/firefox/
LibreOffice Calc	applications/calc/
VLC	applications/vlc/
Okular (PDF viewer)	applications/pdf_viewers/okular/
Thunderbird (email client)	applications/emailclients/thunderbird/

Each application directory contains the recorded .parrot macro file and a usage_scenario.yml for use with the Green Metrics Tool.

Three directories hold whole comparisons rather than a single application, where the same script is recorded against every candidate:

Comparison	Path	What it provides
PDF viewers	applications/pdf_viewers/	Six viewers over one local document
Email clients	applications/emailclients/	Eight clients over a local IMAP server holding a deterministic ~500 MB mailbox
Code editors	applications/codeeditors/	Eleven editors over one Python project, including a generated 10 MB source file

One directory holds a comparison of websites rather than of applications, and it sits at the repository root because of that:

Comparison	Path	What it provides
Websites	websites/	Any page, passed in as a GMT variable, loaded in a real Firefox and a real Chrome behind a warm caching proxy

Its macros are written by hand rather than recorded, and it carries no reference screenshots: the page under test is a variable, so there is nothing to compare pixels against. See websites/README.md.

Quick Start

1. Start the Window Container

Every application ships with a Docker Compose file (or uses the shared ribalba/xwindow-server image).

2. Record a Macro

./record-macro.py applications/firefox/firefox.parrot

• Interact with the application in the browser-based VNC viewer.
• Press Pause to stop recording.
• Press Scroll Lock at any point to capture a reference screenshot (inserts a Check assertion into the macro).
• Pass --script path/to/script.md to attach one note per checkpoint; trimmed blank lines and lines starting with # are ignored.
• A script line may be written as label: detailed instruction. Recording shows the whole line, so whoever drives the application knows exactly what to do and every application is driven identically. Replay emits only the label as the note. See Script labels.
• When a script is provided, recording shows a live checklist and highlights the next pending item in green.

Override the hotkeys if your browser intercepts them:

STOP_KEYSYM=F9 CHECK_KEYSYM=F3 ./record-macro.py applications/firefox/firefox.parrot

Output files are written to recordings/<app-name>/:

recordings/firefox/firefox.parrot
recordings/firefox/firefox-check-001.png

3. Replay a Macro

./replay.py applications/firefox/firefox.parrot

Optional speed multiplier:

REPLAY_SPEED=2.0 ./replay.py applications/firefox/firefox.parrot   # faster
REPLAY_SPEED=0.5 ./replay.py applications/firefox/firefox.parrot  # slower

Replay finds the application window by class/title metadata embedded in the macro, focuses it, and plays back every action. Any Check line triggers a screenshot comparison against the saved reference image. A failed check exits non-zero.

Skip the screenshot comparisons entirely with --no-checks (or REPLAY_IGNORE_CHECKS=1). Check lines are then logged and skipped instead of asserted, so replay never fails on a mismatched display. This is useful for benchmarking on hardware where the reference screenshots were not captured:

./replay.py --no-checks applications/firefox/firefox.parrot
REPLAY_IGNORE_CHECKS=1 ./replay.py applications/firefox/firefox.parrot

Screen Recording

Set RECORD_VIDEO to capture the full replay as an MP4 video:

# Save to a specific file
RECORD_VIDEO=/tmp/my-replay.mp4 ./replay.py applications/firefox/firefox.parrot

# Auto-generate a timestamped file under /tmp
RECORD_VIDEO=1 ./replay.py applications/firefox/firefox.parrot

The video is recorded at 25 fps using ffmpeg's x11grab input. The output path is printed to stderr at the start and end of the run. If a Check step fails and replay exits early, the video is still finalized and saved.

Recording Any X Application

Parrot is not limited to the bundled applications. You can record interactions with any X11 GUI application by supplying the relevant metadata at record time:

APP_STARTCOMMAND='xterm' \
APP_WINDOW_TITLE='xterm' \
APP_WINDOW_CLASS='xterm' \
./record-macro.py <your-parrog-file>

Metadata is embedded into the .parrot macro file and used by replay to locate and focus the correct window.

File Extension

Parrot macro files support two extensions — .parrot and .🦜 — and both work identically. This documentation uses .parrot throughout because it is easier to type in terminals and scripts. Feel free to use .🦜 if you prefer the native extension:

./record-macro.py applications/firefox/firefox.🦜
./replay.py applications/firefox/firefox.🦜

Macro File Format

Recorded macros (.parrot files) contain:

• #APP metadata lines — window class, title, and optional start command
• Timed xmacro events — MotionNotify, ButtonPress, KeyStrPress, etc.
• #WAIT_SEC <seconds> — timing gaps between events
• Check <path>.png — screenshot assertion lines
• log <text> — note lines emitted to stdout during replay as <timestamp_microseconds> <label>, where the label is the part of <text> before the first colon (see Script labels)

Example header:

#APP startcommand='firefox https://browserbench.org/Speedometer3.1/'
#APP windowtitle=Firefox
#APP windowclass=firefox

Script labels

A checkpoint script line can carry both a short label and the full instruction, separated by the first colon:

* Reply and send: reply to message 2 with the body text `Thank you so much` and send it

The two audiences want different things from that line, so each gets what it needs:

Where	What it uses	Why
Recording checklist	the whole line	Whoever drives the application has to know exactly what to do, or two recordings of the "same" step end up being different steps.
The .parrot file	the whole line	A recording stays self-documenting; you can read what a block was meant to do a year later.
Replay output and Green Metrics Tool notes	the label only	The note names a measurement phase. A full sentence per phase makes the report unreadable.
tools/check_blocks.py table	the label only	Same reason — but block identity still compares the whole line, so two files disagreeing on the detail is still an error.

Only lines starting with * become checkpoints. Blank lines and lines starting with # are skipped — but an HTML comment block is not. <!-- ... --> in a script file is parsed as ordinary note text, so every line of it is consumed as a checkpoint note and the real steps shift out of alignment. The recording still has the right number of blocks, which is what makes it easy to miss: the block names are the comment prose. Keep explanations in a README rather than in the script.

Scripts without colons keep working unchanged: the whole line becomes the label. Only the first colon splits, so an instruction may contain as many more as it likes. A line starting with a colon is left alone rather than producing an empty note.

applications/emailclients/script.md uses this throughout — the same scenario has to be reproduced by hand against eight different email clients, which is exactly the case where a vague instruction turns into eight subtly different benchmarks.

Screenshot Assertions

Press Scroll Lock during recording to insert a checkpoint. During replay, Parrot compares the current window screenshot to the saved reference using RMSE.

Tune the comparison:

# Loosen the threshold
CHECK_MAX_RMSE=0.02 ./replay.py applications/firefox/usage_scenario.yml

# Ignore dynamic regions (e.g. toolbars, clocks) — format: x,y,width,height
CHECK_IGNORE_RECT=0,0,420,40 ./replay.py applications/firefox/usage_scenario.yml

# Multiple regions (semicolon-separated)
CHECK_IGNORE_RECT="0,0,420,40;300,580,120,30" ./replay.py applications/firefox/usage_scenario.yml

Green Metrics Tool Integration

Parrot has native support for the Green Metrics Tool. Each application includes a usage_scenario.yml that defines the container setup and the replay command:

name: Parrot Firefox
author: Didi <didi@green-coding.io>
description: Benchmarks Firefox using Parrot

services:
  window-container:
    image: ribalba/xwindow-server
    environment:
      DEBUG: 0
    setup-commands:
      - command: bash /tmp/repo/applications/firefox/install.sh
      - command: bash /usr/local/bin/entrypoint.sh

flow:
  - name: Run Benchmark
    container: window-container
    commands:
      - type: console
        command: python3 /usr/local/bin/replay.py /tmp/repo/applications/firefox/firefox.parrot

Point the Green Metrics Tool at the repository and it will set up the container, run the replay, and collect energy and performance metrics automatically.

Running the Same Software Everywhere

The benchmarks run on several machines over several weeks. Three things keep the runs comparable and make them work on every GMT setup.

A frozen Ubuntu archive

Every window-container starts with applications/common/apt-snapshot.sh. It points apt at snapshot.ubuntu.com as of 20260929T150000Z, the hour of commit cdbdd06, which defined the sweep. Every later apt-get in that container installs from the archive as it was then.

Without it two things go wrong:

• Pinned versions vanish. Ubuntu keeps only the newest version of a package in noble-updates. vim 2:9.1.0016-1ubuntu7.18 disappeared on 2026-09-29, libreoffice-* 4:24.2.7-0ubuntu0.24.04.6 between 2026-10-01 and 2026-10-06, and the installs failed with "Version ... was not found".
• Everything unpinned moves silently. All terminals, all video players and every dependency of every app install whatever the archive holds that day, so two machines or two days measure different software without any error.

apt-get --snapshot does not work here: the image's Release files carry no Snapshots: field, so apt ignores the option and keeps reading the live archive. The script therefore writes the snapshot URL into the sources itself, turns off Check-Valid-Until (a snapshot's Release files are past it by design), and fails if the lists did not come from the snapshot. Vendor repositories that install scripts add later are untouched. Their packages are pinned by version and sha256 in the install scripts.

To move the snapshot, change SNAPSHOT in the script and check that every pinned version still resolves there.

GMT's network proxy provider

GMT's network_connections_proxy_container provider starts tinyproxy on the host and sets HTTP_PROXY, HTTPS_PROXY and their lower-case twins in every container. Its NO_PROXY lists only the compose service names. The chat and email clients, however, talk to their servers by network alias (matrix.parrot.test, mail.parrot.test). Those requests went to the host proxy, which cannot resolve Docker-internal names, and the chat client setup failed with "http://matrix.parrot.test:8008/... never answered".

The chat and email scenarios therefore set NO_PROXY and no_proxy in window-container to the server aliases, the .parrot.test domain, the service names and localhost. GMT puts its own value earlier on the docker run line, and Docker keeps the last one. Without the provider the variables change nothing.

Everything else, including apt and Flathub downloads, still goes through the proxy, which is what the provider is there to record.

Rootless Docker

In rootless Docker the container's root is the host user, and only the uids in that user's subordinate range exist inside it (65536 by default). Extracting a tarball as root makes tar restore the owners recorded in the archive. The Eclipse tarball records uid 1001330000, which has no mapping, and tar fails with "Cannot change ownership to uid 1001330000 ... Invalid argument". Every tar extraction in the install and seeding scripts therefore uses --no-same-owner. The files then belong to root, which changes nothing on a rootful host: the apps that come from tarballs all run as root.

Two things are host setup, not code:

• The network driver. Without slirp4netns or passt installed, dockerd-rootless.sh falls back to rootlesskit's gvisor-tap-vsock driver. It loses a few new connections per thousand (measured 2026-10-12: 34 of 10,000 connects timed out, none with slirp4netns, pasta or rootful Docker). Behind GMT's proxy every plain-HTTP download is a new connection, so long apt installs failed with "Unable to connect to <host>:8889". Install slirp4netns and set DOCKERD_ROOTLESS_ROOTLESSKIT_NET=slirp4netns in the rootless docker service.
• Flatpak under AppArmor. Rootless Docker cannot apply the parrot-flatpak profile. On Ubuntu 24.04 (kernel.apparmor_restrict_unprivileged_userns=1) bubblewrap's user namespace may therefore be refused. Check with the probe in host-setup/install-apparmor-profile.sh before running the Flatpak scenarios on such a host.

Comparing Recordings Across Applications (tools/check_blocks.py)

When the same script is recorded against several applications (e.g. multiple PDF viewers running the same checklist), each .parrot file should contain the same ordered set of blocks. A block is the sequence of actions ending in a log <message> line followed by a check <ref>.png line — the log message is the block's identity.

tools/check_blocks.py cross-checks those files, prints a per-block wait-time table, and can split or time-normalize them.

# Verify structure and print the wait-time table
./tools/check_blocks.py applications/pdf_viewers/

# Write each block as its own self-contained .parrot under <app>/blocks/
./tools/check_blocks.py applications/pdf_viewers/ --split

# Write <name>-normalized.parrot next to each file, padding every block to
# the longest duration that block has across all files
./tools/check_blocks.py applications/pdf_viewers/ --normalize-time

The script recursively finds every *.parrot under the given folder, ignoring -normalized.parrot outputs and anything inside a blocks/ directory. If the files disagree on block names or order, mismatches are listed on stderr and the script exits non-zero.

Time normalization inserts a single extra wait <padding> line immediately before each block's log line, so the action timing within a block is preserved while the total runtime of each block matches across files — useful for fair side-by-side benchmarking.

One GMT phase per block (replay.py --block)

replay.py <file>.parrot --block N replays only block N of a recording, where block N is everything after checkpoint N-1 up to and including checkpoint N (the same boundary check_blocks.py uses). Block 1 starts the app exactly as a whole-file replay does. Every later block continues the app the earlier blocks left running and touches nothing first: no repositioning, no raise or focus, no lock-key reset, because a whole-file replay does none of that between blocks either. If the window is gone, the block fails instead of starting the app again. Replaying blocks 1 to N one after another yields the same events with the same waits as replaying the whole file.

tools/make_block_scenarios.py uses this to write usage_scenario_blocks_normalized.yml next to every usage_scenario_normalized.yml: the same services, with one flow step per block, so every block is a GMT phase of its own. Each step is named after its block's label, so phase N carries the same name for every app in a group. Rerun the tool after re-recording or re-normalizing; --check exits non-zero when a copy is missing or out of date.

./tools/make_block_scenarios.py                 # write or refresh all of them
./tools/make_block_scenarios.py --check         # write nothing, report stale copies

Deterministic Window Layout

To keep click coordinates and screenshots stable across runs, configure fixed window geometry in the compose environment:

Variable	Description
AUTO_POSITION	1 to enable, 0 to disable
WINDOW_X / WINDOW_Y	Window position
WINDOW_WIDTH / WINDOW_HEIGHT	Window size
APP_WINDOW_CLASS	xdotool window class matcher
APP_WINDOW_TITLE	xdotool window title matcher

The class is tried first and the title only if it finds nothing, so an app whose windows all share one WM_CLASS needs the class left empty to be matched reliably — otherwise the lookup returns whichever window it happens to hit first. Recording with --windowclass '' stores an empty value and matching falls through to the title alone. Thunderbird needs this: its main window is "Mail", its composer "Msgcompose" and its dialogs "Thunderbird", so no single class identifies the window the coordinates were measured against.

Empty is therefore meaningful and is preserved end to end — through the .🦜 file, position-window.sh and check-image.sh. Only an unset matcher falls back to the gnome-calculator demo default.

Apps whose window appears only after the macro clicks

After launching the app, replay.py waits for its window to map before it plays the first event, and fails with the launch log if it never does. For most apps the window it waits for is the window the macro drives. The JetBrains IDEs are the exception: they run on their own defaults here, so a fresh install shows the user agreement, the data-sharing consent and the trust prompt first, and the window titled after the project does not exist until the macro has clicked through all three. Waiting for it deadlocks — the click that creates it is behind the wait — and the run dies with

[replay] FATAL: the app did not map a window in 90s

Such a recording names a second matcher, used for that wait and nothing else:

windowtitle = project
windowclass =
startupwindowtitle =
startupwindowclass = jetbrains-idea

Positioning and every Check keep using windowtitle/windowclass, so it is safe to name a class here that also matches the app's dialogs and helper windows. While only the startup window is up, position-window.sh is skipped — a recording that needs this gets its geometry from pin-windows.sh before the window manager starts. Record it with --startupwindowclass / --startupwindowtitle; omit both for every other app.

Environment Variables

All variables can be set on the command line or, when running inside the Green Metrics Tool, in the environment: section of a service in your usage_scenario.yml:

services:
  window-container:
    image: ribalba/xwindow-server
    environment:
      REPLAY_SPEED: "0.8"
      CHECK_MAX_RMSE: "0.05"
      PARROT_DEBUG_DIR: /tmp/parrot-debug

Replay (replay.py)

Variable	Default	Description
DISPLAY	:99	X display to replay on.
REPLAY_SPEED	1.0	Playback speed multiplier. 2.0 is twice as fast, 0.5 is half speed.
RECORD_VIDEO	(off)	Set to a file path to save an MP4 of the replay, or 1/true to auto-generate a timestamped path under /tmp.
REPLAY_INIT_CAPSLOCK	off	Desired Caps Lock state before replay starts (on, off, or keep).
REPLAY_INIT_NUMLOCK	off	Desired Num Lock state before replay starts (on, off, or keep).
REPLAY_INIT_SCROLLLOCK	keep	Desired Scroll Lock state before replay starts (on, off, or keep).
REPLAY_IGNORE_CHECKS	(off)	Set to 1/true to skip all Check screenshot comparisons (same as the --no-checks flag). Checks are logged and skipped instead of asserted.

Screenshot checks (check-image.sh)

Variable	Default	Description
CHECK_MAX_RMSE	0.2	Maximum allowed normalised RMSE between the captured screenshot and the reference image. Lower values are stricter.
CHECK_IGNORE_RECT	(none)	Mask one or more regions before comparing. Format: x,y,width,height. Separate multiple rectangles with ;. Useful for toolbars, clocks, or other dynamic areas.
CHECK_SCALE_ON_MISMATCH	0	Set to 1 to scale the captured screenshot to the reference size when dimensions differ, instead of failing immediately.
PARROT_DEBUG_DIR	(none)	Directory where failure artifacts (actual screenshot, diff overlay, metadata) are copied when a check fails. The directory must already exist. When unset, artifacts are only kept in a temporary directory that is deleted on exit.

Window server (entrypoint.sh)

Variable	Default	Description
DISPLAY	:99	X display number used by Xvfb and all X tools.
SCREEN_SIZE	1440x900x24	Virtual display resolution and colour depth passed to Xvfb.
DEBUG	0	Set to 1 to enable verbose entrypoint logging.

Recording (record-macro.py)

Variable	Default	Description
STOP_KEYSYM	Pause	Key that stops the recording session. Override if your browser or OS intercepts the default.
CHECK_KEYSYM	Scroll_Lock	Key that inserts a screenshot checkpoint during recording. Must differ from STOP_KEYSYM.

Troubleshooting

• Stop key not working in browser — set STOP_KEYSYM to another key, e.g. F9.
• Check hotkey affects the app — Scroll Lock is the default; if your keyboard or environment handles it poorly, choose another key via CHECK_KEYSYM.
• Screenshot checks fail due to minor UI variation — increase CHECK_MAX_RMSE slightly or mask dynamic regions with CHECK_IGNORE_RECT.
• Click coordinates are off — rebuild/restart the container and re-record; window geometry may have shifted.

Workflow to record an application

1. Copy a usage_scenario file into a new folder. Edit it so that the applications are installed in the setup commands section. Also change the paths to the new folder

2. Start the application with the GMT to make sure that everything is set with

   ./runner.py --uri /home/didi/code/parrot --filename applications/pdf_viewers/okular/okular.yml --dev-no-sleep --allow-unsafe --debug

   Then step to the point when the container is started and the setup-commands are exectuted.

3. Connect to the VPN through http://localhost:6080/vnc.html

   You should see a blank screen with no application loaded

4. Start the recording program with

   ./record-macro.py --script applications/pdf_viewers/script.md --startcommand okular --windowtitle Okular --windowclass okular  applications/pdf_viewers/okular/okular.parrot

   make sure to adapt the commands. Now the application should be loaded in the VNC and have the focus. Now everytime you select Scroll Lock the script should advance.

Credit / Funding

This work is funded by the Deutsche Bundesstiftung Umwelt (DBU) under the number DBU Project 39703/01

Project details are to be found on the project page

We are super grateful for this funding and are blessed to have been granted the opportunity to create this data repository for the greater software community!