Skip to main content
reticle grade answers one question: did a run leave the app the way the correct steps leave it, and give the correct answer? A run is any sequence of steps on an app you can reset: an agent, a script or a person. The command needs no daemon, no browser and nothing installed in the app. It reads the app’s stored data through the state sources that reticle compare --state uses.

How to grade a run

  1. Reset the app. Do the correct steps. Run grade snapshot --out reference.json.
  2. Reset the app. Do the correct steps again. Run grade snapshot --out again.json.
  3. Reset the app. Do the run to grade. Run grade snapshot --out candidate.json.
  4. Run grade judge --reference reference.json --again again.json --candidate candidate.json.
Steps 1 and 2 are done one time for each task. Keep the two files, and grade every run of the task against them. judge compares the lines of the three snapshots. A line that differs between the two correct runs (a clock, a random id) is noise and is ignored. Any other line that differs between the reference and the candidate is a difference, and the grade fails. So the grade fails when the run changed the wrong record, changed a record the task did not name, or did not change the record at all. A source that could not be read makes the grade unverified, never a pass. A row that the two correct runs wrote with different values in some cells (a time of the run, written into updated_at) has noise in those cells only. A candidate row that differs from the reference row only in those cells is not a difference. A change to another cell of that row, or to the same cell of another row, is a difference. Many apps save a record by deleting its child rows and writing them again, with new ids. A run that saves a record two times then has the same data under other ids. judge matches these rows: a removed row and an added row of one source are the same record when every cell after the first (the id) is equal, after each id it already matched is replaced by its new id. So a row that links to a written-again row matches too. A match needs an old id that the candidate no longer has, and a new id that the reference did not have, so a changed value in a first column that is not an id is still a difference. --json gives the count as state.renumbered. This applies to text rows (a command source, or a file of plain lines) whose cells are separated by tabs. --json prints one JSON object: verdict, state.removed and state.added (every line that differs), state.ignored, answer and unread. The text output shows the first 10 lines on each side and says how many it left out.

Snapshot

grade snapshot --state <file> --out <file> reads each declared source and writes its rows as state <label> <row> lines, sorted. A snapshot keeps every row: unlike compare, it has no 1000-row cap, so a changed row always names its record. A read that fails gives a state read failed: <label>: <reason> line, and the command exits 2. --url <url> adds the page the run ended on, as a url line. Use it for a task whose goal is to open a page. The fragment and a final slash are removed, and the query values are sorted.

Check the answer

A task that asks a question has a final answer. A fixed answer string goes stale when the data changes. Declare a source whose rows are the correct answer instead, and judge reads it when it grades:
Give the file with --expect, and the run’s answer with --answer. Text is folded before it is compared: Unicode normal form, lower case, spacing, quotes and a final full stop. A source with no rows means that the task cannot be done. Then the answer must be one of the none words. The default is N/A. A row with more than one column is one value, with the columns joined by a space.

Write the declaration

The declaration must list every table that a run can change, or a change to that table is not seen. Do not write it by hand. Generate it, then remove the tables that change on every run:
tables takes the first column of each line of the --list output as a table name. It skips a name with a character other than A-Z, 0-9, _ or $, and says so. {table} in --rows is the table name. --skip <prefix,...> leaves out each table whose name starts with one of the prefixes. Separate the prefixes with commas. Use it for tables that hold no task data and that the app changes at its own time: a message queue that a background worker empties, a cron schedule, a session store. Two correct runs do not always show these tables as noise, because the worker can finish before both snapshots and not before a third. For Magento, --skip queue_,cron_,session. Some apps write an empty row that means the same as no row. Magento saves a product form with a NULL row for each empty attribute, so a run that saves the form one more time than the reference adds rows with no value. Add an ignore pattern that matches all of such a row to the source in state.json. A pattern that removes all of a row removes the row. For the Magento attribute tables (catalog_product_entity_int, _decimal, _varchar, _text, _datetime), use "ignore": ["^\\d+\\t\\d+\\t\\d+\\t\\d+\\tNULL$"]. suggest keeps a source that reads the same on both runs, also a source the correct steps did not change: a change to it by a run is a change to a record the task did not name. A source that differs between the two runs only by a date and time is kept, with the time removed. Every other source that differs (a session table, a log) is dropped, because each run adds its own new rows to it, and every run would fail. Each dropped source is printed with the first row that differs. A change to a dropped table is not seen.
Last modified on November 13, 2026