How we verify

A test suite that has never failed proves nothing at all. So every contract in this catalogue is measured the only way a test can be: by breaking the implementation on purpose, one defect at a time, and requiring the suite to notice.

The apparatus is calibrated before anything is measured

A run that is stuck red catches everything and means nothing; a run that is stuck green catches nothing and looks like a clean bill of health. So each battery names one obvious defect and refuses to measure anything until two things have happened in front of it: the unmutated code passes, and that obvious defect fails. Neither half alone is a calibration.

A run also counts the tests it collected and compares them with a figure declared per file, because a configuration that quietly collects a fraction of the suite leaves every result agreeing with itself. That has happened here three times, all three by accident.

What this does not prove

A high score does not say the code is correct. It says the tests notice the defects that were tried.

That is the whole limit of the method, and it is worth stating plainly because the number below reads like a correctness claim and is not one. What makes it worth publishing anyway is the part almost nobody else can offer: the defects that were tried are not a summary or a percentage. They are written down, named, and readable — every one of them is a committed file in this repository, with the exact edit it makes and the verdict it must produce.

What the tests catch

every figure here is read off the batteries in this repository, where each cell carries the verdict it must produce. That is what this project asserts about its own tests. It is not yet something you have seen happen.

npm run mutation is what turns it into something you have watched happen: it injects every defect below into a working tree, runs the suite once per cell, compares what happened against what the battery pinned, and prints the total. A single cell that disagrees fails the run. One run of it over the 19 batteries took 41 min 24 s, measured at 1fb1d85; every population here is counted in defect cells, and this reading is of 660 where the figure it replaces was of 631 - so the 34 min 30 s at 3949c87 is a real past reading of different work rather than a faster version of this one, and the instrument gained a reading of its own between them. This population has one reading, so the figure above is bounded by the readings of every other population rather than by a twin of its own. The closest two of one population this machine has given are the 623 cells: 28 min 9 s at b438de2 and 29 min 2 s at 75d3358, fifty-three seconds apart on identical work. The widest is four and a half minutes, over the 621: 28 min 1 s at 6226769 and 32 min 28 s at 5bb4e97. Every reading below can only approximate the same point by comparing populations, and they do not order themselves by population either: the 618 ran at 27 min 22 s and at 28 min 42 s, the 615 at 27 min 34 s, the 614 at 34 min 6 s, and the 612 at 27 min 8 s - so the smallest population is the fastest run and the second smallest is the slowest by seven minutes, and the 614 came within half a minute of the 631 with seventeen cells fewer. The variation is the machine rather than the cells. Before this regime a cell of a contract battery collected all five contracts rather than its own: the same 612 cells ran there at 31 min 25 s, the 610 before them at 29 min 13 s, the 606 before those from 28 min 19 s to 35 min 10 s, the 605 before those from 29 min 22 s to 37 min 0 s, and the 592 before those from 25 min 8 s to 28 min 59 s. No share of any of those steps is attributed to anything, and the reason is measurable rather than modest: cli-install has run from 364 s to 484 s on identical work, a quarter of its own duration, and took 576 s in the run above - so a machine this variable cannot support an account of where any of it went. That is one run rather than how long it takes. npm run tally prints that total again from the results a replay left, measuring nothing - and refuses a set that is not one complete replay of the commit it would describe, so a figure taken over a stale or partial run cannot be produced by accident.

660 defect cells, 625 caught. The other 35 are 12 equivalent, 6 outside what the contract specifies, 4 unreachable on this catalogue, 1 a declared open class, 12 only where a lens blinded the suite. Each one is below, with the sentence its own battery carries about it.

12 cells of 7 defects — equivalent

the edit cannot change an answer, so no test could catch it. Measured rather than assumed: each one carries what was compared against what.

fixture · FX-2 on C/as-committed

adds the value to itself instead of multiplying it by two - an equivalent mutant, and the one the meta-tests need: an apparatus pointed at it for calibration has been shown able to see nothing at all, and must be refused

number-parse · P-02 on C/as-committed and C/reason-blind

memoises into a bare object, consulted after the grammar guard. It survives because the guard is in front of it: only a string the decimal grammar has already accepted reaches the cache, so the inherited keys that make P-17 a defect - constructor, toString - are not keys it can hold. Measured differentially against the reference over a corpus of 300 067 inputs, each seen once cold and once with a foreign call in between: nothing tells the two apart

number-parse · P-06 on C/as-committed and C/reason-blind

uses parseFloat instead of Number. It survives because the grammar has already run: the two differ on exactly the language the grammar exists to reject - parseFloat reads a prefix and Number refuses the whole string - and nothing that reaches this line is outside the decimal grammar. Measured differentially against the reference over a corpus of 300 067 inputs: nothing tells the two apart

number-parse · P-14 on C/as-committed and C/reason-blind

uses the global isFinite instead of Number.isFinite. It survives because the coercion the two differ by cannot happen here: value is what Number(...) just returned, so it is already a number and the global has nothing left to convert. The edit is a real defect in a module where that line takes an argument, and this is not one. Measured differentially against the reference over a corpus of 300 067 inputs: nothing tells the two apart

number-parse · P-19 on C/as-committed and C/reason-blind

memoises into a Map, consulted first - the same cache without the inherited keys. It survives because a Map has no prototype to serve from and the key is the exact string the analysis is a function of, so a hit and a miss cannot answer differently. It is the control for P-17: same position, same advance, and the only difference is the container - which is what makes P-17 a defect about Object.prototype and nothing else. Measured differentially against the reference over a corpus of 300 067 inputs, each seen once cold and once with a foreign call in between: nothing tells the two apart

number-parse · P-20 on C/as-committed and C/reason-blind

writes the trim by hand as /^\s+|\s+$/g. It survives because \s and what trim removes are the same set - WhiteSpace and LineTerminator, which is where the no-break space and the byte-order mark this contract has cases for both sit - and because the literal stays inside the call, so no lastIndex outlives one. It is the near miss of P-08, which adds the global flag to a literal at module scope and is killed by two guards. Measured differentially against the reference over a corpus of 300 067 inputs: nothing tells the two apart

packaging · A-13 on A/as-committed

stops removing the output folder before writing it, which build.ts calls the whitelist failing in the one direction a whitelist cannot catch. Measured by planting a file no source produces and building twice: it is gone with the clean and gone without it, because a stale module is one nothing imports and the prune already drops whatever the entry point cannot reach. The two are not independent here - the second dominates the first

6 cells of 3 defects — outside what the contract specifies

the edit changes something a caller could observe, and the contract deliberately makes no promise about it. What this marks is the edge of the specification, not a gap in the tests.

array-group-by · M-18 on C/as-committed and C/identity-blind

rebuilds the group array on every insertion instead of pushing into it. The answer is right on every input and the cost is quadratic in the size of a group - measured, 5392 ms against 0.7 ms on the fifty-thousand-element single-group sample. It survives, and it is the mutant that found a defect in this contract's own guards: under vitest's default five-second limit it was killed by the block 4.5 shape test, which asserts a shape and was silently asserting a duration as well. A verdict eight per cent away from flipping with the speed of the machine is not a verdict, so that test now carries an explicit timeout and this cell records the truth: nothing in this contract constrains complexity

array-group-by · M-20 on C/as-committed and C/identity-blind

recycles one module-scope Map, cleared at the start of every call and handed to every caller. Pinned as a survivor on purpose: the contract requires each *group* to be a fresh array and says nothing about the Map that holds them, so a caller who keeps a result watches it empty itself on the next call. The determinism property is green here for the reason date/add@1 recorded of its D-02 - both calls return the one object, and it is compared against itself

number-parse · P-16 on C/as-committed and C/reason-blind

writes a call counter onto globalThis. It is the one survivor of this battery that is observable from outside the function, and that is why it is not filed as an equivalence: every answer is the reference's, and the counter is there to be read afterwards - measured, 900 201 after a differential run. What the contract constrains is what this function *reads* - the freedom-from-ambient-input property is about exactly that - and it says nothing about what it writes. So nothing here is contradicted, and what the cell records is the edge of the contract rather than a defect the guards missed

4 — unreachable on this catalogue

the rule is real and no input in this catalogue tells the two apart. A larger catalogue would reach it, so the rule is kept and the cell records that nothing here exercises it.

cli-search · S-11 on S/as-committed

searches the bare name instead of the rendered address, so 1 in number/parse@1 is a word nothing carries. It survives, and the reason is the other rule: 1 is set aside, and what remains - number, parse - names the bare name in full, so the query answers anyway. The rendered address changes what the result scores and not whether it is one, and nothing in this catalogue can order it differently. Recorded rather than deleted: the address a tool prints is the address it should match, and the argument for that is not the score.

cli-search · S-12 on S/as-committed

stops splitting camel case. It survives because the split is applied to both sides: the query and the field go through one tokeniser, so removing it from the tokeniser removes it from both and they go on agreeing - groupBy typed becomes groupby, and the alias groupBy becomes groupby too. What that measures is the catalogue rather than the code: array/group-by@1 declares group by *and* groupBy, so neither spelling depends on the split. A contract declaring only one of them would, and no mutant here can arrange that.

cli-search · S-13 on S/as-committed

lets a summary be what a query names in full. It survives because no query covers a summary: naming one in full means typing every word of a sentence, and the shortest of the five is eighty-five characters. The exclusion is a statement about which fields are deliberate, kept because it is one, and its effect is unreachable on any catalogue whose summaries are sentences.

cli-search · S-14 on S/as-committed

strips a trailing s from any word at all, so is becomes i and as becomes a. It survives because neither i nor a is carried by anything those two queries would then reach: a is a word four summaries hold, so is would answer them - and no negative query here contains is without another word that already fails. The bound is kept for the same reason a bound is always kept, and what this records is that the catalogue does not currently contain the input that would show it.

1 — a declared open class

a limit this repository has written down, with its price, in the list of what it declares and nothing keeps. This is the only one of the five that is a debt.

string-slugify-spec · GS-11 on S/as-committed

the declared output alphabet widened to admit the underscore, while the implementation goes on never emitting one. It survives, and that is the finding this battery reports rather than repairs: the alphabet is checked in one direction only. Every property asks whether an answer falls outside the declared alphabet, and nothing asks whether the declared alphabet is wider than the answers need - so a contract can promise a larger surface than it keeps, and a caller who escapes on the strength of that promise is escaping more than they have to. The same hole would admit any character; the underscore is chosen because it is the one readers actually ask for

12 — only where a lens blinded the suite

the same defect dies on the column that reads the contract as committed, and lives only where a lens took part of the suite's sight away. That difference is what the lens exists to measure: it is what a contract without that half of its surface would have caught, which is nothing.

date-add · D-11 on C/reason-blind

no-input-validity-check: drops the early rejection of an Invalid Date input. NaN reaches the final range check by every path, so the value is unchanged; only the reason moves, from invalid-date to out-of-range

date-add · R-1 on C/reason-blind

right value, wrong reason: a field that is not a whole number is reported as out-of-range

date-add · R-2 on C/reason-blind

a plausible but false reason: an unknown field is reported as field-not-whole, which is what a developer would guess if they had to guess

date-add · R-3 on C/reason-blind

collapse: every reason reported as one. This is a diagnostic carrying exactly as much information as null, and a contract that cannot kill it has bought nothing but syntax

date-add · R-4 on C/reason-blind

the pair null renders indistinguishable: invalid-date and unknown-field exchanged. Under null this defect has no observable consequence whatsoever

number-parse · P-17 on C/reason-blind

memoises into a bare object, consulted first - so an inherited key such as "constructor" is served from Object.prototype. Measured, and the one place where publishing a reason changed which guard catches a defect rather than adding one: under the bare null convention the poisoned entry was the answer, and parseNumber("constructor") returned a function, so the named case caught it on the value. Here the poisoned entry is an analysis whose ok is undefined, so parseNumber answers null - which is the right answer - and only the diagnostic still sees anything, returning undefined where the contract requires "not-decimal". The blinded column is what this contract would catch if it published no reason at all: nothing

number-parse · N-1 on C/reason-blind

right value, wrong reason: overflow reported as not-decimal

number-parse · N-2 on C/reason-blind

a plausible but false reason: the empty string reported as not-decimal, which it also is

number-parse · N-3 on C/reason-blind

collapse: every reason reported as one. The form carrying exactly as much information as null while looking like it carries more

number-parse · N-4 on C/reason-blind

the separator literal declared and never produced: the second look is dropped and every separator mistake falls back into the residual reason. No value changes, because both branches refuse - this is what the contract looked like before the literal existed, and it is the shape a later optimisation of the refusing path would reach by accident

number-parse · N-5 on C/reason-blind

the separator family widened to whitespace, so "4 2" is reported as a separator mistake. It polices the line block 4.4 draws from the other side: the family is the formatting a document emits, never the ordinary space, and without this mutant that line would be a sentence rather than a measurement

string-slugify · G-21 on S/table-blind

transliterates Cyrillic to Latin, which is what three of the four measured libraries do and the single thing this contract exists to refuse. It is the only defect in this battery that the properties do not catch, and the reason is that it is not a broken fold: it answers a well-formed, lower-case, idempotent slug that retains a subsequence and carries no absorbable mark, so every property is satisfied, every profile keeps its class and every type assertion passes. One guard in the whole suite kills it, and it is a named case of block 4.4 - which is the measurement this lens exists to take, and the answer to what a table buys a contract that has no axioms to rest on

Questions rather than defects

A probe asks whether a region of a contract can be reached at all, rather than whether a defect in it is caught. It never enters the score, because a probe that survives is that question answered no — and folding it in would measure the question instead of the contract.

26 probe cells, 22 caught. The other 4 are 2 equivalent, 2 outside what the contract specifies. Each one is below, with the sentence its own battery carries about it.

2 cells of 1 defect — equivalent

the edit cannot change an answer, so no test could catch it. Measured rather than assumed: each one carries what was compared against what.

string-slugify · F-11 on S/as-committed and S/table-blind

trims the text before folding - the shape an implementation reaches when a leading separator was once a defect and trim was the fix that stayed. It survives, and that is the finding rather than a gap: every character trim removes is already a boundary, and a boundary before the first run or after the last one already produces nothing, so the edit cannot change an answer. It is a probe rather than a defect because an equivalent mutant that counted against the score would measure the question instead of the contract

2 cells of 1 defect — outside what the contract specifies

the edit changes something a caller could observe, and the contract deliberately makes no promise about it. What this marks is the edge of the specification, not a gap in the tests.

string-slugify · F-9 on S/as-committed and S/table-blind

never resets the base of the run at a boundary, so a mark that opens a run is absorbed by a letter from the previous one. It asks whether anything in this contract knows that a boundary ends a run for the purposes of the absorb step, and the answer is no: nothing reddens on either column. The region is reachable - a text of the shape e! followed by a combining acute answers differently under it - and neither the table nor any arbitrary draws that shape. What is missing is a case, and it is recorded here rather than added, because a probe that turns into a case stops measuring what the contract was

Guards no defect here reddens

Publishing these is the same decision as publishing the survivors. A guard that never goes red is either out of a battery's reach by construction, or it marks a region no defect has been written for yet — and those are different things, so each battery says which of the two each of its silent guards is. The second list is a measurement of the battery, not of the guards: what it asks for is more defects.

24 — out of a battery's reach by construction

45 — regions no defect here probes

Reading a contract with one eye shut

Some contracts are measured twice: once as they are, and once through a lens that takes part of the suite's sight away — the failure reason unread, the declared type unchecked, the table of settled cases blind. The difference between the two columns is what that half of the contract is worth, stated as a number instead of asserted. There are 24 such readings over the 19 batteries.

What you can check yourself

None of the following needs us to be honest, and none of it needs the registry to be reachable when you check. Each says what it establishes and, beside it, what it still does not.

What you have to take from us

The longer list, and that is the honest shape rather than a failure. Nearly all of it is the registry's own opinion — which implementation is recommended, what a machine measured, what we chose to call something. Two entries are outside what any arithmetic reaches, whatever anybody publishes. Where something narrows a claim, it is named; where nothing does, that is said rather than left blank.

Field by field

A contract is a record, and not every field of it is checked the same way. Each of the 61 fields carries the stratum it is verified at, so you are told which sentences no run could falsify instead of being left to assume they were all checked.

12 — executable

a guard in the contract's own suite fails when an implementation contradicts this. You can run that suite yourself, on the harness we serve, and see it.

identity.exportName, surface.exports[].name, surface.exports[].typeName, surface.exports[].text, surface.supportingTypes[].name, surface.supportingTypes[].text, properties.runs, caseTables[].cases[].data, benchmarks.vocabulary[].name, benchmarks.profiles[].class, benchmarks.profiles[].data, benchmarks.profiles[].samples.values[]

24 — structural

a malformed value here is refused without any implementation being involved. You can re-check it from the record alone.

address.language, address.name, address.major, surface.exports[].parameters[].name, surface.exports[].parameters[].type, surface.failureReasons[], properties.universal[].name, properties.universal[].applicable, properties.universal[].reason, caseTables[].groups[].id, caseTables[].cases[].group, caseTables[].cases[].id, caseTables[].cases[].provenance.kind, caseTables[].cases[].provenance.mutant, caseTables[].cases[].rationale, benchmarks.profiles[].samples.kind, benchmarks.profiles[].samples.count, benchmarks.profiles[].samples.encodedBytes, benchmarks.profiles[].samples.sha256, ownDeclarations[].executableBy.battery, ownDeclarations[].executableBy.guard, harness[].path, harness[].sha256, harness[].bytes

2 — one directional

a guard exists and keeps one half of what it appears to keep: the declared value may be wider than what the guard requires, and nothing notices. Read it as a claim, not as a check.

benchmarks.profiles[].name, benchmarks.profiles[].samples.producedBy

22 — documentary

published prose. Nothing can contradict it, because it makes no claim a run could falsify.

lifecycle.state, lifecycle.decidedAgainst, lifecycle.measurement, lifecycle.keptAs, lifecycle.answeredBy, identity.summary, identity.description, identity.inputDomain, identity.searchAliases[], identity.relationToTheLanguage, surface.exports[].role, surface.couplingRule, environments[], caseTables[].name, caseTables[].purpose, caseTables[].groups[].title, caseTables[].groups[].note, caseTables[].cases[].provenance.report, benchmarks.vocabulary[].meaning, benchmarks.profiles[].description, ownDeclarations[].name, ownDeclarations[].verification

1 — stated per declaration

what refuses a wrong value here depends on which declaration it is, and the record carries the answer beside each one.

ownDeclarations[].value

The draws are re-seeded every run

The property tests are not run against a frozen sample: a property whose draws are frozen with the major explores one sample set for ever and can be passed row by row by a submitter who knows it, which is what re-seeding every run exists to prevent. So your run is a different sample of the same properties, and a run of ours passing is not a promise that yours will draw what ours drew.

A verdict is taken from 1 run and an attribution — which guard caught which defect — from the intersection of 3. Measured: three complete runs of the string-slugify battery, 58 cells each: 0 verdicts moved and 6 attributions moved, on G-07, G-22 and F-10, each on both lenses. Every pinned guard held. Earlier, at fc51b00: L-18 red on 23 runs of 25 and G-22 on 8 of 9, measured while checking declared pins. The intersection of three runs is what the batteries already take.

What a signature does not prove

a signature attests who published this snapshot and from what build; the contract's verification says an implementation answers the contract; neither says the contract is the right specification

Back to the catalogue