Two lists, both maintained by audit rather than by memory: where the documents outside Superseded disagree with each other, and what has to be settled before a first Rust prototype can be built.
The prototype in question is one that can create a file, open it, check the schema, hand out a Kladde<T>, accept mutations and record them in the journal, and flush.
Segments are deferred, and so is anything that only affects how well it performs.
Contradictions
Ordered most to least severe, where severity means how likely this is to make someone write the wrong code.
1. The journal is two different things — resolved
Was: Transactions and batches treated the journal as a fixed-capacity allocation, grown by being freed and re-allocated while empty, while Durability and File format described a chain of journal pages drawn from the reusable-page pool.
Resolved in favour of the chain. Journal now specifies the page layout, and Transactions and batches treats the segment’s page budget as a flush trigger rather than a capacity: a transaction is appended whole, taking as many pages as it needs, and the flush that follows folds it.
2. Journal pages cannot satisfy the page framing — resolved
Was: page framing required every non-header page, Journal kind included, to carry a CRC over its content, while journal appends were exempted from whole-page writes — and a page appended to incrementally cannot keep a valid CRC unless every append rewrites it.
Resolved by scoping the framing to “every page except the pages that make up the current journal” and dropping Journal from the kind values.
Journal pages now carry no framing at all, and their integrity rests solely on the chained, epoch-salted transaction CRCs — which is what How much of the framing is load-bearing already implied and now says.
Kept here, struck through, for one more audit; the shape of a journal page is now specified as well.
3. The durability promise is stated unqualified where users read it
Durability is careful: under a power cut, transactions issued after the last completed flush survive only up to a valid journal prefix, “minus whatever the operating system never wrote out”, because journal appends are not fsynced.
The two places an application author actually reads make the unqualified promise:
- the hub page — “an append-only on-disk journal that is durable by the time the mutating call returns”;
- the tutorial — “Nothing that has been acknowledged. If
pushreturned, the element is in the file.”
Both are true for an application crash and false for a power cut. This is severe not because it is subtle but because it is a promise, and because the tutorial is where someone decides whether kladde is safe enough for their use.
4. Framing is per-record in one place and per-transaction in another
Journal and Conformance frame transactions, with a chained epoch-salted CRC. The tutorial frames records: “Every record carries a length prefix and a checksum.”
A format-level disagreement, and the two imply different recovery code. The tutorial’s version also predates the chaining, so it describes recovery as truncating a torn record rather than as keeping the longest valid prefix.
5. mentions is claimed to have one reader and has at least three
Liveness states flatly: “This is the only reader of mentions”, meaning the release of a tombstone anchor’s pin at mentions == 1.
In-memory state repeats it as “one thing only”.
But Id recycling reads it twice — to prefer ids whose tombstone is already dead, and as the tiebreak that argues for taking high-mentions ids first — and Liveness’s own ranking section reads it again, to score “dead statements whose removal would let a tombstone retire”.
The claim matters because it is the argument for how cheap mentions is.
With three readers the field is harder to remove than the text suggests, and anyone auditing whether it can be dropped will reach the wrong conclusion.
6. Reclamation is optional and also required
Allocations says whether an implementation reclaims garbage is unconstrained, and that one which never consolidates is conforming.
Guarantees says “reclamation is incremental by construction” as though it were a property the spec asserts.
What remains is the wording of the guarantees row: the resolution is presumably that incrementality is required of whatever reclamation an implementation does, while doing any is not.
7. Conversion is gone, and the flush still reasons about it
Sizedness conversion was removed along with sizedness, taking the Convert record with it (the superseded heap).
The flush still uses it as the motivating example in three load-bearing places: the three-cycle that forces frees-first to be a priority rather than an edge, the content-blind fast path’s flag condition, and the implementation order’s step 5.
The arguments themselves survive — Alloc(B); Copy(A→B); Free(A) has the same shape as the conversion it describes — so this is stale vocabulary attached to live reasoning rather than a collapsed argument.
It is listed here because a reader who checks whether conversion exists will conclude the section is obsolete, which it is not.
8. “Five layers”, six rows
The layers says a kladde file is described by five layers and then tabulates six. Trivial, and only listed because the table is the first thing a new implementer reads.
9. Which “read” the complexity bound governs — narrowed, not closed
Guarantees requires “read any byte of any allocation” in O(log F), and now also that “a non-mutating operation once the file is loaded” have no overhead over a non-persisted type.
The second row is the application read, so the ambiguity the first row used to carry is mostly gone.
What remains: the first row still does not say it means the storage-level read, and it counts only the fragment-map lookup.
If data pages are not resident after loading, reaching the bytes is a pread rather than a dereference, so the honest bound is O(log F) lookups plus whatever fetching the bytes costs.
Worth saying, since the whole point of the row is to tell an implementer what is achievable.
10. Grouping is “free” but constrained
What is fixed and what is free lists batching — “how an implementation groups application operations into transactions” — as explicitly not fixed. Transactions then requires that a record occurring outside any transaction be recorded as a transaction of its own.
That is a real constraint on grouping: an implementation may not leave a record ungrouped. The intent is clearly “how you batch is free, but everything lands in some transaction”; the wording does not say so.
Blockers for a first prototype
Ordered most to least difficult. Difficulty here means design work plus implementation risk, not lines of code — several large items are only large.
Items marked deferrable are not blockers; they are listed so that the line between them and the blockers is explicit.
1. The journal’s on-file representation — resolved
Was: unspecified beyond “the header names a journal segment’s start page”.
Resolved by Journal: a segment is a singly linked chain of pages, each reserving an 8-byte trailer for the next page’s number and a checksum; transactions and records may span pages; and every checksum is a value of one CRC-32C chain salted with the segment’s epoch.
2. The byte encoding of journal records — resolved
Was: the record set had semantics but no bytes.
Resolved by Records and Framing.
3. Bootstrapping — partly described
Still open in Header pages: what the first header’s address-table payload contains, how the root and schema-table allocations come into existence before an address table describes them, and whether creation is a degenerate flush or a distinct path.
4. File extension is not covered by the durability argument
The reuse rule permits “extend the file”, and I1 says a header is issued only after an fsync covering everything it references.
But extending a file changes metadata, and on common filesystems the new length is not durable merely because the data blocks were fsynced — it can require fsyncing the directory, or an explicit allocation step. A power cut can therefore leave a valid header referencing a page beyond the file’s recovered length, which breaks I1 in exactly the way the design says cannot happen.
Small to fix, easy to miss, and it invalidates the central invariant if missed — which is why it ranks above items with far more code.
5. The header’s byte layout
Header pages lists the fields and says “exact widths, ordering, and the reservation of space for future roots are TBD”. Blocking for both creation and open, and it is the one structure whose layout can never change, so the reservation decision has to be made now rather than discovered later.
6. The pointer encoding inside allocations
Pointer encoding is TBD: width, null representation, alignment.
This blocks the very first derived struct with a container field, because storing a PersistableVec means writing a pointer into the parent’s bytes.
The reference implementation’s choice — 32-bit, zero reserved for null — is probably just right, but it needs to be decided, since it is a format-level fact that readers depend on.
7. The in-memory address table
The largest body of new code, and the only large item that is purely implementation: Address-table operations already gives pseudocode for load, resolve, read, the fragment-map primitives, apply-a-statement, resize, free, page rewrite and eviction.
Risk is concentrated in two places — the partition invariant of the fragment map, where a bug returns a neighbour’s bytes rather than failing, and the anchor replacement obligation on page rewrites, where a bug silently resurrects truncated data. Both deserve assertions from the first commit rather than tests added later.
8. The fold, plus its differential oracle
The flush is fully designed. A prototype needs only the naive in-order replayer and the piece-table fold that must agree with it — but the oracle comes first, and building it in the wrong order is the documented way to get this wrong.
The scheduler is explicitly not needed: hoisting collapses the graph, and the recommendation is to skip it until a workload proves otherwise.
9. Recording every mutation
The current JournaledWriteBackend does not, which is the one defect that is not a performance matter.
Design work is zero — the log is the sole authority — but every container and the derive macro have to be revisited to confirm each mutation actually emits its records, and nothing mechanically checks this yet.
10. Freeing
Freeing is designed and not built, so today a dropped or overwritten value orphans its allocation.
A prototype can ship leaking — it is harmless against a file that is being exercised rather than kept — but it interacts with the prototype’s other goals: a leak makes the file grow, which exercises consolidation, which the prototype is otherwise entitled to skip. Worth deciding deliberately rather than by omission.
11. Choosing values for things that only need a value
Each of these blocks the prototype only in the sense that a number must be typed:
- page size — take 4 KiB and defer the 16 KiB measurement;
- the
Inlinethreshold — the documented starting policy of ~64 bytes; - the journal segment’s page budget and the batch size — see flush triggers;
- consolidation constants — see below.
Deferrable, and why
- Data-page consolidation. The reverse-index gap is unresolved, but a prototype may consolidate address-table pages only, which are self-describing, or nothing at all. Garbage accumulates; nothing breaks.
- Schema evolution. The prototype checks the schema, which means comparing the root fingerprint and failing closed. Resolution is a separate and much larger feature.
- A flush that re-points moved bytes. The
Moverecord is specified, and a prototype can fold it like aCopyfollowed by zeroing the source range, giving up only the rewrite it would save; re-pointing the bytes instead can wait. - Non-owning references, application versioning, and concurrency. All marked TBD and none reachable from the prototype’s feature list.
- Unwind behaviour. Poisoning on panic is aspirational; the interim contract — reopen the file — is adequate for a prototype.