DataEngine
Donkeh.DataEngine is the bridge between a token list and the
flat DataTable. It is a single forward pass over tokens:
no source text ownership, no AST, no backtracking, no re-tokenization.
It also owns the condition-compiler hook that turns a query prelude into a
predicate for the table.
What the engine owns
The parser below this layer has already removed whitespace and comments and classified every token. DataEngine’s job is structural: decide whether a token run is a rule, a declaration, an at-rule, a nested rule or a descriptor, then write ids into the table. The source string is not retained, and selector or condition token sequences are copied into the table’s arenas so the returned table is independent of the token list.
DataEngine itself is tiny and immutable. It stores one
ConditionCompiler and exposes three entry points:
Build(string), Build(List<Token>) and
Build(List<Token>, DataTable). The last one appends several
stylesheets into the same table, preserving call order for the cascade.
No compiler means AlwaysFalseCompiler, so gated rules fail closed
rather than applying by accident.
A ref struct, one pass
CssStreamParser is a ref struct that borrows the
token span and the destination table. It is stack-only, cannot cross an
await, and owns a pooled character buffer that must be returned.
That shape is deliberate: it keeps the parser allocation-light while making
the lifetime of its borrowed inputs explicit.
| State | Purpose |
|---|---|
_tokens | Borrowed token span. Original source offsets survive so gaps can be detected. |
_table | Destination table, mutated in place. |
_compiler | Invoked once per distinct condition prelude. |
StreamContext | Nested condition, container and rule frames plus a flat selector stack. |
_layerStack | Cascade layers currently entered, outermost first. |
_composer | Reused token buffer for parent/child selector composition. |
_selectorScratch | Selector ids for one rule, de-duplicated with a linear scan. |
_textBuffer | ArrayPool-backed scratch for property and value normalisation. |
The constructor rents 256 characters from
ArrayPool<char>.Shared; Dispose returns it.
Everything else is reused across the stylesheet, including the selector
composer and the selector-id scratch list. The table is pre-sized from
tokens.Count / 8 so most growth reallocations are avoided
without keeping a per-token buffer.
Blocks, statements and local recovery
Parse calls ParseBlock(topLevel: true), which loops
until EOF. A } closes a non-top-level block; stray closing braces
at top level are consumed and ignored. Semicolons at block level are skipped.
At-keywords go to ParseAtRule; everything else goes to
ParseSegment.
ParseSegment first checks for a custom property declaration
(--name:) before looking for the rule/declaration split. That
ordering matters because a custom-property value may contain braces, and it
must not be mistaken for a nested rule. Otherwise
FindHeaderEnd scans for the first top-level {,
; or }:
| Terminator | Interpretation |
|---|---|
{ | A style rule; the prelude is the selector list and the block is parsed recursively. |
; or } | A declaration or malformed segment; declaration parsing or local skip recovery. |
| EOF | End of input; a truncated statement is abandoned. |
Error recovery is intentionally local and silent. A malformed declaration is
skipped to the next top-level ;, { or },
parenthesis- and bracket-aware so a semicolon inside url(...) or
an attribute selector does not end the skip. A malformed selector prelude
still consumes its block, but pushes a container frame that hides the enclosing
selector, so declarations cannot leak onto the parent. An at-rule whose
prelude ends at EOF or at a closing brace is abandoned without consuming the
brace. No parse exception is reported to the caller.
At-rules and the condition chain
ParseAtRule finds the prelude terminator first. A semicolon
terminates statement at-rules; a brace starts a block; EOF or an enclosing
closing brace abandons the at-rule. Block at-rules are dispatched in a fixed
order:
@nest— parsed as a nested style rule.- Conditions —
@media,@supports,@container,@document,@-moz-document,@starting-style. @keyframesand@-webkit-keyframes.@scope,@layer.@pagemargin boxes and descriptor at-rules.- An unknown container, traversed with the selector context hidden.
Only condition blocks keep the enclosing selector context. A rule inside
@media therefore inherits the selector of the rule it is nested
in; a rule inside @layer, @scope or an unknown
container does not accidentally attach declarations to the parent. Conditions
call DataTable.GetOrAddCondition with the current condition id as
the parent, so nesting becomes an AND chain.
@media (min-width: 768px) {
@media (orientation: portrait) {
.card { color: red; }
}
}
// condition ids form a parent chain:
// root -1
// └── (min-width: 768px)
// └── (orientation: portrait)
// both executors are compiled once and cached.
The parser stores the outermost condition id on each rule. Evaluation walks
the parent chain and stops at the first failure. Conditions are interned by
(tokens, parent), while executors are interned by tokens alone,
so the same media query nested under different parents shares one compiled
delegate. The engine never interprets the query text during matching.
Selector list flattening and nesting
ParseRule splits a selector prelude on commas that are not inside
parentheses or brackets, so :is(.a, .b) and
[data-x="a,b"] stay intact. Each branch is flattened against
every selector of the enclosing rule. At the root the branch is interned
verbatim; inside a nested rule the result is the cartesian product of parent
and child selectors, followed by a linear de-duplication pass because selector
lists are short.
Compose preserves CSS nesting semantics. A child without
& becomes a descendant of the parent. Each &
is replaced in place by the entire parent selector. Whitespace is reconstructed
as a synthetic gap, so & .a stays a descendant while
&.a stays one compound. A leading & never
introduces a leading gap. Composed tokens are copied into the table before the
next pair, so the composer buffer can be reused for the entire stylesheet.
.card, .panel {
& > .title { color: red; }
.body { color: blue; }
}
// flattened:
// .card > .title, .panel > .title
// .card .body, .panel .body
If no selector survives flattening, the block is still consumed as a container so the token stream stays synchronized. That is the same “hide the parent” rule used for malformed rule headers.
Declarations and normalization
ParseDeclaration locates the top-level colon with
FindColon, then scans the value with FindValueEnd.
Both scans track parentheses, square brackets and — for values — balanced
curly braces, so function arguments, attribute selectors and custom-property
blocks do not terminate early. The parser then chooses one of four paths:
| Context | Destination |
|---|---|
| Descriptor at-rule | CssDescriptor, or CssMarginBox inside an @page margin box. |
Inside @keyframes | One CssKeyframe per current keyframe selector, with the condition id. |
| No selector or empty value | Discarded without interning. |
| Normal style rule | One Rule per selector id, carrying property, value, condition and layer ids. |
Property names are ASCII-lower-cased unless they are custom properties.
Values preserve case because CSS values can be case-sensitive. Token text is
written into the pooled character buffer with one space wherever the original
source had a gap; comments and whitespace never appear as tokens, so a gap is
simply previous.EndIndex < current.StartIndex. A single-token
property or value whose interned text already has the right casing reuses its
existing id and never touches the buffer.
NormalizeAndIntern fast path checks for exactly one token and
returns its interned id directly. Multi-token values such as
0 8px 16px rgba(0,0,0,.2) pay for normalization once, then all
repeated uses share the resulting id.
Layers, keyframes and descriptors
@layer blocks push an id onto a layer stack; every rule created
while that frame is active records CurrentLayerId. Statement
layers and block layers both call GetOrAddLayer, so the layer
order is assigned once and reused. Unlayered rules keep -1, which
Data treats as stronger than every layered rule.
@keyframes sets _inKeyframes and records the
animation name. Its inner selectors (from, to,
50%) are interned like normal selectors, but each declaration
becomes a CssKeyframe rather than a matchable style rule.
Descriptor at-rules such as @font-face, @page,
@property, @counter-style,
@font-palette-values and @view-transition store their
prelude as descriptor context and their declarations as
CssDescriptor records. @page margin boxes such as
@top-center become CssMarginBox records.
@import, @charset, @namespace and
@scope are recorded as metadata. Imports are never fetched;
resolving them is the host’s job. Scope preludes are stored as interned text,
not rewritten into selectors.
Performance and allocation
The engine is built around reuse and one-way movement. There is no AST to allocate, no backtracking cursor to reset and no per-selector token array. The only large temporary is the token list produced by the tokenizer; the parser itself reuses its composer, selector scratch, layer stack and pooled text buffer.
| Bootstrap 5.3.8 | Result |
|---|---|
| Input | 232,108 characters, 46,438 tokens |
| Output | 2,762 selectors, 6,496 rules, 18 conditions, 8 keyframes, 1 charset |
| Tokenize + build | ~13 ms total, ~17.7 MB/s |
| Transient allocation | ~2.4 MB for tokens, parsing and interning |
| GC collections | 0 / 0 / 0 in the debugger run |
| Tokenizer-only baseline | ~5.5 ms of the ~13 ms total |
The benchmark suite scales the same pipeline over 1M, 4M and 16M character synthetic corpora, with a second corpus that repeats selectors and conditions to stress identity lookup and dedup. The query benchmarks separately measure rule scans, discriminator lookups and cached versus invalidated condition evaluation, so build cost and match cost are not conflated.
Trade-offs and sharp edges
| Decision | Cost |
|---|---|
| Single forward pass, no AST | Very little temporary structure, but debugging is token-index based and there is no rich parse tree to inspect. |
| Local recovery, no diagnostics | Malformed sections are skipped or hidden; nothing is reported to the caller. A typo can look like a missing rule. |
| Table mutation is eager | The table is complete only when the parse finishes successfully; there is no rollback if a later section fails. |
ref struct parser | Stack-only and allocation-light, but cannot be stored in a field, used across await or shared between threads. |
| Condition compiler is injected | Media queries are supported by one compiler; @supports and @container need a host-supplied compiler or fail closed. |
| Nesting is flattened at build time | Matching stays simple, but deeply nested selector lists can multiply quickly before deduplication. |
| Normalization is pooled, not free | Multi-token values copy into the text buffer; repeated values then share one interned id. |
Code tour
| File | Read it for… |
|---|---|
DataEngine/Source/DataEngine.cs | The three build entry points, table pre-sizing and the compiler injection point. |
DataEngine/Source/CssStreamParser.cs | The entire forward-only parser: blocks, at-rules, rules, nesting, declarations, recovery and normalization. |
Data/Source/StreamContext.cs | Condition/container/rule frames and the flat selector stack used for nesting. |
Data/Source/DataTable.cs | InternSelector, GetOrAddCondition, AddRule, layers and metadata tables. |
DataEngine/Source/MediaConditionCompiler.cs | Token-level media-query compilation into ConditionFunction. |
DataEngine/Benchmarks/Benchmarks.cs | The 1M/4M/16M corpora, repeated-selector stress test and query benchmarks. |
The mental model is a one-way funnel: the tokenizer decides what each token is; DataEngine decides how tokens relate structurally; Data stores those relationships as integer edges and makes them queryable. Nothing in the pipeline goes back to the source text after tokenization.