D Donkeh docs
GitHub

Parser

The CSS parser is a single-pass lexer wrapped around a declaratively registered exact-match engine. The outer scan stays hand-written and allocation-conscious; the keyword, property, function and pseudo-selector tables are compiled into specialised IL once, on first use.

The design goal

The target was an unusual combination: the authoring ergonomics of a rule-based tokenizer with the execution profile of hand-written code. Rule engines usually pay for their flexibility with intermediate representations, dictionary probes, virtual dispatch or an interpreter in the hot loop. None of those are acceptable when a stylesheet is re-tokenized often enough to show up in frame time.

The split that made it work is simple. The declaration of lexical knowledge is separate from its execution. Rules are registered through small helpers, then a runtime IL compiler turns that registration table into a dense first-character dispatch table and one specialised matcher per first character. After type initialisation there is no rule interpreter, no name lookup and no general-purpose matching engine between the input span and the generated code.

It is a runtime compiler, not a build-time source generator The generated code is emitted with System.Reflection.Emit from the tokenizer's static constructor. The rule definitions themselves are ordinary C# registration calls in Rules/*.cs, Tokenizer/PseudoClasses.cs, PseudoElements.cs and Functions/*.cs. A future build-time generator could bake the same matcher into the assembly, but the current implementation trades a small first-touch cost for a tiny, declarative source surface.

Two layers, one pass

Tokenizer.Parse runs a hand-written switch over the current character. That outer loop owns everything structural: whitespace, comments, strings, numbers, dimensions, hashes, at-keywords, punctuation, escapes, CDO/CDC and the declaration-colon disambiguation. It emits at most one token per iteration.

Names are the only place where the lexer needs a large table of lexical knowledge. Once ScanName has consumed a maximal identifier, the exact matcher decides whether it is a property name, a function name, a pseudo-class, a pseudo-element or just an identifier. That boundary is deliberate: the hot structural scan stays obvious and debuggable, while the large, repetitive keyword table is generated.

Tokenizer.Parse(ReadOnlySpan<char>) └── hand-written scan ├── whitespace / comments / CDO / CDC ├── strings, escapes, numbers, dimensions, percentages ├── punctuation and combinators └── ScanName(...) └── ClassifyName(...) ├── '(' lookahead → Function ├── MinLengthByFirstChar early-out └── ExactMatch(ReadOnlySpan<char>) // generated IL └── first-character group → length switch → trie

The declarative rule set

RuleBuilder exposes the authoring API: Exact, CssPropertyName, CssPropertyGroup, CssPseudoClass, CssPseudoElement and Function. The per-letter PropertiesA() through PropertiesZ() methods are just registration calls, grouped so the source stays auditable and diff friendly.

CssPropertyName("aspect-ratio");

CssPropertyGroup("border",
    "width", "style", "color",
    "top", "right", "bottom", "left",
    "radius");

CssPseudoClass("focus-visible");
CssPseudoElement("placeholder");
Function("linear-gradient");

The current table contains 795 exact rules:

Rule kindCountExample
CSS property names577grid-template-columns
Functions100clamp, linear-gradient
Pseudo-classes78:focus-visible
Pseudo-elements30::placeholder
Punctuation rules10:;(){}[]>

The 577 property names demonstrate why the table is generated rather than hand-coded. The layout engine only implements a subset (168 registered properties today), but the tokenizer still recognises the wider CSS vocabulary so later stages can make the policy decision. A name being tokenized as a property does not mean its value grammar is fully supported.

Duplicate kinds are resolved deliberately A name can be both a property and a function, such as color, rotate or scale. RuleBuilder.Exact lets the property registration win, and ClassifyName upgrades the token to TokenKind.Function when the next character is (. The lexer never has to guess from the name alone.

Compiling the matcher

The static constructor in Rules/Rules.cs registers every rule, computes a 65,536-entry MinLengthByFirstChar table, and calls Compiler.CompileExactRules. Compilation groups the rules by first character and emits one DynamicMethod per group.

Compiled shapeValue
Distinct first-character groups34
Single-rule groups (skip the length switch)11
Multi-rule groups23
Largest group110 rules, starting with :
Shortest / longest rule text1 / 39 characters

The public entry point is a dense array of delegates plus the lowest first character covered by the table. An empty span or an out-of-range first character returns -1 without calling generated code:

if (span.Length == 0)
    return -1;

var index = span[0] - _minChar;
if ((uint)index >= (uint)_groups.Length)
    return -1;

return _groups[index](span);

Inside a group, generated IL switches on exact length first. This is the key to longest-match behaviour and to killing prefix ambiguity: border and border-radius are different rules with different lengths, so a six-character candidate can never fall through to a shorter rule. The lexer already scanned the maximal identifier, so the matcher receives the complete candidate rather than a prefix.

// Conceptually, per first-character group:
switch (span.Length - minLength)
{
    case 0:  return MatchLength(minLength);
    case 6:  return MatchLength(minLength + 6);
    case 13: return MatchLength(minLength + 13);
    default: return -1;
}

// Within one length bucket, longer prefixes are emitted first.
// A node with more than six candidates switches on the next character;
// a smaller node becomes a short SequenceEqual chain.

The trie only splits when a node still has more than six candidates. Below that threshold, a linear chain of MemoryExtensions.SequenceEqual<char> calls is smaller and often faster than another switch. Single-rule groups skip the length switch entirely, which keeps punctuation buckets tiny. The compiler also emits longer buckets before shorter ones so the generated method layout favours the more specific rules.

Why one method per first character? The compiler comments call out a measurable JIT cliff: a single matcher of roughly 20 KB was slower than a table of small methods. Splitting by first character keeps each generated method below the large-method optimisation threshold, gives the JIT a simple constant range for its switch tables, and makes the hot path a predictable array index plus a short delegate call.

What the hot path actually does

For a property-heavy stylesheet, most name classification follows this sequence:

  1. ScanName walks the maximal identifier, handling escapes and non-ASCII characters without allocating.
  2. ClassifyName checks whether the next character is (. If so, the token is a function and the exact table is skipped entirely.
  3. MinLengthByFirstChar[first] rejects candidates shorter than any rule with that first character. Single-letter identifiers never enter the generated matcher.
  4. ExactMatch indexes the first-character group, switches on length and walks a small character trie or comparison chain.
  5. The result is written into Token as a packed (start, length) location plus an interned text id.

The submitted character stream is never sliced into temporary strings. A token stores only the interned id and a 32-bit location: 24 bits of start offset and 8 bits of length. A Token is 12 bytes, so a megabyte-scale stylesheet with roughly 380,000 tokens uses about 4.6 MB for the token buffer before list overhead.

The 255-character and 16.7-million-character limits are real The packed location cannot address a source longer than 16,777,215 characters. Tokens longer than 255 characters, such as a large data: URI or string, are split into consecutive chunks that share a kind; downstream consumers concatenate chunk text to recover the original value.

Allocation behaviour

The tokenizer consumes ReadOnlySpan<char> and advances by index; it does not materialise an intermediate token-text array. The token destination is a List<Token> written through CollectionsMarshal.AsSpan, so the loop avoids List<Token>.Add and its bounds/count bookkeeping.

MechanismWhat it buys
ReadOnlySpan<char> + indicesNo substring allocation for scanning, matching or token boundaries.
Packed Token.Location12-byte tokens instead of separate start/end integers.
Pre-sized token bufferInitial capacity is max(16, sourceLength / 2 + 16); the benchmark corpus needs no growth.
CollectionsMarshal.SetCountWrites directly into the backing array and trims to the exact count once at the end.
Interned textRepeated names and punctuation become stable integer ids used by later stages.

The honest caveat is that “zero allocation” describes the steady state, not the first time a document introduces a new token text. InternTable allocates the string on first sighting, then stores it in a dense global table. Subsequent lookups use a thread-static front cache and AlternateLookup<ReadOnlySpan<char>>, so the common case does not need a lock or a new string. Single ASCII punctuation characters use a pre-computed 128-entry id table, which removes the global lock from the most frequent tokens in CSS.

The table never evicts. Its size follows the number of distinct token texts, not the number of tokens, which is the right trade-off for finite stylesheets but something to keep in mind for long-running apps that generate unbounded CSS text. Reusing a cleared token list avoids the one large backing-array allocation per document.

Performance

The parser benchmark has two measurements: the exact matcher over a hot token stream, and a full Tokenizer.Parse over a synthetic stylesheet built from every registered rule. The quick runner uses warmup plus repeated runs, and reports the best run.

dotnet run --project Donkeh/Parser/Benchmark/Benchmark.csproj \
  -c Release -- --quick
MeasurementResult
Corpus1,013,028 bytes, 381,640 tokens
Full parse time4.97 ms per document
Document throughput203.8 MB/s
Token throughput76.8 M tokens/s
Steady-state allocation0 B per parse
Exact matcher4.81 ns/token, 207.9 M tokens/s
Matcher text throughput1,926.4 MB/s
Matcher allocation0 B per pass
IL compilation0.98 ms, 484,952 B, 0 GC collections

These numbers were measured on .NET 10.0.10, Windows 10, a 12-logical-core workstation with workstation GC in interactive latency mode. They are a shape, not a promise: token distribution, CPU, GC mode and whether the intern table is warm all move the result. The important part is the ratio between full-document throughput and raw matcher throughput: the structural scan and token writing dominate, while keyword classification is already in the low-single-digit-nanoseconds-per-token range.

Why the compiler cost is acceptable Compilation is a one-time type-initialisation cost, not a per-document cost. The quick benchmark measures roughly a millisecond and half a megabyte for the IL pass after the rules are registered. In a desktop app this happens once before the first stylesheet is parsed; the alternative is interpreting a 795-rule table on every token.

Trade-offs and sharp edges

DecisionCost or limitation
Runtime IL instead of a build-time artifactAdds a small first-touch cost and depends on DynamicMethod. A build-time generator would remove both and be friendlier to ahead-of-time publishing.
One generated method per first characterMore methods and delegates to manage, but each method stays below the JIT large-method threshold.
Exact matching, not prefix matchingRequires the scanner to consume the maximal identifier first. That is exactly what makes longest-match correct and keeps the generated trie small.
Linear comparison below six candidatesDeliberately trades a few comparisons for smaller code and better instruction-cache behaviour.
Global interningIds are stable for the process but the table grows forever; it is not a cache with eviction.
Packed token locationExcellent density, but a 16.7M-character source limit and 255-character token chunks are visible constraints.
Hand-written structural lexerSome CSS context is resolved with look-behind, for example distinguishing a declaration colon from a pseudo-class colon after a property name.

One deliberately non-obvious case is background-position:right with no whitespace after the colon. A context-free lexer would see :right as a pseudo-class. IsDeclarationColon checks whether the previous token is a property name preceded by { or ;, and emits a plain colon instead. Selectors keep their pseudo-class token because their preceding tokens do not match that shape.

Code tour

FileStart here for…
Parser/Source/Tokenizer.csThe outer scan, token emission, capacity growth, strings, numbers and escapes.
Parser/Source/Rules/Rules.csThe static constructor, registration order and MinLengthByFirstChar setup.
Parser/Source/Rules/*.csPer-letter property registrations. Diff-friendly, no runtime logic.
Parser/Source/Tokenizer/PseudoClasses.csThe 78 pseudo-class names, registered as complete :name rules.
Parser/Source/Tokenizer/PseudoElements.csThe 30 pseudo-element names, registered as complete ::name rules.
Parser/Source/Functions/*.csThe function-name table, including property/function overlaps.
Parser/Source/Builder/RuleBuilder.csThe declarative authoring API and duplicate-kind policy.
Parser/Source/Compilation/Compiler.csGrouping, length switches, trie emission and the first-character jump table.
Parser/Source/Compilation/ILGeneratorWrapper.csReadable IL emission and the optional IGNITE_SHOW_IL=1 dump.
Parser/Source/Objects/Token.csThe 12-byte token and its packed 24/8-bit location layout.
Parser/Source/InternTable.csDense ids, thread-local caching, ASCII fast path and span lookups.
Parser/Benchmark/Program.csThe quick benchmark, corpus builder and allocation columns.

The useful mental model is: the scanner decides where a token ends, and the generated matcher decides what a maximal name means. Keeping those responsibilities separate is what allows the declarative rule table to be large without putting an interpreter in the per-character path.