Parser
The CSS parser is a single-pass lexer wrapped around a declaratively registered exact-match engine. The outer scan stays hand-written and allocation-conscious; the keyword, property, function and pseudo-selector tables are compiled into specialised IL once, on first use.
The design goal
The target was an unusual combination: the authoring ergonomics of a rule-based tokenizer with the execution profile of hand-written code. Rule engines usually pay for their flexibility with intermediate representations, dictionary probes, virtual dispatch or an interpreter in the hot loop. None of those are acceptable when a stylesheet is re-tokenized often enough to show up in frame time.
The split that made it work is simple. The declaration of lexical knowledge is separate from its execution. Rules are registered through small helpers, then a runtime IL compiler turns that registration table into a dense first-character dispatch table and one specialised matcher per first character. After type initialisation there is no rule interpreter, no name lookup and no general-purpose matching engine between the input span and the generated code.
System.Reflection.Emit from
the tokenizer's static constructor. The rule definitions themselves are
ordinary C# registration calls in Rules/*.cs,
Tokenizer/PseudoClasses.cs, PseudoElements.cs and
Functions/*.cs. A future build-time generator could bake the
same matcher into the assembly, but the current implementation trades a
small first-touch cost for a tiny, declarative source surface.
Two layers, one pass
Tokenizer.Parse runs a hand-written switch over the current
character. That outer loop owns everything structural: whitespace, comments,
strings, numbers, dimensions, hashes, at-keywords, punctuation, escapes,
CDO/CDC and the declaration-colon disambiguation. It emits at most one token
per iteration.
Names are the only place where the lexer needs a large table of lexical
knowledge. Once ScanName has consumed a maximal identifier, the
exact matcher decides whether it is a property name, a function name, a
pseudo-class, a pseudo-element or just an identifier. That boundary is
deliberate: the hot structural scan stays obvious and debuggable, while the
large, repetitive keyword table is generated.
The declarative rule set
RuleBuilder exposes the authoring API:
Exact, CssPropertyName,
CssPropertyGroup, CssPseudoClass,
CssPseudoElement and Function. The per-letter
PropertiesA() through PropertiesZ() methods are
just registration calls, grouped so the source stays auditable and diff
friendly.
CssPropertyName("aspect-ratio");
CssPropertyGroup("border",
"width", "style", "color",
"top", "right", "bottom", "left",
"radius");
CssPseudoClass("focus-visible");
CssPseudoElement("placeholder");
Function("linear-gradient");
The current table contains 795 exact rules:
| Rule kind | Count | Example |
|---|---|---|
| CSS property names | 577 | grid-template-columns |
| Functions | 100 | clamp, linear-gradient |
| Pseudo-classes | 78 | :focus-visible |
| Pseudo-elements | 30 | ::placeholder |
| Punctuation rules | 10 | :;(){}[]> |
The 577 property names demonstrate why the table is generated rather than hand-coded. The layout engine only implements a subset (168 registered properties today), but the tokenizer still recognises the wider CSS vocabulary so later stages can make the policy decision. A name being tokenized as a property does not mean its value grammar is fully supported.
color,
rotate or scale. RuleBuilder.Exact
lets the property registration win, and ClassifyName upgrades
the token to TokenKind.Function when the next character is
(. The lexer never has to guess from the name alone.
Compiling the matcher
The static constructor in Rules/Rules.cs registers every rule,
computes a 65,536-entry MinLengthByFirstChar table, and calls
Compiler.CompileExactRules. Compilation groups the rules by
first character and emits one DynamicMethod per group.
| Compiled shape | Value |
|---|---|
| Distinct first-character groups | 34 |
| Single-rule groups (skip the length switch) | 11 |
| Multi-rule groups | 23 |
| Largest group | 110 rules, starting with : |
| Shortest / longest rule text | 1 / 39 characters |
The public entry point is a dense array of delegates plus the lowest first
character covered by the table. An empty span or an out-of-range first
character returns -1 without calling generated code:
if (span.Length == 0)
return -1;
var index = span[0] - _minChar;
if ((uint)index >= (uint)_groups.Length)
return -1;
return _groups[index](span);
Inside a group, generated IL switches on exact length first. This
is the key to longest-match behaviour and to killing prefix ambiguity:
border and border-radius are different rules with
different lengths, so a six-character candidate can never fall through to a
shorter rule. The lexer already scanned the maximal identifier, so the
matcher receives the complete candidate rather than a prefix.
// Conceptually, per first-character group:
switch (span.Length - minLength)
{
case 0: return MatchLength(minLength);
case 6: return MatchLength(minLength + 6);
case 13: return MatchLength(minLength + 13);
default: return -1;
}
// Within one length bucket, longer prefixes are emitted first.
// A node with more than six candidates switches on the next character;
// a smaller node becomes a short SequenceEqual chain.
The trie only splits when a node still has more than six candidates. Below
that threshold, a linear chain of
MemoryExtensions.SequenceEqual<char> calls is smaller and
often faster than another switch. Single-rule groups skip the length switch
entirely, which keeps punctuation buckets tiny. The compiler also emits
longer buckets before shorter ones so the generated method layout favours
the more specific rules.
What the hot path actually does
For a property-heavy stylesheet, most name classification follows this sequence:
ScanNamewalks the maximal identifier, handling escapes and non-ASCII characters without allocating.ClassifyNamechecks whether the next character is(. If so, the token is a function and the exact table is skipped entirely.MinLengthByFirstChar[first]rejects candidates shorter than any rule with that first character. Single-letter identifiers never enter the generated matcher.ExactMatchindexes the first-character group, switches on length and walks a small character trie or comparison chain.- The result is written into
Tokenas a packed(start, length)location plus an interned text id.
The submitted character stream is never sliced into temporary strings. A
token stores only the interned id and a 32-bit location: 24 bits of start
offset and 8 bits of length. A Token is 12 bytes, so a
megabyte-scale stylesheet with roughly 380,000 tokens uses about 4.6 MB for
the token buffer before list overhead.
data: URI or string, are split into consecutive chunks that
share a kind; downstream consumers concatenate chunk text to recover the
original value.
Allocation behaviour
The tokenizer consumes ReadOnlySpan<char> and advances by
index; it does not materialise an intermediate token-text array. The token
destination is a List<Token> written through
CollectionsMarshal.AsSpan, so the loop avoids
List<Token>.Add and its bounds/count bookkeeping.
| Mechanism | What it buys |
|---|---|
ReadOnlySpan<char> + indices | No substring allocation for scanning, matching or token boundaries. |
Packed Token.Location | 12-byte tokens instead of separate start/end integers. |
| Pre-sized token buffer | Initial capacity is max(16, sourceLength / 2 + 16); the benchmark corpus needs no growth. |
CollectionsMarshal.SetCount | Writes directly into the backing array and trims to the exact count once at the end. |
| Interned text | Repeated names and punctuation become stable integer ids used by later stages. |
The honest caveat is that “zero allocation” describes the steady state, not
the first time a document introduces a new token text. InternTable
allocates the string on first sighting, then stores it in a dense global
table. Subsequent lookups use a thread-static front cache and
AlternateLookup<ReadOnlySpan<char>>, so the common
case does not need a lock or a new string. Single ASCII punctuation
characters use a pre-computed 128-entry id table, which removes the global
lock from the most frequent tokens in CSS.
The table never evicts. Its size follows the number of distinct token texts, not the number of tokens, which is the right trade-off for finite stylesheets but something to keep in mind for long-running apps that generate unbounded CSS text. Reusing a cleared token list avoids the one large backing-array allocation per document.
Performance
The parser benchmark has two measurements: the exact matcher over a hot
token stream, and a full Tokenizer.Parse over a synthetic
stylesheet built from every registered rule. The quick runner uses warmup
plus repeated runs, and reports the best run.
dotnet run --project Donkeh/Parser/Benchmark/Benchmark.csproj \
-c Release -- --quick
| Measurement | Result |
|---|---|
| Corpus | 1,013,028 bytes, 381,640 tokens |
| Full parse time | 4.97 ms per document |
| Document throughput | 203.8 MB/s |
| Token throughput | 76.8 M tokens/s |
| Steady-state allocation | 0 B per parse |
| Exact matcher | 4.81 ns/token, 207.9 M tokens/s |
| Matcher text throughput | 1,926.4 MB/s |
| Matcher allocation | 0 B per pass |
| IL compilation | 0.98 ms, 484,952 B, 0 GC collections |
These numbers were measured on .NET 10.0.10, Windows 10, a 12-logical-core workstation with workstation GC in interactive latency mode. They are a shape, not a promise: token distribution, CPU, GC mode and whether the intern table is warm all move the result. The important part is the ratio between full-document throughput and raw matcher throughput: the structural scan and token writing dominate, while keyword classification is already in the low-single-digit-nanoseconds-per-token range.
Trade-offs and sharp edges
| Decision | Cost or limitation |
|---|---|
| Runtime IL instead of a build-time artifact | Adds a small first-touch cost and depends on DynamicMethod. A build-time generator would remove both and be friendlier to ahead-of-time publishing. |
| One generated method per first character | More methods and delegates to manage, but each method stays below the JIT large-method threshold. |
| Exact matching, not prefix matching | Requires the scanner to consume the maximal identifier first. That is exactly what makes longest-match correct and keeps the generated trie small. |
| Linear comparison below six candidates | Deliberately trades a few comparisons for smaller code and better instruction-cache behaviour. |
| Global interning | Ids are stable for the process but the table grows forever; it is not a cache with eviction. |
| Packed token location | Excellent density, but a 16.7M-character source limit and 255-character token chunks are visible constraints. |
| Hand-written structural lexer | Some CSS context is resolved with look-behind, for example distinguishing a declaration colon from a pseudo-class colon after a property name. |
One deliberately non-obvious case is background-position:right
with no whitespace after the colon. A context-free lexer would see
:right as a pseudo-class. IsDeclarationColon checks
whether the previous token is a property name preceded by { or
;, and emits a plain colon instead. Selectors keep their
pseudo-class token because their preceding tokens do not match that shape.
Code tour
| File | Start here for… |
|---|---|
Parser/Source/Tokenizer.cs | The outer scan, token emission, capacity growth, strings, numbers and escapes. |
Parser/Source/Rules/Rules.cs | The static constructor, registration order and MinLengthByFirstChar setup. |
Parser/Source/Rules/*.cs | Per-letter property registrations. Diff-friendly, no runtime logic. |
Parser/Source/Tokenizer/PseudoClasses.cs | The 78 pseudo-class names, registered as complete :name rules. |
Parser/Source/Tokenizer/PseudoElements.cs | The 30 pseudo-element names, registered as complete ::name rules. |
Parser/Source/Functions/*.cs | The function-name table, including property/function overlaps. |
Parser/Source/Builder/RuleBuilder.cs | The declarative authoring API and duplicate-kind policy. |
Parser/Source/Compilation/Compiler.cs | Grouping, length switches, trie emission and the first-character jump table. |
Parser/Source/Compilation/ILGeneratorWrapper.cs | Readable IL emission and the optional IGNITE_SHOW_IL=1 dump. |
Parser/Source/Objects/Token.cs | The 12-byte token and its packed 24/8-bit location layout. |
Parser/Source/InternTable.cs | Dense ids, thread-local caching, ASCII fast path and span lookups. |
Parser/Benchmark/Program.cs | The quick benchmark, corpus builder and allocation columns. |
The useful mental model is: the scanner decides where a token ends, and the generated matcher decides what a maximal name means. Keeping those responsibilities separate is what allows the declarative rule table to be large without putting an interpreter in the per-character path.