Expand description
CSS Tokenizer
This module implements a CSS tokenizer, responsible for converting a raw CSS source string into a flat stream of tokens.
§Responsibilities
- Consume raw characters
- Produce syntactic tokens defined by the CSS specification
- Preserve the original structure of the input as much as possible
§Non-responsibilities
- Parsing selectors or declarations
- Interpreting values (lengths, colors, percentages, etc.)
- Building trees or nested structures
§Design notes
- Tokens are produced in a linear stream
- Function tokens only represent the function name
- Matching of parentheses and function arguments is handled by the parser
§Allocation strategy
Cascade stylesheets are populated with many short identifiers, and the
majority of them contain no escape sequences. Because the tokenizer
borrows from the input string, any such lexeme is returned as a
Cow::Borrowed slice of the input and only escape-containing (or
otherwise transformed) lexemes are materialized onto the heap. Numbers are
parsed directly from the input slice instead of being assembled in a
scratch buffer.
Structs§
- Tokenizer
- CSS tokenizer.
Enums§
- Token
- CSS token produced by the tokenizer.