"std:saurus/scanner"Lexical scanner for Saurus source code. Tokenizes input into a stream of tokens with support for automatic semicolon insertion, Unicode identifiers, and all number/string literal formats.
View source on Codeberg →type ErrorHandler func(any, token.Pos, string)Callback function invoked when a scanning error occurs. Receives a user-provided context value, the source position of the error, and an error message string.
type Scanner struct { /* unexported fields */ }Lexer/tokenizer for Saurus source code. Maintains source text, current position, automatic semicolon insertion state, a single-token peek buffer, and a string builder for constructing literal values.
func New(source, filename string, errHandler ErrorHandler, errCtx any) ScannerCreates a new Scanner for the given source text. The filename is used in position information. The error handler and context are called when scanning errors occur. The caller must call Free when done to release internal resources.
func Free(s *Scanner)Releases the internal string builder used by the scanner. Must be called when the scanner is no longer needed.
func Next(s *Scanner) token.TokenConsumes and returns the next token from the input. Handles whitespace skipping, comment removal, automatic semicolon insertion, and all literal scanning (identifiers, numbers, strings, runes). Returns an EOF token at end of input.
func Peek(s *Scanner) token.TokenReturns the next token without consuming it. The token is buffered internally so the next call to Next returns the same token.
func AtEnd(s *Scanner) boolReports whether the scanner has reached the end of the input.