"std:saurus/token"Defines token kinds, source positions, and token structures used by the scanner and parser. Provides keyword lookup via binary search.
View source on Codeberg →type Pos struct {
file string
line int
column int
}Represents a source position with file, line, and column. Fields are unexported; use the accessor functions below.
type Kind intRepresents the type of a token. Defined as an integer enum using iota. See Constants for all values.
type Token struct {
kind Kind
pos Pos
lit string
floatVal float64
runeVal rune
}Represents a lexical token with its kind, position, and literal values. Fields are unexported; use the accessor functions below.
const (
// Special
EOF Kind = iota
ERROR
// Literals
IDENT; INT; FLOAT; STRING; RUNE
// Keywords (37)
APPEND; AS; ASSERT; ASSUME; BREAK; CAP; CASE
CLEAR; CONST; CONTINUE; DEFER; ELSE; FALLTHROUGH
FALSE; FOR; FREE; FUNC; IF; IMPORT; IN; IOTA_KW
LEN; LIST; MAKE; MAP; NEW; NIL; PACKAGE; PANIC
REMOVE; RETURN; SIZEOF; STRUCT; SWITCH; TRUE; TYPE; VAR
// Intrinsics
INTRINSIC; INTRINSIC_BUILD
// Operators & punctuation
PLUS; MINUS; STAR; SLASH; PERCENT; AMP; PIPE
CARET; TILDE; SHL; SHR; AND; OR; NOT
EQ; NE; LT; LE; GT; GE
ASSIGN; DEFINE; PLUS_ASSIGN; MINUS_ASSIGN
STAR_ASSIGN; SLASH_ASSIGN; PERCENT_ASSIGN
AMP_ASSIGN; PIPE_ASSIGN; CARET_ASSIGN
SHL_ASSIGN; SHR_ASSIGN
INC; DEC
DOT; COLON; COMMA; SEMI; ELLIPSIS
LPAREN; RPAREN; LBRACE; RBRACE; LBRACKET; RBRACKET
KIND_COUNT
)Token kind constants. Includes special tokens (EOF, ERROR), literal types, all 37 language keywords, compiler intrinsics, operators, comparison and assignment operators, increment/decrement, and punctuation.
func NewPos(file string, line, column int) PosCreates a Pos with the given file, line, and column.
func NoPos() PosReturns an invalid (zero-value) position.
func File(p Pos) stringReturns the filename of the position.
func Line(p Pos) intReturns the line number (1-based).
func Column(p Pos) intReturns the column number (1-based).
func IsValid(p Pos) boolReports whether the position has a positive line number.
func NewToken(kind Kind, pos Pos, lit string) TokenCreates a token with the given kind, position, and literal string.
func NewFloatToken(pos Pos, val float64, lit string) TokenCreates a FLOAT token with the given position, parsed value, and literal text.
func NewRuneToken(pos Pos, val rune) TokenCreates a RUNE token with the given position and rune value.
func TokenKind(t Token) KindReturns the kind of the token.
func TokenPos(t Token) PosReturns the position of the token.
func TokenLit(t Token) stringReturns the literal string of the token. For IDENT tokens this is the identifier name, for STRING tokens the string value, for INT tokens the literal text, and for ERROR tokens the error message.
func TokenFloat(t Token) float64Returns the float64 value for FLOAT tokens.
func TokenRune(t Token) runeReturns the rune value for RUNE tokens.
func LookupKeyword(name string) KindReturns the keyword Kind for the given name, or IDENT if the name is not a keyword. Uses binary search over a sorted keyword table.
func KindString(k Kind) stringReturns the string representation of a token kind. Keywords return their lowercase name, operators return their symbol, and special tokens return their uppercase name (e.g. "EOF").
func IsKeyword(k Kind) boolReports whether k is a keyword token (APPEND through VAR).
func IsLiteral(k Kind) boolReports whether k is a literal token (IDENT, INT, FLOAT, STRING, or RUNE).
func IsOperator(k Kind) boolReports whether k is an operator or punctuation token.
func IsAssignOp(k Kind) boolReports whether k is an assignment operator (=, +=, -=, etc.). DEFINE (:=) is a declaration, not an assignment, and returns false.
func TriggersSemi(k Kind) boolReports whether a token of this kind should trigger automatic semicolon insertion when followed by a newline. Returns true for identifiers, literals, certain keywords (break, continue, fallthrough, return, true, false, nil, iota), increment/decrement, and closing brackets.