package lexer type TokenType byte const ( TokenEOF TokenType = iota TokenIdentifier TokenString TokenNumber TokenEqual TokenEcho TokenSemicolon TokenLeftBrace TokenRightBrace TokenLeftBracket TokenRightBracket TokenComma TokenPlus TokenMinus TokenStar TokenSlash TokenLeftParen TokenRightParen TokenIf TokenThen TokenElse TokenTrue TokenFalse TokenEqualEqual TokenNotEqual TokenLessThan TokenGreaterThan TokenLessEqual TokenGreaterEqual TokenEnd TokenAnd TokenOr TokenNot TokenElseIf TokenNil TokenFunction TokenReturn ) type Token struct { Type TokenType Value string Line int } type Lexer struct { input string pos int readPos int ch byte line int } func New(input string) *Lexer { l := &Lexer{input: input, line: 1} l.readChar() return l } func (l *Lexer) readChar() { if l.readPos >= len(l.input) { l.ch = 0 } else { l.ch = l.input[l.readPos] } l.pos = l.readPos l.readPos++ if l.ch == '\n' { l.line++ } } func (l *Lexer) NextToken() Token { var tok Token l.skipWhitespace() l.skipComment() switch l.ch { case '=': if l.peekChar() == '=' { l.readChar() // consume the current '=' tok = Token{Type: TokenEqualEqual, Value: "==", Line: l.line} } else { tok = Token{Type: TokenEqual, Value: "=", Line: l.line} } case '!': if l.peekChar() == '=' { l.readChar() // consume the current '!' tok = Token{Type: TokenNotEqual, Value: "!=", Line: l.line} } else { tok = Token{Type: TokenEOF, Value: "", Line: l.line} // Not supported yet } case '<': if l.peekChar() == '=' { l.readChar() // consume the current '<' tok = Token{Type: TokenLessEqual, Value: "<=", Line: l.line} } else { tok = Token{Type: TokenLessThan, Value: "<", Line: l.line} } case '>': if l.peekChar() == '=' { l.readChar() // consume the current '>' tok = Token{Type: TokenGreaterEqual, Value: ">=", Line: l.line} } else { tok = Token{Type: TokenGreaterThan, Value: ">", Line: l.line} } case '"': tok = Token{Type: TokenString, Value: l.readString(), Line: l.line} return tok case '{': tok = Token{Type: TokenLeftBrace, Value: "{", Line: l.line} case '}': tok = Token{Type: TokenRightBrace, Value: "}", Line: l.line} case '[': tok = Token{Type: TokenLeftBracket, Value: "[", Line: l.line} case ']': tok = Token{Type: TokenRightBracket, Value: "]", Line: l.line} case ',': tok = Token{Type: TokenComma, Value: ",", Line: l.line} case '+': tok = Token{Type: TokenPlus, Value: "+", Line: l.line} case '-': tok = Token{Type: TokenMinus, Value: "-", Line: l.line} case '*': tok = Token{Type: TokenStar, Value: "*", Line: l.line} case '/': if l.peekChar() == '/' { // Comment line, skip it l.skipComment() return l.NextToken() } tok = Token{Type: TokenSlash, Value: "/", Line: l.line} case '(': tok = Token{Type: TokenLeftParen, Value: "(", Line: l.line} case ')': tok = Token{Type: TokenRightParen, Value: ")", Line: l.line} case 0: tok = Token{Type: TokenEOF, Value: "", Line: l.line} default: if isLetter(l.ch) { tok.Value = l.readIdentifier() switch tok.Value { case "echo": tok.Type = TokenEcho case "if": tok.Type = TokenIf case "then": tok.Type = TokenThen case "else": tok.Type = TokenElse case "elseif": tok.Type = TokenElseIf case "true": tok.Type = TokenTrue case "false": tok.Type = TokenFalse case "end": tok.Type = TokenEnd case "and": tok.Type = TokenAnd case "or": tok.Type = TokenOr case "not": tok.Type = TokenNot case "nil": tok.Type = TokenNil case "function": tok.Type = TokenFunction case "return": tok.Type = TokenReturn default: tok.Type = TokenIdentifier } return tok } else if isDigit(l.ch) { tok.Type = TokenNumber tok.Value = l.readNumber() return tok } else { tok = Token{Type: TokenEOF, Value: "", Line: l.line} } } l.readChar() return tok } func (l *Lexer) skipWhitespace() { for l.ch == ' ' || l.ch == '\t' || l.ch == '\n' || l.ch == '\r' { l.readChar() } } func (l *Lexer) readIdentifier() string { pos := l.pos for isLetter(l.ch) || isDigit(l.ch) { l.readChar() } return l.input[pos:l.pos] } func (l *Lexer) readNumber() string { pos := l.pos for isDigit(l.ch) { l.readChar() } return l.input[pos:l.pos] } func (l *Lexer) readString() string { pos := l.pos + 1 for { l.readChar() if l.ch == '"' || l.ch == 0 { break } } str := l.input[pos:l.pos] l.readChar() // Skip closing quote return str } func isLetter(ch byte) bool { return 'a' <= ch && ch <= 'z' || 'A' <= ch && ch <= 'Z' || ch == '_' } func isDigit(ch byte) bool { return '0' <= ch && ch <= '9' } func (l *Lexer) peekChar() byte { if l.readPos >= len(l.input) { return 0 } return l.input[l.readPos] } func (l *Lexer) skipComment() { if l.ch == '/' && l.peekChar() == '/' { l.readChar() // Skip first '/' l.readChar() // Skip second '/' for l.ch != '\n' && l.ch != 0 { l.readChar() } l.skipWhitespace() } }