package lexer type TokenType byte const ( TokenEOF TokenType = iota TokenIdentifier TokenString TokenNumber TokenEqual TokenEcho TokenSemicolon TokenLeftBrace TokenRightBrace TokenLeftBracket TokenRightBracket TokenComma TokenPlus TokenMinus TokenStar TokenSlash TokenLeftParen TokenRightParen TokenIf TokenThen TokenElse TokenTrue TokenFalse TokenEqualEqual TokenNotEqual TokenLessThan TokenGreaterThan TokenLessEqual TokenGreaterEqual TokenEnd ) type Token struct { Type TokenType Value string Line int } type Lexer struct { input string pos int readPos int ch byte line int } func New(input string) *Lexer { l := &Lexer{input: input, line: 1} l.readChar() return l } func (l *Lexer) readChar() { if l.readPos >= len(l.input) { l.ch = 0 } else { l.ch = l.input[l.readPos] } l.pos = l.readPos l.readPos++ if l.ch == '\n' { l.line++ } } func (l *Lexer) NextToken() Token { var tok Token l.skipWhitespace() l.skipComment() switch l.ch { case '=': if l.peekChar() == '=' { l.readChar() // consume the current '=' tok = Token{Type: TokenEqualEqual, Value: "==", Line: l.line} } else { tok = Token{Type: TokenEqual, Value: "=", Line: l.line} } case '!': if l.peekChar() == '=' { l.readChar() // consume the current '!' tok = Token{Type: TokenNotEqual, Value: "!=", Line: l.line} } else { tok = Token{Type: TokenEOF, Value: "", Line: l.line} // Not supported yet } case '<': if l.peekChar() == '=' { l.readChar() // consume the current '<' tok = Token{Type: TokenLessEqual, Value: "<=", Line: l.line} } else { tok = Token{Type: TokenLessThan, Value: "<", Line: l.line} } case '>': if l.peekChar() == '=' { l.readChar() // consume the current '>' tok = Token{Type: TokenGreaterEqual, Value: ">=", Line: l.line} } else { tok = Token{Type: TokenGreaterThan, Value: ">", Line: l.line} } case ';': tok = Token{Type: TokenSemicolon, Value: ";", Line: l.line} case '"': tok = Token{Type: TokenString, Value: l.readString(), Line: l.line} return tok case '{': tok = Token{Type: TokenLeftBrace, Value: "{", Line: l.line} case '}': tok = Token{Type: TokenRightBrace, Value: "}", Line: l.line} case '[': tok = Token{Type: TokenLeftBracket, Value: "[", Line: l.line} case ']': tok = Token{Type: TokenRightBracket, Value: "]", Line: l.line} case ',': tok = Token{Type: TokenComma, Value: ",", Line: l.line} case '+': tok = Token{Type: TokenPlus, Value: "+", Line: l.line} case '-': tok = Token{Type: TokenMinus, Value: "-", Line: l.line} case '*': tok = Token{Type: TokenStar, Value: "*", Line: l.line} case '/': if l.peekChar() == '/' { // Comment line, skip it l.skipComment() return l.NextToken() } tok = Token{Type: TokenSlash, Value: "/", Line: l.line} case '(': tok = Token{Type: TokenLeftParen, Value: "(", Line: l.line} case ')': tok = Token{Type: TokenRightParen, Value: ")", Line: l.line} case 0: tok = Token{Type: TokenEOF, Value: "", Line: l.line} default: if isLetter(l.ch) { tok.Value = l.readIdentifier() switch tok.Value { case "echo": tok.Type = TokenEcho case "if": tok.Type = TokenIf case "then": tok.Type = TokenThen case "else": tok.Type = TokenElse case "true": tok.Type = TokenTrue case "false": tok.Type = TokenFalse case "end": tok.Type = TokenEnd default: tok.Type = TokenIdentifier } return tok } else if isDigit(l.ch) { tok.Type = TokenNumber tok.Value = l.readNumber() return tok } else { tok = Token{Type: TokenEOF, Value: "", Line: l.line} } } l.readChar() return tok } func (l *Lexer) skipWhitespace() { for l.ch == ' ' || l.ch == '\t' || l.ch == '\n' || l.ch == '\r' { l.readChar() } } func (l *Lexer) readIdentifier() string { pos := l.pos for isLetter(l.ch) || isDigit(l.ch) { l.readChar() } return l.input[pos:l.pos] } func (l *Lexer) readNumber() string { pos := l.pos for isDigit(l.ch) { l.readChar() } return l.input[pos:l.pos] } func (l *Lexer) readString() string { pos := l.pos + 1 for { l.readChar() if l.ch == '"' || l.ch == 0 { break } } str := l.input[pos:l.pos] l.readChar() // Skip closing quote return str } func isLetter(ch byte) bool { return 'a' <= ch && ch <= 'z' || 'A' <= ch && ch <= 'Z' || ch == '_' } func isDigit(ch byte) bool { return '0' <= ch && ch <= '9' } func (l *Lexer) peekChar() byte { if l.readPos >= len(l.input) { return 0 } return l.input[l.readPos] } func (l *Lexer) skipComment() { if l.ch == '/' && l.peekChar() == '/' { l.readChar() // Skip first '/' l.readChar() // Skip second '/' for l.ch != '\n' && l.ch != 0 { l.readChar() } l.skipWhitespace() } }