Class Lexer

java.lang.Object

org.episteme.natural.computing.compiler.Lexer

public class Lexer
extends Object

A simple lexer (tokenizer) for DSL processing.

Converts input text into a stream of tokens based on configurable patterns.

Since:

1.0

Author:

Silvere Martin-Michiellot, Gemini AI (Google DeepMind)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Lexer.LexerException	Exception thrown when lexer encounters invalid input.
static class	Lexer.Token	Represents a single token.
static class	Lexer.TokenType	Represents a token type with its pattern.

Constructor Summary

Constructors

Constructor	Description
Lexer()	Creates a lexer with default token types.
Lexer(List<Lexer.TokenType> tokenTypes)	Creates a lexer with custom token types.

Method Summary

Modifier and Type	Method	Description
void	addTokenType(Lexer.TokenType type)	Adds a custom token type (higher priority than defaults).
List<Lexer.Token>	tokenize(String input)	Tokenizes the input string.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Lexer

public Lexer()

Creates a lexer with default token types.

Lexer

public Lexer(List<Lexer.TokenType> tokenTypes)

Creates a lexer with custom token types.

Method Details

addTokenType

public void addTokenType(Lexer.TokenType type)

Adds a custom token type (higher priority than defaults).

tokenize

public List<Lexer.Token> tokenize(String input)
                           throws Lexer.LexerException

Tokenizes the input string.

Parameters:

input - Source code to tokenize

Returns:

List of tokens

Throws:

Lexer.LexerException - if unrecognized characters are found