This exploration delves deeper into LALRPOP's internal mechanisms. Specifically, it examines the significance of string literals and regular expressions used in earlier tutorials, and how they facilitate input processing—a process that can be customized. The initial stage of decomposing input through regular expressions is commonly referred to as lexical analysis or tokenization.

MST - This exploration goes deeper into LALRPOP's inner workings. Particularly, it investigates the importance of string literals and regular expressions used in previous tutorials, and how they are employed to process input—a process you can customize. The initial phase of breaking down input using regular expressions is typically known as lexical analysis or tokenization.

GPT - This section explores LALRPOP's underlying mechanics in greater detail. More specifically, it examines the meaning behind the string literals and regular expressions we used in prior tutorials, and how these elements are utilized to process input (a process that can be tailored). The first step of segmenting input via regular expressions is generally termed lexical analysis or tokenization.

If you're already familiar with the concept of a tokenizer, you might want to proceed directly to the calculator3 example, which handles more complex expressions, returning here only when additional control is needed. You may also be interested in the guide covering custom lexer implementation.

MST - If you're already comfortable with the concept of a tokenizer, you might want to jump straight to the calculator3 example, which deals with more complex expressions, coming back here only when you need more control. You might also be interested in the guide about creating a custom lexer.

GPT - If you're already comfortable with the idea of a tokenizer, you can skip ahead to the calculator3 example, which covers larger expressions, and return here only when you require more control. You might also find the tutorial on implementing a custom lexer interesting.

Terminals versus Nonterminals

You may have observed that our grammar defines two distinct categories of symbols. The nonterminals, such as Term and Num, are defined by outlining the sequence of symbols they must match, accompanied by associated action code that executes upon successful matching:

MST - You might have noticed that our grammar defines two distinct types of symbols. The nonterminals, like Term and Num, are defined by describing the sequence of symbols they must match, together with corresponding action code that runs once they are matched:

GPT - You may have noticed that our grammar contains two distinct types of symbols. Nonterminals like Term and Num are defined by specifying the sequence of symbols they must match, along with action code that executes when they are successfully matched:

   Num: i32 = r"[0-9]+" => i32::from_str(<>).unwrap();
// ~~~  ~~~   ~~~~~~~~~    ~~~~~~~~~~~~~~~~~~~~~~~~~~
// |    |     |                Action code
// |    |     Symbols to match
// |    Return type
// Nonterminal name

Additionally, there are terminals, composed of string literals and regular expressions scattered throughout the grammar. (Terminals are sometimes referred to as tokens, and these terms are used interchangeably.)

MST - Additionally, there are terminals, made up of string literals and regular expressions found throughout the grammar. (Terminals are sometimes called tokens, and these terms are used interchangeably.)

GPT - Furthermore, there are terminals, consisting of string literals and regular expressions distributed across the grammar. (Terminals are sometimes referred to as tokens, and we'll use these terms interchangeably.)

The distinction between terminals and nonterminals is crucial for LALRPOP's operation. In fact, when LALRPOP constructs a parser, it operates in a two-stage procedure. The initial stage is termed the lexer or tokenizer. Its role is to identify the sequence of terminals: essentially analyzing the raw characters of your text and splitting them into a series of terminals. It does this without considering your grammar or your position within it. Subsequently, the actual parser is code that reads this token stream and determines which nonterminals apply:

MST - The difference between terminals and nonterminals is fundamental to how LALRPOP functions. Indeed, when LALRPOP creates a parser, it follows a two-step approach. The first step is the lexer or tokenizer. Its purpose is to determine the order of terminals: basically analyzing the raw characters of your text and dividing them into a sequence of terminals. It performs this task without knowledge of your grammar or your place in it. Following that, the real parser is code that reads this token stream and identifies which nonterminals are applicable:

GPT - The distinction between terminals and nonterminals is vital to how LALRPOP works. When LALRPOP builds a parser, it operates in a two-stage process. The first stage is the lexer or tokenizer. It's responsible for identifying the sequence of terminals: essentially examining the raw characters of your text and segmenting them into a series of terminals. It does this without any awareness of your grammar or your location within it. Then, the core parser is code that consumes this token stream and figures out which nonterminals apply:

        +-------------------+    +---------------------+
Text -> | Lexer             | -> | Parser              |
        |                   |    |                     |
        | Applies regex to  |    | Consumes terminals, |
        | produce terminals |    | executes your code  |
        +-------------------+    | as it recognizes    |
                                 | nonterminals        |
                                 +---------------------+

LALRPOP's standard lexer uses regular expressions. By default, it collects all terminals (e.g., "(" or r"\d+") from your grammar and compiles them into a single comprehensive list. During execution, it scans the string, and at every position, identifies the longest match from the literals and regular expressions in your grammar and emits one of those. Let's revisit our sample grammar for illustration:

MST - LALRPOP's default lexer relies on regular expressions. By default, it gathers all terminals (e.g., "(" or r"\d+") from your grammar and compiles them into one large list. At runtime, it traverses the string and, at each position, finds the longest match among the literals and regular expressions in your grammar and outputs one of those. Let's review our example grammar for clarity:

GPT - LALRPOP's built-in lexer is based on regular expressions. By default, it extracts all terminals (e.g., "(" or r"\d+") from your grammar and compiles them into a single large list. At runtime, it walks through the string, and at each position, finds the longest match from the literals and regular expressions in your grammar and produces one of those. Let's take another look at our example grammar:

pub Term: i32 = {
    <n:Num> => n,
    "(" <t:Term> ")" => t,
};

Num: i32 = <s:r"[0-9]+"> => i32::from_str(s).unwrap();

This grammar effectively contains three terminals:

MST - This grammar actually includes three terminals:

GPT - This grammar contains three terminals:

• "(" -- a string literal, requiring exact matching // A string literal, requiring exact matching
• ")" -- a string literal, requiring exact matching // A string literal, requiring exact matching
• r"[0-9]+" -- a regular expression

When generating a lexer, it effectively checks each of these terminals in succession, similar to this pseudocode:

MST - When generating a lexer, it effectively evaluates each of these terminals in sequence, somewhat like this pseudocode:

GPT - When building a lexer, it effectively checks each of these terminals in order, much like this pseudocode:

let mut i = 0; // index into string
loop {
    skip whitespace; // done implicitly, at least by default
    if (data at index i is "(") { produce "("; }
    else if (data at index i is ")") { produce ")"; }
    else if (data at index i matches regex "[0-9]+") { produce r"[0-9]+"; }
}

Note that this is independent of your grammar. For instance, the tokenizer would happily process a string like this one, which doesn't conform to our grammar:

MST - Note that this is independent of you're grammar. For example, the tokenizer would readily process a string like this one, which doesn't comply with our grammar:

GPT - Keep in mind that this is independent of your grammar. For instance, the tokenizer would readily tokenize a string like this one, which doesn't align with our grammar:

  (  22   44  )     )
  ^  ^^   ^^  ^     ^
  |  |    |   |     ")" terminal
  |  |    |   |
  |  |    |   ")" terminal
  |  +----+
  |  |
  |  2 r"[0-9]+" terminals
  |
  "(" terminal

When these tokens are passed to the parser, it detects that we have one left parenthesis followed by two numbers (r"[0-9]+" terminals), thus reporting an error.

MST - When these tokens are sent to the parser, it detects that we have one left parenthesis followed by two numbers (r"[0-9]+" terminals), therefore reporting an error.

GPT - When these tokens are fed into the parser, it recognizes that we have one left parenthesis followed by two numbers (r"[0-9]+" terminals), and thus reports an error.

Precedence of Fixed Strings

Terminals in LALRPOP can be defined (by default) in two manners. Either as fixed strings (like "(") or as regular expressions (like r[0-9]+). There's a notable distinction: if at any point in the input both a fixed string and a regular expression could match, LALRPOP prioritizes the fixed string. To illustrate, let's adjust our parser. As recalled, the current parser interprets parenthesized numbers, yielding an i32. We'll modify it to yield a string, and introduce an "easter egg" where 22 (or (22), ((22)), etc.) produces the string "Twenty-two":

MST - Terminals in LALRPOP can be specified (by default) in two ways. Either as fixed strings (like "(") or as regular expressions (like r[0-9]+). There's a significant difference: if at any moment both a fixed string and a regular expression could match, LALRPOP favors the fixed string. To demonstrate, let's tweak our parser. As mentioned, the existing parser handles parenthesized numbers, returning an i32. We'll alter it to return a string and include an "easter egg" where 22 (or (22), ((22)), etc.) yields the string "Twenty-two":

GPT - Terminals in LALRPOP can be defined (by default) in two ways: as fixed strings (like "(") or as regular expressions (like r"[0-9]+"). There is a key difference: if at any point in the input both a fixed string and a regular expression could match, LALRPOP gives preference to the fixed string. To demonstrate, let's modify our parser. As noted, the current parser parses parenthesized numbers, returning an i32. We'll change it to return a string, and add an "easter egg" such that 22 (or (22), ((22)), etc.) returns the string "Twenty-two":

pub Term = {
    Num,
    "(" <Term> ")",
    "22" => "Twenty-two!".to_string(),
};

Num: String = r"[0-9]+" => <>.to_string();

If we perform basic unit tests, we observe that input 22 indeed matches the string literal. Notably, input 222 matches the regular expression instead; this happens because LALRPOP prefers the longest match. Afterward, if two matches are of equal length, it chooses the fixed string:

MST - If we conduct basic unit tests, we see that input 22 indeed matches the string literal. Notably, input 222 matches the regular expression instead; this is because LALRPOP prefers the longest match. Afterwards, if two matches are equally long, it selects the fixed string:

GPT - Running basic unit tests reveals that input 22 indeed matches the string literal. Interestingly, input 222 matches the regular expression instead; this occurs because LALRPOP prefers the longest match. If two matches are of equal length, it defaults to the fixed string:

#[test]
fn calculator2b() {
    let result = calculator2b::TermParser::new().parse("33").unwrap();
    assert_eq!(result, "33");

    let result = calculator2b::TermParser::new().parse("(22)").unwrap();
    assert_eq!(result, "Twenty-two!");

    let result = calculator2b::TermParser::new().parse("(222)").unwrap();
    assert_eq!(result, "222");
}

Ambiguities Between Regular Expressions

In the prior section, we saw fixed strings take precedence over regular expressions. What happens if two regular expressions can match the same input? Which one prevails? For instance, consider this variation of the grammar above, where we also aim to support parenthesized identifiers like ((foo22)):

MST - In the previous section, we saw that fixed strings have precedence over regular expressions. What if two regular expressions can match the same input? Which one takes precedence? For example, consider this variant of the grammar above, where we also want to accommodate parenthesized identifiers like ((foo22)):

GPT - In the preceding section, we observed that fixed strings take precedence over regular expressions. What happens when two regular expressions could match the same input? Which one wins? For instance, consider this variation of the previous grammar, where we also aim to support parenthesized identifiers such as ((foo22)):

pub Term = {
    Num,
    "(" <Term> ")",
    "22" => format!("Twenty-two!"),
    r"\w+" => format!("Id({})", <>), // <-- we added this
};

Num: String = r"[0-9]+" => <>.to_string();

I've defined the regular expression r\w+. However, according to regex documentation, \w matches both alphabetic characters and digits. Hence, there's an ambiguity: for instance, 123 could match either r"[0-9]+" or r"\w+". Attempting this grammar leads LALRPOP to report an error:

MST - I've defined the regular expression r\w+. However, according to regex documentation, \w matches both alphabetic characters and digits. Therefore, there's an ambiguity: for instance, 123 could match either r"[0-9]+" or r"\w+". Trying this grammar causes LALRPOP to report an error:

GPT - I've written the regular expression r\w+. However, according to regex documentation, \w matches both alphabetic characters and digits. Thus, there's an ambiguity: for example, 123 could match either r"[0-9]+" or r"\w+". If you attempt this grammar, LALRPOP will report an error:

error: ambiguity detected between the terminal `r#"\w+"#` and the terminal `r#"[0-9]+"#`

      r"\w+" => <>.to_string(),
      ~~~~~~

Several approaches exist to resolve this. One might adjust the regular expression so that the first character isn't a digit, perhaps using r"[[:alpha:]]\w*". While this works, it effectively changes what was previously matched (e.g., 123foo won't match anymore, for better or worse). Moreover, making regular expressions mutually exclusive isn't always practical. An alternative is to use a match declaration, allowing control over precedence among regular expressions.

MST - Several methods exist to address this. One could modify the regular expression so the first character isn't a digit, perhaps using r"[[:alpha:]]\w*". Although this works, it effectively alters what was matched before (e.g., 123foo won't match anymore, either good or bad). Furthermore, making regular expressions completely non-overlapping isn't always feasible. An alternative is to use a match declaration, enabling control over precedence between regular expressions.

GPT - There are several ways to fix this. One might adjust the regular expression so that the first character is not a digit, such as r"[[:alpha:]]\w*". While this works, it effectively changes the previous matching behavior (for instance, 123foo won't match anymore, whether that's good or bad). Also, ensuring regular expressions are completely disjoint isn't always convenient. Another option is to use a match declaration, which allows you to control precedence among regular expressions.

Simple Match Declarations

A match declaration enables explicit precedence assignment among terminals. In its simplest form, it groups regular expressions and string literals, placing higher precedence items first. For instance, we can resolve the conflict above by prioritizing r"[0-9]+" over r"\w+", implying that if something can be parsed as a number, it's treated as such, otherwise as an identifier.

MST - A match declaration allows explicit precedence assignment among terminals. In its simplest form, it groups regular expressions and string literals, with higher precedence items placed first. For instance, we can resolve the conflict above by giving r"[0-9]+" precedence over r"\w+", indicating that if something can be parsed as a number, it's treated as such, otherwise as an identifier.

GPT - A match declaration lets you explicitly define precedence between terminals. In its simplest form, it organizes regular expressions and string literals into groups, placing items with higher precedence first. For example, we can resolve the above conflict by assigning priority to r"[0-9]+" over r"\w+", meaning that if something can be lexed as a number, we treat it as such, otherwise as an identifier.

match {
    r"[0-9]+"
} else {
    r"\w+",
    _
}

This match declaration has two tiers; each tier can contain multiple items. The top tier holds only r"[0-9]+", signifying this regex has the highest priority. The second tier includes r\w+, which is evaluated afterward.

MST - This match declaration has two levels; each level can hold multiple items. The top level contains only r"[0-9]+", indicating this regex has the highest priority. The second level includes r\w+, which is processed subsequently.

GPT - This match declaration contains two levels; each level can include multiple items. The top level has only r"[0-9]+", indicating this regex has the highest priority. The second level includes r"\w+", which is evaluated afterwards.

The final _ signifies that other string literals and regular expressions appearing elsewhere in the grammar (e.g., "(" or "22") should be appended to the final precedence level (without _, it's invalid to use a terminal not in the match declaration).

MST - The final _ indicates that other string literals and regular expressions appearing elsewhere in the grammar (e.g., "(" or "22") should be appended to the final precedence level (without _, it's invalid to use a terminal not present in the match declaration).

GPT - The final _ means that other string literals and regular expressions in the grammar (e.g., ( or 22) should be added to the lowest precedence level (without _, using a terminal not in the match declaration is invalid).

Adding this match block to our example compiles successfully, but doesn't behave exactly as desired. Let's slightly modify our unit tests to include identifier cases:

MST - Adding this match block to our example compiles correctly, but doesn't behave exactly as intended. Let's slightly adjust our unit tests to include identifier cases:

GPT - Adding this match block to our example compiles without errors, but doesn't function precisely as intended. Let's slightly update our unit tests to include identifier cases:

#[test]
fn calculator2b() {
    // These will all work:

    let result = calculator2b::TermParser::new().parse("33").unwrap();
    assert_eq!(result, "33");

    let result = calculator2b::TermParser::new().parse("foo33").unwrap();
    assert_eq!(result, "Id(foo33)");

    let result = calculator2b::TermParser::new().parse("(foo33)").unwrap();
    assert_eq!(result, "Id(foo33)");

    // This one will fail:

    let result = calculator2b::TermParser::new().parse("(22)").unwrap();
    assert_eq!(result, "Twenty-two!");
}

The issue arises when parsing 22. Previously, the fixed string 22 had precedence, but with the new match declaration, we've explicitly stated that the regular expression r"[0-9]+" has full precedence. Since 22 isn't explicitly listed, it's added to the last level where _ resides. We can resolve this by explicitly including 22 in our match:

MST - The issue occurs when parsing 22. Earlier, the fixed string 22 had precedence, but with the new match declaration, we've explicitly stated that the regular expression r"[0-9]+" has full precedence. Since 22 isn't explicitly listed, it's added to the last level where _ resides. We can fix this by explicitly mentioning 22 in our match:

GPT - The problem emerges when parsing 22. Previously, the fixed string 22 had precedence, but with the new match declaration, we've explicitly stated that the regular expression r"[0-9]+" has full precedence. Since 22 isn't explicitly listed, it's added to the last level where _ appears. We can correct this by explicitly listing 22 in our match:

match {
    r"[0-9]+",
    "22"
} else {
    r"\w+",
    _
}

This raises the question of precedence within a match rung—both the regex and "22" can match the same string. The answer is that within a match rung, fixed literals take precedence over regular expressions, just as before, and all regular expressions must not overlap.

MST - This raises the question of precedence within a match rung—both the regex and "22" can match the same string. The answer is that within a match rung, fixed literals take precedence over regular expressions, as before, and all regular expressions must not overlap.

GPT - This brings up the question of precedence within a match tier—both the regex and "22" can match the same string. The answer is that within a match tier, fixed literals have precedence over regular expressions, just as before, and all regular expressions must not overlap.

With this updated match declaration, all tests pass.

MST - With this updated match declaration, all tests pass.

GPT - With this revised match declaration, all tests pass.

Renaming Match Declarations

Before reaching the final version of our example in the repository, there's one more aspect to consider. We can also use match declarations to assign names to regular expressions, avoiding direct typing in the grammar. For example, instead of writing r"\w+", we might prefer to write ID. This can be achieved by altering the match declaration as follows:

MST - Before arriving at the final version of our example in the repository, there's one more consideration. We can also use match declarations to assign names to regular expressions, avoiding direct typing in the grammar. For instance, instead of writing r"\w+", we might prefer to write ID. This can be accomplished by modifying the match declaration like so:

GPT - Before reaching the final version of our example in the repository, there's one more detail. We can also use match declarations to name regular expressions, so we don't have to type them directly in the grammar. For instance, rather than writing r"\w+", we might prefer to write ID. This can be done by modifying the match declaration as follows:

match {
    r"[0-9]+",
    "22"
} else {
    r"\w+" => ID, // <-- name it here
    _
}

Then, adjust the Term definition to reference ID instead:

MST - Then, adjust the Term definition to reference ID instead:

GPT - Then, modify the Term definition to reference ID instead:

pub Term = {
    Num,
    "(" <Term> ")",
    "22" => "Twenty-two!".to_string(),
    ID => format!("Id({})", <>), // <-- changed this
};

In fact, the match declaration can map a regular expression to any symbol you choose (i.e., you can also map to a string literal or another regular expression). Whatever symbol follows the => should be used in your grammer. For instance, some languages feature case-insensitive keywords; if you want to write "BEGIN" in the grammar but map it to a lexer regex, you might write:

MST - In fact, the match declaration can map a regular expression to any symbol you desire (i.e., you can also map to a string literal or another regular expression). Whatever symbol follows the => should be used in your grammar. For example, some languages have case-insensitive keywords; if you want to write "BEGIN" in the grammar but map it to a lexer regex, you could write:

GPT - Actually, the match declaration can map a regular expression to any symbol you like (i.e., you can also map to a string literal or another regular expression). Whatever symbol appears after the => should be used in your grammar. As an example, some languages have case-insensitive keywords; if you want to write "BEGIN" in the grammar but map it to a lexer regex, you could write:

match {
    r"(?i)begin" => "BEGIN",
    ...
}

Now, any reference to "BEGIN" in your grammar will match any capitalization.

MST - Now, any reference to "BEGIN" in your grammar will match any capitalization.

GPT - Now, any reference to "BEGIN" in your grammar will match any capitalization.

Customizing Token Skipping

To support comments, we need to skip more than just whitespace in our lexer. For this purpose, ignore patterns can be defined.

MST - To support comments, we need to skip more than just whitespace in our lexer. For this reason, ignore patterns can be specified.

GPT - To support comments, we need to skip more than just whitespace in our lexer. To achieve this, ignore patterns can be defined.

match {
    r"\s*" => { }, // Disabling default whitespace skipping if an `ignore pattern` is specified
    r"//[^\n\r]*[\n\r]*" => { }, // Skip `// comments`
    r"/\*[^*]*\*+(?:[^/*][^*]*\*+)*/" => { },  // Skip `/* comments */`
}

Unicode Compatibility

LALRPOP supports tokenization of inputs matching the entire Unicode character set or just ASCII. Enable features = [ "unicode" ] in your Cargo.toml if Unicode is required. Since Unicode tokenization requires loading the complete Unicode character set, enabling this feature increases binary size, so consider disabling it if Unicode support isn't needed.

MST - LALRPOP supports tokenization of inputs matching the entire Unicode character set or just ASCII. Enable features = [ "unicode" ] in your Cargo.toml if Unicode is required. Because Unicode tokenization requires loading the full Unicode character set, enabling this feature increases binary size, so consider disabling it if Unicode support isn't needed.

GPT - LALRPOP can tokenize inputs that match the full Unicode character set or just ASCII. If Unicode is needed, enable features = [ "unicode" ] in your Cargo.toml. Since Unicode tokenization requires loading the entire Unicode character set, enabling this feature increases binary size, so you might want to avoid it if Unicode support isn't required.

It's important to note that certain Perl regex character classes are "Unicode-friendly" and require Unicode support. For example, "\s" matches Unicode whitespace characters, not just ASCII ones, and "\d" matches Unicode digits (such as numerals in non-Latin scripts). If these patterns are used in the lexer, Unicode support becomes necessary.

MST - It's important to note that certain Perl regex character classes are "Unicode-friendly" and require Unicode support. For instance, "\s" matches Unicode whitespace characters, not just ASCII ones, and "\d" matches Unicode digits (such as numerals in non-Latin scripts). If these patterns are used in the lexer, Unicode support is required.

GPT - Important to note that certain character classes from Perl regex extensions are "Unicode-friendly" and require Unicode support. For example, \s matches Unicode whitespace characters, not just ASCII ones, and \d matches Unicode digits (such as numerals in non-Latin character sets). If these patterns are used in the lexer, Unicode support is necessary.

You might want to match only the ASCII subset of these characters. In such cases, you can use the ASCII-only character classes described here as alternatives, avoiding Unicode dependencies.

MST - You might want to match only the ASCII subset of these characters. In such cases, you can use the ASCII-only character classes described here as alternatives, avoiding Unicode dependencies.

GPT - You might prefer to match only the ASCII subset of these characters. In such scenarios, you can use the ASCII-only character classes described here as alternatives, thereby avoiding Unicode dependencies.