Lumen expression language

Simple Regular Expression (PCRE) Matching

The expression language supports simple matching on regular expressions using the ~= operator and its variations. With these operators, the right hand side is interpreted as a PCRE (Perl-Compatible Regular Expression).

Regular expressions offer far more powerful pattern matching capabilities than globs, but are generally less efficient (even in relatively simple forms) and can be harder to read. The use of globs is preferred in cases where the glob match functionality can meet the requirements.

This document does not describe the PCRE syntax in any detail. For more information, refer to the PCRE documentation.

Some common pitfalls of using regular expressions:

• Regular expressions are not anchored by default. That means that unless explicitly anchored, a regular expression can match anywhere in the subject string (the left hand side of the expression) where a match is possible. For example, the regular expression foo matches “foo”, “foofoo”, “some computer engineers use ‘foo’ a lot but no one knows what it means”, etc.

   

• There are many special characters in regular expressions, far more than there are in globs. This includes dot (“.”), which matches any character. Therefore, the regular expression foo.html matches “foo.html” and “fooFhtml”, etc.

   

• A star (“*”) in a regular expression does not mean the same as it does in a glob. Rather, it modifies the meaning of the prior match to mean “match any number of times, including 0”. So f* matches “f”, and also matches “fff” (anywhere in the subject string). It also matches zero occurrences of “f”. This means that even if the subject string has no “f”s at all, f* will match it.

   

A common case where a regular expression match is useful is in alternation, that is, matching two or more alternate sequences of characters. For example, the following is true if the path ends in “.jpeg”, “.jpg”, or “.gif”:

This example illustrates some other common pitfalls of regular expressions:

• The backslash before the initial dot (“.”) removes the special meaning of dot (which normally means to match any character) and requires matching a literal dot. Backslash escapes are common in regular expressions, but are also the escape character used in a regular string in the expression language, and also in JSON strings that generally carry expressions. This example uses a raw string constructor (the initial r) to avoid requiring a “double backslash”.

   

• Frequently users of regular expressions forget to use appropriate anchors. This regular expression is anchored at the end of the string thanks to the $ special character, and so therefore requires that the string end with “.jpeg”, “.jpg”, or “.gif”. Note that it is not anchored at the beginning of the string (which would be indicated by a ^ character as the first character of the regular expression), which is appropriate because we want to match any initial path components and filenames up to the final suffix.

   

When all that is wanted is a simple regex match/doesn’t match test, the ~= operator and its variations are the operators to use.

“Rex” Match/Replace

The /= operator and its variations support an extended form of regular expression that includes a replacement string and optional flags. The name for this construct is “rex”. This form requires the use of a delimiter to delimit the various portions of the expression. The canonical form is:

Here, the delimiter character is a slash (“/”). However, any delimiter character may be chosen. The requirement is that the delimiter cannot appear anywhere in the regular expression, the replacement string, or the flags (not even if escaped by a backslash).

The rex form is documented externally here. (This is the documentation for the library facility that implements support for rexes.)

If the given regular expression (a PCRE regular expression, same as is used with the ~= operator) matches, then the result is the value of the replacement string, or an empty string if no replacement string has been specified. If the regular expression does not match, then the result is nil. This means that the /= operator and its variations can be used in boolean context with the same meaning as the ~= operator (but still requiring a delimited rex rather than a simple regular expression).

Note that when negated or when used in lists, the /= form reverts to boolean-only semantics. Only the /= and %/= operators provide access to the replacement string.

By default, the replacement string only replaces the first portion of the subject string that is matched by the regular expression. Therefore, given /foo/bar as a rex, the following holds:

“This is foo” becomes “This is bar”.

“This is foo and here’s another foo” becomes “This is bar and here’s another foo”.

To replace ALL matches, use the g flag: /foo/bar/g:

“This is foo and here’s another foo” becomes “This is bar and here’s another bar”.

Finally, to get only the value of the replacement string, use the t flag: /foo/bar/t

“This is foo” becomes “bar”.

“This is foo and here’s another foo” also becomes “bar”.

The t flag is handy when using subgroups and wanting to evaluate only to the portion matched by a subgroup. For example, a handy rex to retrieve the last path component of a URI is #([^/]*)$#\1#t. Note that use of alternate delimiter (“#”) in this case since the regular expression contains a slash. Without the ‘t’ flag the equivalent expression is a bit less intuitive: #^.*?([^/]*)$#\1#

Finally, note that the b (basic regex) and e (extended regex) flags are suppressed by the expression language. The expression language only supports PCRE.

Generally speaking, comparison tests use the highest ranked type. For example, comparing a string to a boolean results in the string being promoted to boolean first.

There is no automatic conversion to numeric form for the numeric operators. The unary “+” operator may be used to force a string or boolean to a numeric value, however. Strings are converted to the highest required numeric type, and booleans are converted to unsigned (0 for false, 1 for true).

Notes:
 

• All decimal numeric types also allow the specification of an exponent. This is indicated by the letter “e” followed by the exponent (a positive or negative integer).
   

• All hexadecimal numbers support a power-of-two exponent. This is indicated by the letter “p” followed by the exponent (a positive or negative integer).

Three lua bindings have been created for evaluating expressions:
 

• leapfrog.expr_test(<expression>): Test whether an expression is valid. Returns a flag (true or false) and an error message (or nil if no error).
   

• leapfrog.expr_val(<txn>, <expression>): Evaluate the expression and return the result.
   

• leapfrog.expr_val_bool(<txn>, <expression>): Evaluate the expression, convert the result to a boolean value, and return the result.
   

Both of the bindings take a transaction object and an expression (as a string). If errors occur during evaluation, a lua error is thrown. the lua pcall() mechanism may be used to do a protected call of an expression evaluation so that any errors can be caught and reported.

“English” operators

The existence of “English” operators (e.g. "globmatch") leads to complications in parsing , since "globmatch" would also be recognized as an identifier. For the Enlish operators se to be correctly used as an operator, they must be followed by a non-identifier character. This allows for the “first-match-wins” semantics to be used.

Whitespace

It should be noted that whitespace is only necessary to resolve lexical ambiguity across different tokens or expressions.
 

Example:
 

• “123.45” is a numeric constant

• “123 . 45” is a concatenation expression
   

While that is the case, generally whitespace use is recommended for increased readability within expressions.