Due to its importance in security, syntax analysis has found usage in many high-level programming languages. The Lisp language has its share of operations for evaluating regular expressions, but native parsing of Lisp code in this way is unsupported. Matching on lists requires a significantly more complicated model, with a different programmatic approach than that of string matching. This work presents a new automata-based approach centered on a set of functions and macros for identifying sequences of Lisp S-expressions using finite tree automata. The objective is to test that a given list is an element of a given tree language. We use a macro that takes a grammar and generates a function that reads off the leaves of a tree and tries to parse them as a string in a context-free language. The experimental results indicate that this approach is a viable tool for parsing Lisp lists and expressions in the abstract interpretation framework
There have been many studies on how to generate a tree parser over Lisp lists, so that the existence of a list as an element of a regular language can be determined through equality checks [
Another requirement of this approach is to provide a certain malleability in the interface—the ability for a programmer to define his or her own transition rules (or rewrite rules) on-the-fly—in the make-tree-matcher
The inability to use regular expressions in a simple format was perhaps the biggest hurdle. We will use standard definitions, but we give some of these for convenience of the reader.
• Analytic grammar—A set of rules for parsing and returning of truth values confirming a string as either consistent or inconsistent with the rules of a formal language.
• Context-free grammar—A set of rules over an alphabet, defined by a quad-tuple G = (Vt, Vn, P, S), where P is a set of production rules;
Vn is a set of non-terminals;
Vt is a set of terminals; and S is a starting non-terminal and an element of Vn.
• Context-free language—All strings which can be generated by a context-free grammar.
• Finite automaton—A set of states and transitions, commonly expressed in a flow diagram. Also called a finite state machine.
• Finite state machine: a model of computation composed of states, a transition function, and an input alphabet.
• Transition function: describes a condition that would need to be fulfilled to enable the transition.
• input alphabet: input recognized by the finite state machine
• Formal grammar—A description of a set of rules for a given alphabet over which a set of finite strings can
be defined.
• Kleene closure/Kleene star—The set of all possible combinations of non-terminals of a regular language. Specifically, the superset of a set of strings containing the empty string ε and closed on the string concatenation function. Every string that is part of a regular language can be found in its Kleene expansion.
• Parse tree—A description of the syntax of a string within a formal grammar.
• Parser—A method or algorithm or its implementation which examines the application of a given string within an analytic grammar.
• Regular tree language—The set of trees accepted by a finite tree automaton.
• Terminal—A constant, indivisible value that cannot be further reduced to a more simplified form within its own grammar.
• Tree automaton—While finite automata typically act on strings, tree automata are used for tree expressions.
• Yield—The string pattern formed from a tree’s leaves as encountered in an ordered traversal.
Finite tree automata are much more difficult to implement than finite automata, and regular tree languages do not have a nice compact notation like regular (string) languages do. Instead of reading only one next symbol, finite state machines that are used to recognize regular expressions can read any finite number of next symbols. Each of these next symbols can have any finite number of next states. Since parse trees are not generally unique, we do not know anything about the structure of the tree when we decide it’s a member of the regular tree language by parsing its yield. We know it’s a parse tree for a string in the grammar, but there can be more than one, and we don’t know which it is.
Although our initial vision involved the usage of regular expressions, which would be appropriate for normal strings [5,6], the nested form of S-expressions required a more iterative and comprehensive method. Traversing a Lisp list involves the usage of a parse tree. Through this application, one can back trace the sequence of an expression through the parent nodes that generated each step [
Generalized nondeterministic finite automata can be defined as the 5-tuple (S, Σ, δ, s, a), where:
S = A complete set of states;
Σ = A finite alphabet;
δ = A transition function (δ: (S − {a}) × (S − {s}) → R);
s = A start state; and a = A accept state;
where R is the set of all regular expressions over Σ (“Automata”).
This can be similarly migrated to tree parsing. Recognizing trees as elements of a regular language only allows us to say that the input is part of the language, or set of all possible elements of the regular grammar. This is done through Boolean comparisons in a top-down tree automaton context. Such automata can be represented with the four-tuple (Q, F, Qf, Δ), where:
Q = A set of states;
F = A ranked alphabet;
Qf = A subset of terminal states in Q; and
Δ = A set of transition rules [8,9].
As our implementation uses a nondeterministic pushdown approach, we can only test for the presence of a list in a regular tree language. As the language can have any number of similarly organized tree structures, we cannot tell with our algorithm which one of them has been found—only that the pattern in question is present in the language.
Using a finite tree automaton to match a Lisp list would require every cons cell in the pattern to have its own production, all labeled “cons”.
Theorem 1. A regular tree language is the set of parse trees for a context-free grammar [
We parse the yield of the tree to show that it’s a member of the context-free language, and use the theorem 1 to conclude that it’s a member of the regular tree language. This does not provide us any information about the structure of the original tree, but we do know whether it is a member of the tree language we defined. If it is a member, then we know that it matches the pattern we are looking for.
Parsing the string allowed us to test whether it was a member of a given string language. A language is a set of sentences (strings and trees are “sentences” in the sense that we use them in formal language theory). A contextfree language is the set of all strings that can be generated by its grammar. We also know because of the theorem that there is a regular tree language composed of all its parse trees.
We test whether that tree is in the regular tree language by testing whether the string is a parse tree of the context-free language.
Our tree pattern matching implementation allows for parsing of a tree structure as a string for subsequent comparison. By seeing that a given list is a parse tree accepted by a finite tree automaton, we know, by the general theorem which states “A regular tree language is the set of parse trees for a context-free grammar”. That means the list in question is a member of the regular language tested by the automaton. The use of tree yield functions simplified this step. Conceptually, trees consist of parent nodes, where child nodes extend from the right leaf of each node, with the outermost levels on the left side of the graph. Our custom yield function generates usable strings from tree inputs. With the make-tree-matcher function, the yield of a given tree is produced, and recursive-descent parsing is performed on the value by the same function
We experimented and provided the results with the make-tree-matcher function in Figures 2 and 3. Notice that the BOOLEVALUATION function is the language of all true Boolean expressions without variables using efficient implementations of automata operations.
From the definition of a context-free language, we know that a regular tree language is the set of all parse trees of this language’s grammar. Therefore, a tree is in the regular tree language if its yield is in the context-free language.
We proposed a symbolic approach for pattern matching on LISP programs. We use a symbolic automata representation and implement set of functions and macros for identifying sequences of Lisp S-expressions using finite
tree automata. Our experiments demonstrate that the study has produced a viable tool for parsing Lisp lists and expressions, and because of the language used, has the added bonuses of portability, small size, customizability, and clarity. The study of formal grammar and regular expressions has shown us with those topics the utility, robustness, and sometimes elegance of regular languages and Lisp. The same approach can also be applied to variety of other functional programming languages. Finally, the use of automata as a symbolic representation for verification has been investigated in other contexts (e.g., [