lexextract: Tool for making derivation and lexicon

This tool is for making derivation trees and lexical entries from parse trees.

This tool converts a treebank made by "treetrans" into a derivbank (derivation trees of the target grammar theory). It also extracts lexical entries from the derivbank.

First, write inverse schemas with the following interfaces, in order to convert parse trees in an input treebank into derivation trees. The inverse schemas are applied in the following order.

1. Apply "root_constraints/1" to the value of "TREE_NODE\NODE_SIGN\" of the root node of a parse tree.

   root_constraints(-$Sign)
   $Sign	sign of the root node
   Unify $Sign with the sign of the root of the derivation tree.

2. To "TREE_NODE\NODE_SIGN\" of each node, apply "inverse_schema_binary/4" or "inverse_schema_unary/3" in a topdown way. The value of "TREE_NODE\SCHEMA_NAME\" is used as the name of a schema. Schemas are applied in a depth-first order.

   inverse_schema_binary(+$SchemaName, +$Mother, -$Left, -$Right)
   $SchemaName	schema name
   $Mother	sign of the mother
   $Left	sign of the left daughter
   $Right	sign of the right daughter
   Apply a binary schema to $Mother and obtain daughter signs.

   inverse_schema_unary(+$SchemaName, +$Mother, -$Dtr)
   $SchemaName	schema name
   $Mother	sign of the mother
   $Dtr	sign of the daughter
   Apply a unary schema to $Mother, and obtain a daughter sign.

3. After applying inverse schemas to all internal nodes, apply "lexical_constraints/2" to "TREE_NODE\NODE_SIGN\" of terminal nodes. Since this is done after all applications of inverse schemas, you can coerce default constraints using this interface.

   lexical_constraints(+$Word, -$Sign)
   $Word	feature structure representing a word (the value of "TREE_NODE\WORD\")
   $Sign	sign of a terminal node
   Unify $Sign with the sign of a terminal node.

Next, from terminal nodes of derivation trees, extract lexical entry templates and mappings from a word into lexical entry templates. Interfaces for lexicon extraction are defined in "lexextract.lil". The extraction algorithm is presented below. In each of the following steps, the target feature structures are copied. This means that even when you modify the target feature structures with new constraints or destructive operations the modifications will not affect derivation trees nor other lexical entries.

1. Apply "lexical_entry_template/3" to "DERIV_SIGN\" of each terminal node of a derivation tree. The result is stored in "LEXENTRY_SIGN\" of the derivation tree.

   lexical_entry_template(+$Word, +$Sign, -$Template)
   $Word	feature structure representing a word
   $Sign	lexical sign
   $Template	lexical entry template
   Make a lexical entry template $Template from lexical sign $Sign of the word $Word.

2. Apply "reduce_lexical_template/5" to the lexical entry template, and obtain a key to look up a lexicon, a sign of a lexeme, and a history of lexical rule applications. The obtained lexeme will be stored in a template database. Lexeme signs are also stored in "LEXEME_SIGN\" of derivation trees.

   reduce_lexical_template(+$Word, +$InTemplate, -$Key, -$OutTemplate, -$LexRules)
   $Word	feature structure representing a word
   $InTemplate	input lexical entry template (the output of "lexical_entry_template/3")
   $Key	key to look up a lexicon
   $OutTemplate	sign of a lexeme
   $LexRules	a list of applied lexical rules
   Obtain a sign of a lexeme by inversely applying lexical rules to a lexical entry template obtained by "lexical_entry_template/3"

3. If a lexeme sign is not stored in the database yet, i.e., it is first to see, apply "lexeme_name/4" to the lexeme sign to obtain the name of a lexeme. The pair of this name and the history of the application of lexical rules will be the name of a lexical entry template. A mapping from a key to a lexical entry template is stored in a lexicon database. Template names are stored in "TERM_TEMPLATE\" of derivation trees.

   lexeme_name(+$Word, +$Template, +$ID, -$Name)
   $Word	feature structure representing a word
   $Template	sign of a lexeme
   $ID	identification number (integer)
   $Name	name of a lexeme (string)
   Assign a unique name to a lexeme

4. Increment the occurrence count of a word. Occurrence counts will be used for cutting off infrequent words in "lexrefine".

   word_count_key(+$LexKey, -$CountKey)
   $LexKey	key to look up a lexicon
   $CountKey	key to be used for counting a word
   Obtain a key to count the occurrence of a word. If you want to count different keys as an identical word, implement this predicate to return the same $CountKey for the different keys.