Texinfo::Parser
***************

NAME
====

Texinfo::Parser - Parse Texinfo code into a Perl tree

SYNOPSIS
========

  use Texinfo::Parser;
  my $parser = Texinfo::Parser::parser();
  my $tree = $parser->parse_texi_file("somefile.texi");
  my ($errors, $errors_count) = $parser->errors();
  foreach my $error_message (@$errors) {
    warn $error_message->{'error_line'};
  }

  my $index_names = $parser->indices_information();
  my $float_types_arrays = $parser->floats_information();
  my $internal_references_array
    = $parser->internal_references_information();
  # An hash reference on normalized node/float/anchor names
  my $labels_information = $parser->labels_information();
  # A hash reference, keys are @-command names, value is an
  # array reference holding all the corresponding @-commands.
  my $global_commands_information = $parser->global_commands_information();
  # a hash reference on some document informations (encodings,
  # input file name, dircategory and direntry list, for exampel).
  my $global_informations = $parser->global_informations();

DESCRIPTION
===========

Texinfo::Parser will parse Texinfo text into a perl tree.  In one pass it expands user-defined @-commands, conditionals (@ifset, @ifinfo...) and @value and constructs the tree.  Some extra information is gathered while doing the tree: for example, the block command associated with @end, the number of rows in a multitable, or the node associated with a section.

METHODS
=======

No method is exported in the default case.  The module allows both an object-oriented syntax, or traditional function, with the parser as an opaque data structure given as an argument to every function.

Initialization
--------------

The following method is used to construct a new Texinfo::Parser object:

$parser = Texinfo::Parser::parser($options);

     This method creates a new parser.  The options may be provided as a hash reference.  There are two types of option.  The first type of option change the way the parser behaves; they are described right here.  The other type of option allows giving the parser some information as if it came from texinfo code; for example, allow setting aliases (as with @alias), values (as with @set), or merged indices (as with @synindex).  These options are described below in Texinfo Parser options.

     expanded_formats

          An array reference of the output formats for which @ifFORMAT conditional blocks should be expanded.  Default is empty.

     include_directories

          An array reference of directories in which @include files should be searched for.  Default contains the working directory, ..

     IGNORE_BEFORE_SETFILENAME

          If set, and @setfilename exists, everything before @setfilename is put in a special container type, @preamble_before_setfilename. This option is set in the default case.

     IGNORE_SPACE_AFTER_BRACED_COMMAND_NAME

          If set, spaces after an @-command name that take braces are ignored. Default on.

     MAX_MACRO_CALL_NESTING

          Maximal number of nested user-defined macro calls.  Default is 100000.

     SHOW_MENU

          If false, no menu-related errors are reported.  Default is true.

Parsing Texinfo text
--------------------

There are three methods that may be called to parse some Texinfo code: parse_texi_line for a line, parse_texi_text for a text fragment, and parse_texi_file for a file.

For all those functions, if the $parser argument is undef, a new parser object is generated to parse the line.  Otherwise the parser given as an argument is used to parse into a tree.

When parse_texi_text is used, the resulting tree is rooted at a root_line type container.  Otherwise, the resulting tree should be rooted at a text_root type container if it does not contain nodes or sections, at a document_root type container otherwise.

$tree = parse_texi_line($parser, $text, $first_line_number, $file_name, $macro_name, $fixed_line_number)

     This function is used to parse a short fragment of Texinfo code.

     $text may be either an array reference of lines, or a text.

     The other arguments are optional and allow specifying the position information of the Texinfo code.  $first_line_number is the line number of the first text line.  $file_name is the name of the file the text comes from.  $macro is for the user-defined macro name the text is expanded from.  If $fixed_line_number is set, the line number is not increased for the different lines, as if the text was the expansion of a macro.

$tree = parse_texi_text ($parser, $text, $line_numbers_specification, $file_name, $macro_name, $fixed_line_number)

     This function is used to parse some Texinfo text.

     $text may be either an array reference of lines, or a text.

     The other arguments are optional and allow specifying the position information of the Texinfo code.  There are two distinct cases for $line_numbers_specification.

      1. If it is an array reference, it is considered to hold objects describing the position information of each text line.

      2. If $line_numbers_specification is a scalar, it is the line number of the first text line.  In that case (like for parse_texi_text), $file_name is the name of the file the text comes from. and $macro is for the user-defined macro name the text is expanded from.  If $fixed_line_number is set, the line number is not increased for the different lines, as if the text was the expansion of a macro.

$tree = parse_texi_file($parser, $file_name)

     The file with name $file_name is considered to be a Texinfo file and is parsed into a tree.

     undef is returned if the file couldn't be read.

The errors collected during the tree parsing are available through the errors method.  This method comes from Texinfo::Report, and is described in errors.

Getting information on the document
-----------------------------------

After parsing some information about the Texinfo code that was processed is available from the parser.

Some global information is available through global_informations

$info = global_informations($parser)

     The $info returned is a hash reference.  The possible keys are

     input_file_name

          The name of the main Texinfo input file.

     input_encoding_name

     input_perl_encoding

          input_encoding_name string is the encoding name used for the Texinfo code. input_perl_encoding string is a corresponding perl encoding name.

     dircategory_direntry

          An array of successive @dircategory and @direntry as they appear in the document.

     novalidate

          If set, it is as if @novalidate was set in the document.

Some command lists are available, such that it is possible to go through the corresponding tree elements without walking the tree.  They are available through global_commands_information

$commands = global_commands_information($parser)

     $commands is an hash reference.  The keys are @-command names.  The associated values are array references containing all the corresponding tree elements.

All the @-commands that have an associated label (so can be the target of cross references) - @node, @anchor and @float with label - have a normalized name associated, constructed as described in the HTML Xref node in the Texinfo manual.  Those normalized labels and the association with @-commands is available through labels_information:

$labels_information = labels_information($parser)

     $labels_information is a hash reference whose keys are normalized labels, and the associated value is the corresponding @-command.

Information on @float is also available, grouped by type of floats, each type correponding to potential @listoffloats. This information is available through the method floats_information.

$float_types = floats_information($parser)

     $float_types is a hash reference whose keys are normalized float types (the first float argument, or the @listoffloats argument). The normalization is the same as for node names. The value is the list of float tree elements appearing in the texinfo document.

Internal references, that is, @-commands that refer to node, anchors or floats within the document are also available:

$internal_references_array = internal_references_information($parser);

     The function returns a list of cross-reference commands referring to the same document.

Information about defined indices, merged indices and index entries is also available through the indices_information method.

indices_information

       $index_names = indices_information($parser);

     The index names is a hash reference.  The keys are

     in_code

          1 if the index entries should be formatted as code, 0 in the opposite case.

     name

          The index name.

     prefix

          An array reference of prefix associated to the index.

     merged_in

          In case the index is merged to another index, this key holds the name of the index the index is merged into.  It takes into account indirectly merged indices.

     contained_indices

          An hash reference holding names of indices that are merged into the index, including itself.  It also contains indirectly merged indices.  This key is removed if the index is itself later merged to another index.

     index_entries

          An array reference containing index entry structures for index entries associated with the index.  The index entry could be associated to @-commands like @cindex, or @item in @vtable, or definition commands entries like @deffn.

          The keys of the index entry structures are

          index_name

               The index name.

          index_at_command

               The name of the @-command associated with the index entry.

          index_type_command

               The @-command associated with the index entry allowing to find the index type.

          content

               An array reference corresponding to the index entry content.

          content_normalized

               An array reference corresponding to the index entry content, independent of the current language.

          command

               The element in the parsed tree associated with the @-command holding the index entry.

          node

               The node in the parsed tree containing the index entry.

          number

               The number of the index entry.

          region

               The region command (@copying, @titlepage) containing the index entry, if it is in such an environement.

     The following shows the references corresponding to the default indexes cp and fn, the fn index having its entries formatted as code and the indices corresponding to the following texinfo

       @defindex some
       @defcodeindex code

       $index_names = {'cp' => {'name' => 'cp', 'in_code' => 0, },
                       'fn' => {'name' => 'fn', 'in_code' => 1, },
                       'some' => {'in_code' => 0},
                       'code' => {'in_code' => 1}};

     If name is not set, it is set to the index name.

Texinfo Parser options
----------------------

Setting these options is the same as seeing some Texinfo constructs in the document.

aliases

     A hash reference.  The key is a command name, the value is the alias, as could be set by @alias.

clickstyle

     A string, the command name associated with @clickstyle.

documentlanguage

     A string corresponding to a document language set by @documentlanguage.

indices

     If it is a hash reference, the keys are index names, the values are index prefix hash references.  The index prefix hash reference values are prefix, the value is set if the corresponding index entries should be formatted as if in @code.  An example is as indices_information.

     If it is an array reference, it is a list of index names, as if they were entered as

       @defindex name

kbdinputstyle

     A string, the @kbdinputstyle style.

labels

     A hash reference.  Keys are normalized node names as described in the HTML Xref node in the Texinfo manual.  Instead of a node, it may also be a float label or an anchor name.  The value is the corresponding @-command element in the tree.

macros

     The associated hash reference has as keys user-defined macro names.  The value is the reference on a macro definition element as obtained by the Parser when parsing a @macro.  For example

       @macro mymacro{arg}
       coucou \arg\ after arg
       @end macro

     Is associated to a macro definition element

       {'cmdname' => 'macro',
        'args' => [{'text' => 'mymacro', 'type' => 'macro_name'},
                   {'text' => 'arg', 'type' => 'macro_arg}],
        'contents' => [{'text' => "coucou \arg\ after arg\n", 'type' => 'raw'}],
        'extra' => {'arg_line' => " mymacro{arg}\n", }}

     = item merged_indices

     The associated hash reference holds merged indices information, each key is merged in the value.  Same as setting @synindex or syncodeindex.

sections_level

     Modifier of the sections level.  Same as calling @lowersections or @raisesections.

values

     A hash reference.  Keys are names, values are the corresponding values. Same as values set by @set.

TEXINFO TREE
============

A Texinfo tree element (called element because node is overloaded in the Texinfo world) is an hash reference.  There are three main categories of tree element.  Tree elements associated with an @-command have a cmdname key holding the @-command name.  Tree elements corresponding to text fragments have a text key holding the corresponding text. Finally, the last category is other containers (hereafter called containers) which in most cases have a type key holding their name. Text fragments and @-command elements may also have an associated type when such information is needed.

The children of an @-command or container element are in the array referred to with the args key or with the contents key.  The args key is for arguments of @-commands, either in braces or on the rest of the line after the command, depending on the type of command. args is also used for the elements of a menu entry, as a menu entry is well-structured with a limited number of arguments. The contents key array holds the contents of the texinfo code appearing within a block @-command, within a container, or within a @node or sectioning @-command.

Another important key for the elements is the extra key which is associated to a hash reference and holds all kinds of information that is gathered during the parsing and may help with the conversion.

You can see examples of the tree structure by running makeinfo like this:

  makeinfo -c DUMP_TREE=1 -c TEXINFO_OUTPUT_FORMAT=parse document.texi

For a simpler, more regular representation of the tree structure, you can do:

  makeinfo -c TEXINFO_OUTPUT_FORMAT=debugtree document.texi

Element keys
------------

cmdname

     The command name of @-command elements.

text

     The text fragment of text elements.

type

     The type of the element.  For @verb it is the delimiter.  But otherwise it is the type of element considered as a container.  Frequent types encountered are paragraph for a paragraph container, brace_command_arg for the container holding the brace @-commands contents, line_arg and block_line_arg contain the arguments appearing on the line of @-commands.  Text fragments may have a type to give an information of the kind of text fragment, for example empty_spaces_before_argument is associated to spaces after a brace opening and before the argument.  Many @-commands elements don't have a type associated.

args

     Arguments in braces or on @-command line, and the elements of a menu entry.

contents

     The Texinfo appearing in the element.  For block commands, other containers, @node and sectioning commands.

parent

     The parent element.

line_nr

     An hash reference corresponding to information on the location of the element in the Texinfo input manual.  It should only be available for @-command elements, and only for @-commands that are considered to be complex enough that the location in the document is needed, for example to prepare an error message.

     The keys of the line number hash references are

     line_nr

          The line number of the @-command.

     file_name

          The file name where @-command appeared.

     macro

          The user macro name the @-command is expanded from.

extra

     A hash reference holding any additional information. See Information available in the extra key.

Element types
-------------

Types for command elements
..........................

Some types can be associated with @-commands (in addition to the element being described by cmdname), although usually there will be no type at all.  As said above, for @verb the type is the delimiter.  For a @value command that is not expanded because there is no corresponding value set, the type is the value argument string.

The following are the other possible values of type for tree elements for @-commands.

def_line

     This type may be associated with a definition command with a x form, like @defunx, @defvrx.  For the form without x, the associated def_line is the first contents element.  It is described in more details below.

command_as_argument

     This is the type of a command given in argument of @itemize, @table, @vtable or @ftable.  For example in

      @itemize @bullet
      @item item
      @end itemize

     the element corresponding with bullet has the following keys:

       'cmdname' => 'bullet'
       'type' => 'command_as_argument'

     The parent @-command has an entry in extra for the command_as_argument element:

       'cmdname' => 'itemize'
       'extra => {'command_as_argument' => $command_element_as_argument}

index_entry_command

     This is the type of index entry command like @cindex, and, more importantly user-defined index entry commands.  So for example if there is

      @defindex foo
       ...

      @fooindex index entry

     the @fooindex @-command element will have the index_entry_command type.

following_arg

     This type is set for non-alphabetic accent @-commands that don't use braces but instead have their argument right after them, as

       @~n

space_command_arg

     This type is set for accent @-commands that don't use brace but instead have their argument after some space, as

       @ringaccent A

definfoenclose_command

     This type is set for an @-command that is redefined by @definfoenclose. The beginning is in {'extra'}->{'begin'} and the end in {'extra'}->{'end'}.

Types for text elements
.......................

The text elements may have the following types (or may have no type at all):

empty_line

     An empty line (possibly containing whitespace characters only).

empty_line_after_command

empty_spaces_after_command

     The text is spaces for empty_spaces_after_command or spaces followed by a newline for empty_line_after_command, appearing after an @-command that takes an argument on the line or a block @-command.

empty_spaces_before_argument

     The text is spaces appearing after an opening brace or after a comma separating a command's arguments.

spaces_at_end

     Space at the end of an argument to a line command, at the end of an comma-separated argument for some brace commands, or at the end of bracketed content on a @multitable line or definition line.

empty_spaces_after_close_brace

     Spaces appearing after a closing brace, for some rare commands for which this space should be ignorable (like @caption).

empty_spaces_before_paragraph

     Space appearing before a paragraph beginning.

raw

     Text in an environment where it should be kept as is (in @verbatim, @verb, @html, @macro body).

last_raw_newline

     The last end of line in a raw block (except for @verbatim).

preamble_text

     Text appearing before real content, including the \input texinfo.tex.

space_at_end_menu_node

after_description_line

     Space after a node in the menu entry, when there is no description, and space appearing after the description line.

Types of container elements
...........................

The other types of element are the following.  These are containers with other elements appearing in their contents.

text_root

document_root

root_line

     These types correspond to document roots.  text_root is the document root when there is no @node or sectioning command.  When such a command appears, a new root container is used, document_root, and text_root becomes the first element in the contents of document_root. root_line is the type of the root tree when parsing Texinfo line fragments using parse_texi_line.

preamble

     This container holds the text appearing before the first content, including the \input texinfo.tex line and following blank lines.

preamble_before_setfilename

     This container holds everything that appears before @setfilename if IGNORE_BEFORE_SETFILENAME parser option is set.

paragraph

     A paragraph.  The contents of a paragraph (like other container elements for Texinfo content) are elements representing the contents of the paragraph in the order they occur, such as simple text elements without a cmdname or type, or @-command elements for commands appearing in the paragraph.

preformatted

     Texinfo code within a format that is not filled.  Happens within some block commands like @example, but also in menu (in menu descriptions, menu comments...).

brace_command_arg

brace_command_context

line_arg

block_line_arg

     Those containers occur within the args array of @-commands taking an argument.  brace_command_arg is used for the arguments to commands taking arguments surrounded by braces (and in some cases separated by commas).  brace_command_context is used for @-commands with braces that start a new context (@footnote, @caption, @math).

     line_arg is used for commands that take the texinfo code on the rest of the line as their argument (for example (@settitle, @node, @section and similar).  block_line_arg is similar but is used for commands that start a new block (which is to be ended with @end).

     For example

      @code{in code}

     leads to

      {'cmdname' => 'code',
       'args' => [{'type' => 'brace_command_arg',
                   'contents' => [{'text' => 'in code'}]}]}

misc_arg

     Used for the arguments to some special line commands whose arguments aren't subject to the usual macro expansion.  For example @set, @clickstyle, @unmacro, @comment.  The argument is associated to the text key.

menu_entry

menu_entry_leading_text

menu_entry_name

menu_entry_separator

menu_entry_node

menu_entry_description

     A menu_entry holds a full menu entry, like

       * node::    description.

     The different elements of the menu entry are directly in the menu_entry args array reference.

     menu_entry_leading_text holds the star and following spaces. menu_entry_name is the menu entry name (if present), menu_entry_node corresponds to the node in the menu entry, menu_entry_separator holds the text after the node and before the description, in most cases ::   .  Lastly, menu_entry_description is for the description.

menu_comment

     The menu_comment container holds what is between menu entries in menus.  For example in

       @menu
       Menu title

       * entry::

       Between entries
       * other::
       @end menu

     Both

       Menu title

     and

       Between entries

     will be in menu_comment.

macro_name

macro_arg

     Taken from @macro definition and put in the args key array of the macro, macro_name is the type of the text fragment corresponding to the macro name, macro_arg is the type of the text fragments correponding to macro formal arguments.

before_item

     A container for content before the first @item of block @-commands with items (@table, @multitable, @enumerate...).

table_entry

table_term

table_item

inter_item

     Those containers appear in @table, @ftable and @vtable. A table_entry container contains an entire row of the table. It contains a table_term container, which holds all the @item and @itemx lines.  This is followed by a table_item container, which holds the content that is to go into the second column of the table.

     If there is any content before an @itemx (normally only comments, empty lines or maybe index entries are allowed), it will be in a container with type inter_item.

def_line

def_item

inter_def_item

     The def_line type is either associated with a container within a definition command, or is the type of a definition command with a x form, like @deffnx.  It holds the definition line arguments. The container with type def_item holds the definition text content. Content appearing before a definition command with a x form is in an inter_def_item container.

multitable_head

multitable_body

row

     In @multitable, a multitable_head container contains all the rows with @headitem, while multitable_body contains the rows associated with @item.  A row container contains the @item and @<tab> forming a row.

bracketed

     This a special type containing content in brackets in the context where they are valid, in @math.

bracketed_def_content

     Content in brackets on definition command lines.

def_aggregate

     Contains several elements that together are a single unit on a @def* line.

bracketed_multitable_prototype

row_prototype

     On @multitable line, content in brackets is in bracketed_multitable_prototype, text not in brackets is in row_prototype.

Information available in the extra key
--------------------------------------

Extra keys available for more than one @-command
................................................

end_command

     The @end associated to the block @-command.

missing_argument

     Set for some @-commands with line arguments and a missing argument.

arg_line

     The string correspond to the line after the @-command for @-commands that have special arguments on their line, and for @macro line.

text_arg

     The string correspond to the line after the @-command for @-commands that have an argument interpreted as simple text, like @setfilename, @end or @documentencoding.

index_entry

     The index entry information (described in index_entries in details) is associated to @-commands that have an associated index entry.

misc_args

     An array holding strings, the arguments of @-commands taking simple textual arguments as arguments, like @everyheadingmarks, @frenchspacing, @alias, @synindex, @columnfractions. Also filled for @set, @clickstyle, @unmacro or @comment arguments.

spaces_before_argument

     For @-commands with opening brace followed by spaces held in a empty_spaces_before_argument element, a reference to those spaces.

spaces

     For accent commands acting on one letter only, like @ringaccent appearing like

      @ringaccent A

     there is a spaces key which holds the spaces appearing between the command and the argument.

Extra keys specific of certain @-commands or containers
.......................................................

@macro

     invalid_syntax is set if there was an error on the @macro line.  arg_line holds the line after @macro.

@node

     The arguments are in the nodes_manuals array. Each of the entries is a hash with a node_content key for an array holding the corresponding content, a manual_content key if there is an associated external manual name, and a normalized key for the normalized label, built as specified in the Texinfo manual in the HTML Xref node.

     An associated_section key holds the tree element of the sectioning command that follows the node.

@part

     The next sectioning command is in part_associated_section.

sectioning command

     The node preceding the command is in associated_node. The part preceding the command is in associated_part. If the level of the document was modified by @raisections or @lowersections, the differential level is in sections_level.

@float

@listoffloats

     If float has a second argument, and for @listoffloats argument there is a type key which is also a hash reference, with two keys. content is an array holding the associated contents, normalized holds the normalized float type.

     caption and shortcaption holds the corresponding tree elements for float.  The @caption or @shortcaption have the float tree element stored in float.

@float

@anchor

     @-commands that are targets for cross-references have a normalized key for the normalized label, built as specified in the Texinfo manual in the HTML Xref node.  There is also a node_content key for an array holding the corresponding content.

     @anchor also has region set to the special region name if in a special region (@copying, @titlepage).

@ref

@xref

@pxref

@inforef

     The node_argument entry holds a parsed node entry, like the one appearing in the nodes_manuals array for @node.

@abbr

@acronym

     The first argument normalized is in normalized.

definition command

     def_command holds the command name, without x if it is an x form of a definition command. original_def_cmdname is the original def command.

     If it is an x form, it has not_after_command set if not appearing after the definition command without x.

def_line

     For each element in a def_line, the key def_role holds a string describing the meaning of the element.  It is one of category, name, class, type, arg, typearg, spaces or delimiter, depending on the definition.

     The def_parsed_hash hash reference has these strings as keys, and the values are the corresponding elements.

@multitable

     The key max_columns holds the maximal number of columns.  If there are prototypes on the line they are in the array associated with prototypes.  If there is a @columnfractions as argument, then the columnfractions key is associated with the element for the @columnfractions command.

@enumerate

     The extra key enumerate_specification contains the enumerate argument.

@itemize

@table

@vtable

@ftable

     The command_as_argument extra key points to the @-command on as argument on the @-command line.

paragraph

     The indent or noindent key value is set if the corresponding @-commands are associated with that paragraph.

@item in @enumerate or @itemize

     The item_number extra key holds the number of this item.

@item and @tab in @multitable

     The cell_number index key holds the index of the column of the cell.

row

     The row_number index key holds the index of the row in the @multitable.

@author

     If in a @titlepage, the titlepage is in titlepage, if in @quotation or @smallquotation, the corresponding tree element is in quotation.

     The author tree element is in the author array of the @titlepage or the @quotation or @smallquotation it is associated with.

@ifclear

@ifset

     The original line is in line.

@end

     The textual argument is in command_argument. The corresponding @-command is in command.

@documentencoding

     The argument, normalized is in input_encoding_name if it is recognized. The corresponding perl encoding name is in input_perl_encoding.

@click

     In clickstyle there is the current clickstyle command.

@kbd

     code is set depending on the context and @kbdinputstyle.

definfoenclose defined commands

     begin holds the string beginning the definfoenclose, end holds the string ending the definfoenclose.

menu_entry

     The menu_entry_description and menu_entry_name keys are associated with the corresponding tree elements. The menu_entry_node value is a hash with information about the parsed node entry; its keys are the same as those appearing in the elements of the nodes_manuals array for @node.

empty_line_after_command

     The corresponding command is in command.

@inlinefmt

@inlineraw

@inlinefmtifelse

@inlineifclear

@inlineifset

     The first argument is in format.  If an argument has been determined as being expanded by the Parser, the index of this argument is in expand_index.  Index numbering begins at 0, but the first argument is always the format or flag name, so, if set, it should be 1 or 2 for @inlinefmtifelse, and 1 for other commands.

Other information set by the parser
-----------------------------------

The parser creates an array of nodes and stores this in the nodes key of the parser object.

Each element in the tree corresponding to a node contaning a menu has a menus key which refers to an array of references to menu elements occuring in the node.

These are both used by the Texinfo::Structuring module.

SEE ALSO
========

Texinfo manual (http://www.gnu.org/s/texinfo/manual/texinfo/)

AUTHOR
======

Patrice Dumas, <pertusus@free.fr>