mirror of
https://github.com/boostorg/tokenizer.git
synced 2026-01-23 18:12:12 +00:00
146 lines
4.0 KiB
HTML
146 lines
4.0 KiB
HTML
<html>
|
|
|
|
<head>
|
|
<meta http-equiv="Content-Type"
|
|
content="text/html; charset=iso-8859-1">
|
|
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
|
|
<title>TokenizerFunction Concept</title>
|
|
</head>
|
|
|
|
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
|
|
vlink="#551A8B" alink="#FF0000">
|
|
|
|
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
|
|
height="86"> </p>
|
|
|
|
<h1 align="center">TokenizerFunction Concept</h1>
|
|
|
|
<p>A TokenizerFunction is a functor whose purpose is to parse a
|
|
given sequence until exactly 1 token has been found or the end is
|
|
reached. It then updates the token, and informs the caller of the
|
|
location in the sequence of the next element immediately after
|
|
the last element of the sequence that was parsed for the current
|
|
token. </p>
|
|
|
|
<h2>Refinement of</h2>
|
|
|
|
<p>Assignable, CopyConstructable</p>
|
|
|
|
<h2>Notation</h2>
|
|
|
|
<table border="1">
|
|
<tr>
|
|
<td valign="top"><tt>X</tt> </td>
|
|
<td valign="top">A type that is a model of
|
|
TokenizerFunction</td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><tt>func</tt> </td>
|
|
<td valign="top">Object of type <tt>X</tt> </td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top"><tt>tok</tt></td>
|
|
<td valign="top">Object of Token</td>
|
|
</tr>
|
|
<tr>
|
|
<td>next</td>
|
|
<td>iterator that points to the first unparsed element of
|
|
the sequence being parsed</td>
|
|
</tr>
|
|
<tr>
|
|
<td>end</td>
|
|
<td>iterator that points to the past the end of the
|
|
sequence being parsed</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Definitions</h2>
|
|
|
|
<p>A token is the result of parsing a sequence.</p>
|
|
|
|
<h2>Valid expressions</h2>
|
|
|
|
<p>In addition to the expression in Assignable and
|
|
CopyConstructable the following expressions are valid</p>
|
|
|
|
<table border="1">
|
|
<tr>
|
|
<th>Name </th>
|
|
<th>Expression </th>
|
|
<th>Return type </th>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">Functor</td>
|
|
<td valign="top"><tt>func(next, end, tok)</tt></td>
|
|
<td valign="top"><tt>bool</tt></td>
|
|
</tr>
|
|
<tr>
|
|
<td valign="top">reset</td>
|
|
<td valign="top"><tt>reset()</tt></td>
|
|
<td valign="top"><tt>void</tt></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Expression semantics</h2>
|
|
|
|
<p>In addition to the expression semantics in Assignable and
|
|
CopyConstructable, TokenizerFunction has the following expression
|
|
semantcs</p>
|
|
|
|
<table border="1">
|
|
<tr>
|
|
<th>Name </th>
|
|
<th>Expression </th>
|
|
<th>Precondition </th>
|
|
<th>Semantics </th>
|
|
<th>Postcondition </th>
|
|
</tr>
|
|
<tr>
|
|
<td>operator()</td>
|
|
<td><tt>func(next, end, tok)</tt></td>
|
|
<td><tt>next</tt> and <tt>end</tt> are valid iterators to
|
|
the same sequence. next is a reference the function is
|
|
free to modify. tok is constructed. </td>
|
|
<td>The return value indicates whether a new token was
|
|
found in the sequence (next,end]</td>
|
|
<td>If the return value is true, the new token is
|
|
assigned to tok. next is always updated to the position
|
|
where parsing should start on the subsequent call.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>reset</td>
|
|
<td><tt>reset()</tt></td>
|
|
<td><tt>None</tt></td>
|
|
<td>Clears out all state variables that are used by the
|
|
object in parsing the current sequence.</td>
|
|
<td>A new sequence to parse can be given.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<h2>Complexity guarantees</h2>
|
|
|
|
<p>No guarantees. Models of TokenizerFunction are free to define
|
|
their own complexity</p>
|
|
|
|
<h2>Models</h2>
|
|
|
|
<p><a href="escaped_list_separator.htm">escaped_list_separator</a></p>
|
|
|
|
<p><a href="offset_separator.htm">offset_separator</a></p>
|
|
|
|
<p><a href="char_delimiters_separator.htm">char_delimiters_separator</a></p>
|
|
|
|
<p> </p>
|
|
|
|
<p> </p>
|
|
|
|
<hr>
|
|
|
|
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
|
|
modify, sell and distribute this document is granted provided
|
|
this copyright notice appears in all copies. This document is
|
|
provided "as is" without express or implied warranty,
|
|
and with no claim as to its suitability for any purpose.</p>
|
|
</body>
|
|
</html>
|