Zach Laine
3fd285c014
Correct memory safety issue in converting token iteratos to underlying
...
iterators.
See #202 .
2024-12-01 20:26:22 -06:00
Zach Laine
bc6e9e3447
Grooming.
2024-11-30 17:35:06 -06:00
Zach Laine
eab7e82988
Slight simplification of code trying to detect token iterators.
...
See #202 .
2024-11-30 17:24:05 -06:00
Zach Laine
655870000b
Remove a mootetd TODO.
2024-11-29 20:17:49 -06:00
Zach Laine
039453079e
Cruft removal.
2024-11-29 20:11:50 -06:00
Zach Laine
a908e950d5
Address TODO about removing detail::defulat_flags().
2024-11-29 20:10:46 -06:00
Zach Laine
87e00a173d
Add static_assert(std::integral<T> || std::floating_point<T>) to
...
detail::token_with_value, just in case.
See #202 .
2024-11-29 20:03:39 -06:00
Zach Laine
3176c6f823
Fix error detected in tests of token caching introduced in doc examples. Make
...
sure not to copy tokens_view within the *parse() call graph.
See #202 .
2024-11-29 20:00:28 -06:00
Zach Laine
92c4993b87
Documentation pass on the changes made so far to support token parsing.
...
See #202 .
2024-11-29 20:00:28 -06:00
Zach Laine
9d67b0df7f
Cruft removal.
2024-11-29 17:17:49 -06:00
Zach Laine
d4f4589ead
Add TODOs.
2024-11-29 16:29:55 -06:00
Zach Laine
178d62a250
Preclude skipper *parse() overloads from token parsing with a static_assert.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
e654c9fda7
Macro guard private members of tokens_view from inclusion in Doxygen reference
...
entry.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
9dbf30241d
Plumb a pointer to the tokens_view through to the parse context; change
...
seq_parser to use the tokens_view pointer from the contet to trim the cache at
an expectation point.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
f298bfe59b
Remove the support for token_view::iterator -> underlying iterator translation
...
in *parse_impl(); add support to the error handlers and their support
functions directly instead. There are simply too many APIs there that need
the translation to leave it to other code.
Add lex_error exception type, and add support for all the APIs that used to
take a parse_error param to now take either a parse_error or a lex_error.
Throw lex_error from failed parsing of lexed tokens in detail::make_token().
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
07cd667a91
Add TODOs.
2024-11-29 16:29:55 -06:00
Zach Laine
00510ed962
Add cases to tracing.cpp to cover the printing of token parsers; fix errors.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
1a5b7467ca
Add a TODO.
2024-11-29 16:29:55 -06:00
Zach Laine
13a52e10f9
Cruft removal.
2024-11-29 16:29:55 -06:00
Zach Laine
67c3ec180c
Add a smoke test for using token views as input to parse(); fix errors.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
8658f8cd6b
Unify the implementations of the callback parse() API.
2024-11-29 16:29:55 -06:00
Zach Laine
f69d7acdd9
Unify the implementations of the non-callback parse() API, to make it posible
...
for the new token parsingcode path to use that one unified implementation too.
The previous implementation dispatched like parse() -> prefix_parse(), the
latter of which is incompatible with token parsing.
2024-11-29 16:29:55 -06:00
Zach Laine
542bdb0e0e
Unify the detail::{skip_,}parse_impl overloads.
2024-11-29 16:29:55 -06:00
Zach Laine
f3e326e344
Unify the two detail::skip_parse_impl overloads.
2024-11-29 16:29:55 -06:00
Zach Laine
027d861b08
Unify the two detail::callback_*parse_impl overloads.
2024-11-29 16:29:55 -06:00
Zach Laine
6ab8f96e19
Unify the two detail::parse_impl overloads.
2024-11-29 16:29:55 -06:00
Zach Laine
2b518bc74d
Fix multiple thinkos in tokens_view.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
1421592876
Add a std::string data member, and an override of what() to parser_error, to
...
ensure that the messge is preserved exactly when translating exceptions in
failed lexer parsing.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
092a76173b
Change token<> to include the position of the start of the token in the
...
underlying sequence, and change the way that the error handler is invoked, so
that it detects token iterators, and passes iterators into the underlying
range to the error handler, instead of the token iterators.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
8830dfed02
Alter detail::make_input_subrange() to return its input unchanged when the
...
range's value type is a specializtion of token.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
935322649b
Add an iterator/sentinel overload of detail::make_input_subrange(), so that an
...
overload of this function is called in every *parse() overload.
2024-11-29 16:29:55 -06:00
Zach Laine
7566dbdde1
Remove the pointer case from detail::make_input_range().
2024-11-29 16:29:55 -06:00
Zach Laine
298bae0058
Sketch in the printing implemetnation for token_parser.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
b6c1229c54
Reorganize the lexer-related header inclusion scheme slightly. lexer.hpp is
...
now required to come before parser.hpp.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
793e519eb8
Put detail::make_input_subrange back into parer.hpp.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
416607c954
Add token_parser::operator()(range) to support matchign against tokens that
...
producestring_views.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
e419ef2a60
Completely rethink the relationship among token_parser, token_spec_t, and
...
parser_interface. token_spec is now a variable template that generates a
parser_interface wrapping a token_parser, which parameterized on the
token_spec_t. This way, a single token_spec use can be used to specify how to
lex, and how to parse.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
1a405f8133
Sketch in operator() on token_spec_t.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
fbc21ef2fd
Sketch in support for matching tokens against expected values in token_parser.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
f00f4dfa75
Initial, partial sketch of token_parser.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
358adf247a
Address the remaining non-documentation TODOs in the lexer header and tests.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
ee8ab13779
Extend the lexer tests, addressing some of the testing TODOs; fix errors.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
391bb2b5b0
Factor out a lot of the lexer.cpp tests into a separate file.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
ad64fb6973
Use bitfields to reduce the size of token by a few bytes.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
6ed1152390
Switch from double to long double as token's floating point representation.
...
Therer's room for it, since it's in a union with a string_view anyway.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
51cc855dd7
Grooming.
2024-11-29 16:29:55 -06:00
Zach Laine
bf336fb096
Add much longer lexing test; fix errors.
...
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
63483cb284
Specify the true/false and number token_specs in the adobe lexer test code to
...
use a bool and a double as its value, repsectively, instead of a string_view.
See #202 .
2024-11-29 16:29:55 -06:00
Zach Laine
8b24206aee
Add a TODO.
2024-11-29 16:29:55 -06:00
Zach Laine
49213c428e
Change all the existing lex runs to compare their output against an expected
...
sequence of tokens.
See #202 .
2024-11-29 16:29:55 -06:00
