* Add validation for --benchmark-samples to prevent crash with zero samples
* Add automated test for benchmark samples validation
* Update baselines for benchmark-samples validation
* Add missing test for benchmark samples validation
---------
Co-authored-by: Chan Aung <chan@Thinkpad.localdomain>
The original implementation of `UNSCOPED_X` message macros used a
clever hack to make the original implementation simpler: construct
an instance of `ScopedMessage` to manage its lifetime, but store
it in a vector, so its lifetime is not actually scope-based, and
we can manage it through the vector instance.
This hack made it so that the lifetime of the vector that manages
the fake `ScopedMessage`s must be outlived by the vector with the
actual messages. Originally this wasn't a problem, because they both
lived inside the run context instance. However, since then these
vectors became globals and thread-local. When this happened, it
still wasn't a problem; the two globals were declared in the right
order, so they were destroyed in the right order as well.
Then, in f80956a43a, these globals
were turned into magic static globals to improve their behaviour
in MSVC's Debug build mode. This caused their lifetimes to be
runtime-dependent; if a specific test thread added its first scoped
message before it added first unscoped message, the lifetimes
would be correct. If it instead added first unscoped message
before adding first scoped message, then there **might** be
invalid reads during thread destruction.
The fix is simple: do things properly and manage the lifetime of
messages in `UNSCOPED_X` explicitly. Then we don't have to deal
with the destruction of fake `ScopedMessage`s while the thread is
being destroyed, and the lifetime of the two vectors is no longer
tied together.
I also threw them both into a new type, to encapsulate some of the
unscoped message logic.
Specifically, this commit makes the `-c`/`--section` parameter
strictly ordered and hierarchical, unlike how it behaved before,
which was a huge mess -- see #3038 for details.
Closes#3038
Because it requires full rebuild of the base library, it adds about
20% to the build time of the test suite. Between the fact that the
option is deprecated, and that Bazel has added the `BAZEL_TEST`
env var _years_ ago, nobody should be using it, and the chance
of breakage is tiny, the test is not worth its compile-time cost.
This commit causes small (~5%) slowdown when processing strings
that consist mostly of escaped characters, but improves throughput
by about 100% for strings that consist mostly of characters that
do not need escaping.
This cherry-picks the source changes from PR made by @pkleymonov-qnx,
without including the build path changes that are irrelevant unless
we can add a QNX target to our CI.
Close#2953
This tells Catch2 to create an empty file at specified path before
the tests start, and delete it after the tests finish. This allows
callers to catch cases where the test binary silently exits before
finishing (e.g. via call to `exit(0)` inside the code under test),
by looking whether the file still exists.
Closes#3020
Forbid deducing reference types for m_predicate in FilterGenerator to prevent dangling references.
This is needed for out-of-line predicates to work correctly instead of undefined behavior or crashes.
---------
Co-authored-by: Tek Mate <mate.tek@evosoft.com>
As with the JSON writer, the old code was made to be simple and
for each char just decided whether it needs escaping, or should be
written as-is. The new code instead looks for characters that need
escaping and batches writes of characters that do not.
This provides 4-8x speedup (length dependent) for writing strings
that do not need escaping, and keeps roughly the same performance
for those that do need escaping.
The old code was exceedingly simple, as it went char-by-char and
decided whether to write it to the output stream as-is, or escaped.
This caused a _lot_ of stream writes of individual characters.
The new code instead looks for characters that need escaping, and
bulk-writes the non-escaped characters in between them. This leads
to about the same performance for strings that comprise of only
escaped characters, and 3-10x improvement for strings without any
escaping needed.
In practice, we should expect the former rather than the latter,
but this is still nice improvement.
This stops tests failing falsely if the assertion passed, but the
stringification itself failed as the assertion was sent to the reporter.
I don't think that stringification should be fallible, but the
overhead in compilation ended up being small enough (<0.5% on `SelfTest`)
that it might be worth implementing, in case there is more users
with weird `StringMaker`s than just MongoDB.
Closes#2980
As it turns out, enums can be used to declare bitfields, and we
cannot form a reference to a bitfield in the cpature. Thus, we add
`std::is_enum` as a criteria to the default for `capture_by_value`,
so that enum-based bitfields are also captured by value and thus
decomposable.
Closes#3001
Bad handling of `TestFailureException` when translating unexpected
exceptions inside assertion macros led to the unexpected exceptions
handling erroring out through throwing the same exception again.
This was then backstopped by the machinery for handling uncaught
exceptions from assertions, which is normally used by the
`CATCH_CONFIG_FAST_COMPILE` machinery, where we assume that it can
only be invoked because the assertion macros are not configured to
catch assertions.
Closes#1292
This means that:
1) Scoped messages are always removed at the end of their scope,
even if the scope ended due to an exception.
2) Scoped messages outlive section end, if that section's scope is
enclosed in their own.
Previously neither of these were true, which has led to a number
of surprising behaviour, where e.g. this:
```cpp
TEST_CASE() {
try {
INFO( "some info" );
throw std::runtime_error( "ex" );
} catch (std::exception const&) {}
REQUIRE( false );
}
```
would print "some info" as the message for the assertion, while this:
```cpp
TEST_CASE() {
INFO("Hello");
SECTION("dummy") {}
REQUIRE(false);
}
```
would not print out "Hello" as the message for the assertion.
This had an underlying reason, in that it was trying to helpfully
keep the messages around in case of unexpected exceptions, so that
code like this:
```cpp
TEST_CASE() {
auto [input, expected] = GENERATE(...);
CAPTURE(input);
auto result = transform(input); // throws
REQUIRE(result == expected);
}
```
would report the value of `input` when `transform` throws. However,
it was surprising in practice and was causing various issues around
handling of messages in other cases.
Closes#1759Closes#2019Closes#2959
The fast path allows `RunContext` to skip disabling output redirect,
and notifying the reporters, turning `RunContext::notifyAssertionStarted`
into a no-op. This improves the overall performance of assertion handling
significantly, and also prepares ground for future changes around
assertion handling and thread safety.
For simple 10M assertion run, this improves the running time by ~30%
in Debug build and ~40% in Release build.
For backwards-compatibility reasons, the fast path is disabled
by default. However, none of the first party reporters use the
`assertionStarting` event, so all first party reporters are opted-in.
There were only two places where we used the full `AssertionInfo`
instance in `m_lastAssertionInfo`:
1) when reporting unexpected exception from running a test case
2) when reporting fatal error
because in those two places we do not have access to a real
instance of `AssertionInfo`, but we still need to send one to the
reporters. As a bonus, in both of these places we were already
constructing a fake-ish assertion info, by using the last encountered
source location, but dummying out the other information.
Instead, we only keep track of the last encountered source location,
and construct the dummy `AssertionInfo` on-demand.
This finishes the set of refactoring around `m_lastAssertionInfo`
in `RunContext` and improves the performance of running assertions
by ~5% in both Debug and Release mode.
--------------
Note that this change also causes small difference in output. It
could be avoided by having an invalidation flag and tracking where
the information would be invalidated before, but the difference
includes more precise line location for unexpected errors (both
exceptions and fatals), so I prefer the new output.
Adding unreachable to `FAIL` and `SKIP` made Clang become very good
at figuring out that tests of `FAIL` and `SKIP` will never return
and thus could be marked as [[noreturn]].
To avoid introducing lot of warning suppression noise into the
test files, we just disable it instead.
Right now `TEMPLATE_TEST_CASE_SIG` fails to compile when the signature contains only types:
```
TEMPLATE_TEST_CASE_SIG(
"TemplateTestSig: compiles with two type parameters",
"[template][onlytypes]",
((typename U, typename V), U, V), (int,int)) {}
```
The trick is to resolve the ambiguity between the two overloads of
`get_wrapper` (`TypeList` and `Nttp`) by making one match more strongly.
We also need to allow `reg_test` to register more than one type.
Add unit tests.
Fixes#2680
---------
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
We still want to build VS 2017 through AppVeyor, and those images
have CMake 3.16.2 installed. We could install newer CMake as part
of the build, but since we don't use newer CMake features yet, this
is simpler.
`gmtime*` on Windows fails on dates pre 1970, and because we didn't
check the return code, we would then pass invalid `tm` struct to
`strftime` causing it to assert.
Closes#2944
The main reason for this is to be able to distinguish between
different errors (or "errors") based on the return code. Before
this change, it was impossible to use the exit code to figure out
whether a test binary failed because all tests were skipped or
because exactly 4 assertions have failed.
This meant that using `catch_discover_tests` and telling it to
check for exit code == 4 to determine skipped tests could lead to
false negatives.
