diff --git a/benchmark/b.sh b/benchmark/b.sh new file mode 100755 index 0000000..dcd010e --- /dev/null +++ b/benchmark/b.sh @@ -0,0 +1,6 @@ +ninja +rm benchmark.csv +./deep_stack_leaf +./deep_stack_tl +./deep_stack_result +./deep_stack_outcome diff --git a/benchmark/benchmark.md b/benchmark/benchmark.md index e674b03..ea6391f 100644 --- a/benchmark/benchmark.md +++ b/benchmark/benchmark.md @@ -1,25 +1,35 @@ # Benchmark -The LEAF github repository contains two similar benchmarking programs, one using LEAF, the other using Boost Outcome, that simulate transporting error objects across 32 levels of function calls, measuring the performance of the two libraries. +The LEAF github repository contains two similar benchmarking programs, one using LEAF, the other configurable to use `tl::expected` or Boost Outcome, that simulate transporting error objects across 10 levels of stack frames, measuring the performance of the three libraries. -Official library documentation: +Links: * LEAF: https://zajo.github.io/leaf +* `tl::expected`: https://github.com/TartanLlama/expected * Boost Outcome: https://www.boost.org/doc/libs/release/libs/outcome/doc/html/index.html ## Library design considerations -It is important to understand that LEAF and Outcome serve similar purpose but follow very different design philosophy. The benchmarks are comparing apples and oranges. +LEAF serves a similar purpose to other error handling libraries, but its design is very different. The benchmarks are comparing apples and oranges. -The main design difference is that LEAF has a very strong bias towards the use case where, after a call to a function which may fail, any failures are not handled but instead forwarded to the caller. When we merely check "do we have a failure", LEAF does not bother to make available the error objects associated with the failure. This saves a lot of cycles. +The main design difference is that when using LEAF, error objects are not communicated in return values. In case of a failure, the `leaf::result` object transports only an `int`, the unique error ID. -In contrast, Outcome always delivers error objects to each level of function call, even when the user is not handling errors at that point, and therefore has no intention of accessing them. +Error objects skip the error-neutral functions in the call stack and get moved directly to the relevant error-handling scope. This mechanism does not depend on RVO or any other optimization: as soon as the program passes an error object to LEAF, it moves it to the correct error handling scope. -This difference is reflected in how each library is used. The simple "check-only" case is nearly identical, e.g.: +Other error-handling libraries instead couple the static type of the return value of *all* error-neutral functions with the error type an error-reporting function may return. This approach suffers from the same problems as statically-enforced exception specifications: + +* It's difficult or impossible to use in generic contexts, and +* It impedes interoperability between the many different error types a non-trivial program must deal with. + +(The Boost Outcome library is also capable of avoiding such excessive coupling, by passing for the third `P` argument in the `outcome` template a pointer that erases the exact static type of the object being transported. However, this would require a dynamic memory allocation). + +## Syntax + +The most common check-only use case looks almost identically in LEAF and in Boost Outcome (`tl::expected` lacks a similar macro): ```c++ // Outcome { - OUTCOME_TRY(v, f()); // Check for errors, forward failures to the caller + BOOST_OUTCOME_TRY(v, f()); // Check for errors, forward failures to the caller // If control reaches here, v is the successful result (the call succeeded). } ``` @@ -32,28 +42,35 @@ This difference is reflected in how each library is used. The simple "check-only } ``` -However, when we want to handle failures, in Outcome accessing the error object (which is always available in the returned object) is a simple continuation of the error check: +However, when we want to handle failures, in Boost Outcome and in `tl::expected`, accessing the error object (which is always stored in the return value) is a simple continuation of the error check: ```c++ -// Outcome +// Outcome, tl::expected if( auto r = f() ) return r.value()+1; // No error else { // Error! switch( r.error() ) { - error_enum::error1: /* handle */ break; - error_enum::error2: /* handle */ break; - default: return r.as_failure(); + error_enum::error1: + /* handle error_enum::error1 */ + break; + + error_enum::error2: + /* handle error_enum::error2 */ + break; + + default: + /* handle any other failure */ } } ``` -When using LEAF, we must explicitly state our intention to handle some errors, not just check for failures: +When using LEAF, we must explicitly state our intention to handle errors, not just check for failures: ```c++ // LEAF -return leaf::try_handle_some( +leaf::try_handle_all []() -> leaf::result { LEAF_AUTO(v, f()); @@ -61,15 +78,19 @@ return leaf::try_handle_some( }, []( leaf::match ) { - /* handle */ + /* handle error_enum::error1 */ }, []( leaf::match ) { - /* handle */ + /* handle error_enum::error2 */ + }, + []() + { + /* handle any other failure */ } ); ``` -The use of `try_handle_some` reserves storage on the stack for the error object types being handled (in this case, `error_enum`). If the failure is either `error_enum::error1` or `error_enum::error2`, the matching error handling lambda is invoked. Otherwise, the failure is automatically forwarded to the caller. +The use of `try_handle_all` reserves storage on the stack for the error object types being handled (in this case, `error_enum`). If the failure is either `error_enum::error1` or `error_enum::error2`, the matching error handling lambda is invoked. ## Code generation considerations @@ -77,9 +98,9 @@ Benchmarking C++ programs is tricky, because we want to prevent the compiler fro The primary approach we use to prevent the compiler from optimizing everything out to nothing is to base all computations on a call to `std::rand()`. This adds cycles, but it also makes the benchmark more realistic, since functions which may legitimately fail should do _some_ real work. -When benchmarking error handling, it makes sense to measure the time it takes to return a result or error across multiple stack frames. This calls for disabling inlining. On the other hand, in C++ inlining is often possible, so that case must be measured as well. +When benchmarking error handling, it makes sense to measure the time it takes to return a result or error across multiple stack frames. This calls for disabling inlining. -The technique used to disable inlining in this benchmark is to mark functions as `__attribute__((noinline))` / `__declspec(noinline)`. This is imperfect, because optimizers can still peek into the body of the function and optimize things out, as is seen in this example: +The technique used to disable inlining in this benchmark is to mark functions as `__attribute__((noinline))`. This is imperfect, because optimizers can still peek into the body of the function and optimize things out, as is seen in this example: ```c++ __attribute__((noinline)) int val() {return 42;} @@ -93,18 +114,18 @@ Which on clang 9 outputs: ```x86asm val(): - mov eax, 42 - ret + mov eax, 42 + ret main: - mov eax, 42 - ret + mov eax, 42 + ret ``` It does not appear that anything like this is occurring in our case, but it is still a possibility. > NOTES: > -> - For benchmarking, both programs are compiled with exception handling disabled. +> - The benchmarks are compiled with exception handling disabled. > - LEAF is able to work with external `result<>` types. The benchmark uses `leaf::result`. ## Show me the code! @@ -125,42 +146,42 @@ Generates this code on clang ([Godbolt](https://godbolt.org/z/4AtHMk)): ```x86asm g(): # @g() - push rbx - sub rsp, 32 - mov rbx, rdi - lea rdi, [rsp + 8] - call f() - mov eax, dword ptr [rsp + 8] - mov ecx, eax - and ecx, 3 - cmp ecx, 2 - je .LBB0_4 - cmp ecx, 3 - jne .LBB0_2 - mov eax, dword ptr [rsp + 16] - add eax, 1 - mov dword ptr [rbx], 3 - mov dword ptr [rbx + 8], eax - mov rax, rbx - add rsp, 32 - pop rbx - ret + push rbx + sub rsp, 32 + mov rbx, rdi + lea rdi, [rsp + 8] + call f() + mov eax, dword ptr [rsp + 8] + mov ecx, eax + and ecx, 3 + cmp ecx, 2 + je .LBB0_4 + cmp ecx, 3 + jne .LBB0_2 + mov eax, dword ptr [rsp + 16] + add eax, 1 + mov dword ptr [rbx], 3 + mov dword ptr [rbx + 8], eax + mov rax, rbx + add rsp, 32 + pop rbx + ret .LBB0_4: - mov dword ptr [rbx], 2 - movups xmm0, xmmword ptr [rsp + 16] - mov qword ptr [rsp + 24], 0 - movups xmmword ptr [rbx + 8], xmm0 - mov qword ptr [rsp + 16], 0 - mov rax, rbx - add rsp, 32 - pop rbx - ret + mov dword ptr [rbx], 2 + movups xmm0, xmmword ptr [rsp + 16] + mov qword ptr [rsp + 24], 0 + movups xmmword ptr [rbx + 8], xmm0 + mov qword ptr [rsp + 16], 0 + mov rax, rbx + add rsp, 32 + pop rbx + ret .LBB0_2: - mov dword ptr [rbx], eax - mov rax, rbx - add rsp, 32 - pop rbx - ret + mov dword ptr [rbx], eax + mov rax, rbx + add rsp, 32 + pop rbx + ret ``` > Description: @@ -190,10 +211,10 @@ We get: ```x86asm g(): # @g() - mov rax, rdi - mov dword ptr [rdi], 3 - mov dword ptr [rdi + 8], 43 - ret + mov rax, rdi + mov dword ptr [rdi], 3 + mov dword ptr [rdi + 8], 43 + ret ``` With a less trivial definition of `f`: @@ -202,9 +223,9 @@ With a less trivial definition of `f`: leaf::result f() { if( rand()%2 ) - return 42; + return 42; else - return leaf::new_error(); + return leaf::new_error(); } leaf::result g() @@ -218,49 +239,43 @@ We get ([Godbolt](https://godbolt.org/z/4P7Jvv)): ```x86asm g(): # @g() - push rbx - mov rbx, rdi - call rand - test al, 1 - jne .LBB1_5 - mov eax, dword ptr fs:[boost::leaf::leaf_detail::id_factory::next_id@TPOFF] - test eax, eax - je .LBB1_3 - mov dword ptr fs:[boost::leaf::leaf_detail::id_factory::next_id@TPOFF], 0 - jmp .LBB1_4 + push rbx + mov rbx, rdi + call rand + test al, 1 + jne .LBB1_5 + mov eax, dword ptr fs:[boost::leaf::leaf_detail::id_factory::next_id@TPOFF] + test eax, eax + je .LBB1_3 + mov dword ptr fs:[boost::leaf::leaf_detail::id_factory::next_id@TPOFF], 0 + jmp .LBB1_4 .LBB1_5: - mov dword ptr [rbx], 3 - mov dword ptr [rbx + 8], 43 - mov rax, rbx - pop rbx - ret + mov dword ptr [rbx], 3 + mov dword ptr [rbx + 8], 43 + mov rax, rbx + pop rbx + ret .LBB1_3: - mov eax, 4 - lock xadd dword ptr [rip + boost::leaf::leaf_detail::id_factory::counter], eax - add eax, 4 + mov eax, 4 + lock xadd dword ptr [rip + boost::leaf::leaf_detail::id_factory::counter], eax + add eax, 4 .LBB1_4: - and eax, -4 - or eax, 1 - mov dword ptr fs:[boost::leaf::leaf_detail::id_factory::last_id@TPOFF], eax - mov dword ptr [rbx], eax - mov rax, rbx - pop rbx - ret + and eax, -4 + or eax, 1 + mov dword ptr fs:[boost::leaf::leaf_detail::id_factory::last_id@TPOFF], eax + mov dword ptr [rbx], eax + mov rax, rbx + pop rbx + ret ``` Above, the call to `f()` is inlined, most of the code is from the initial error reporting machinery in LEAF. ## Benchmark matrix dimensions -The benchmark matrix has 4 dimensions: +The benchmark matrix has 2 dimensions: -1. Check only vs. handle some errors: - - a. Each function calls the lower level function, does work on success, forwards errors to the caller, but does not handle them (the root of the call chain handles the error). - - b. Each fourth level of function call doesn't only check, but also handles some errors, forwarding other errors to the caller. - -2. Error object type: +1. Error object type: a. The error object transported in case of a failure is of type `e_error_code`, which is a simple `enum`. @@ -268,32 +283,37 @@ The benchmark matrix has 4 dimensions: c. The error object transported in case of a failure is of type `e_heavy_payload`, a `struct` of size 4096. -3. Error rate: 2%, 50%, 98% - -4. Inline vs. noinline (32 levels of function calls). +2. Error rate: 2%, 98% ## Source code [deep_stack_leaf.cpp](deep_stack_leaf.cpp) -[deep_stack_outcome.cpp](deep_stack_outcome.cpp) +[deep_stack_other.cpp](deep_stack_other.cpp) ## Godbolt -LEAF provides a single header which makes it very easy to use online. To see the generated code for the benchmark program, you can copy and paste the following into Godbolt: +LEAF and `tl::expected` both provide a single header, which makes it very easy to use them online. To see the generated code for the benchmark program, you can copy and paste the following into Godbolt: +LEAF: ```c++ -#include "https://raw.githubusercontent.com/zajo/leaf/master/include/boost/leaf/all.hpp" -#include "https://raw.githubusercontent.com/zajo/leaf/master/benchmark/deep_stack_leaf.cpp" +#include "https://raw.githubusercontent.com/zajo/leaf/master/include/boost/> af/all.hpp" +#include "https://raw.githubusercontent.com/zajo/leaf/master/benchmark/> deep_stack_leaf.cpp" ``` See https://godbolt.org/z/DTk4N4. +`tl::expected`: +```c++ +#include "https://raw.githubusercontent.com/TartanLlama/expected/master/include/tl/expected.hpp" +#include "https://raw.githubusercontent.com/zajo/leaf/master/benchmark/> deep_stack_other.cpp" +``` + ## Build options To build both versions of the benchmark program, the compilers are invoked using the following command line options: -* `-std=c++17`: Required by Outcome (LEAF only requires C++11); +* `-std=c++17`: Required by other libraries (LEAF only requires C++11); * `-fno-exceptions`: Disable exception handling; * `-O3`: Maximum optimizations; * `-DNDEBUG`: Disable asserts. @@ -304,187 +324,96 @@ In addition, the LEAF version is compiled with: ## Results -Below is the output the benchmark programs running on a MacBook Pro. The tables show the elapsed time for returning a result across 32 levels of function calls, depending on the error type, the action taken at each level, whether inlining is enabled, and the rate of failures. In addition, the programs generate a `benchmark.csv` file in the current working directory. +Below is the output the benchmark programs running on a MacBook Pro. The tables show the elapsed time for 10,000 iterations of returning a result across 10 stack frames, depending on the error type and the rate of failures. In addition, the programs generate a `benchmark.csv` file in the current working directory. +### gcc: -The following tables show elapsed time +> `leaf::result`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 719 | 562 +> e_system_error | 682 | 1251 +> e_heavy_payload | 787 | 5995 +> +> `tl::expected`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 1283 | 847 +> e_system_error | 858 | 5735 +> e_heavy_payload | 1083 | 21715 +> +> `outcome::result`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 1254 | 858 +> e_system_error | 754 | 1309 +> e_heavy_payload | 8925 | 23146 +> +> `outcome::outcome`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 830 | 1236 +> e_system_error | 991 | 2458 +> e_heavy_payload | 10113 | 25330 -#### 10000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096 (clang, LEAF): +### clang: -Error type | At each level | Inlining | Rate | μs -----------------|--------------------|----------|------:|-------: -e_error_code | LEAF_AUTO | Disabled | 2% | 920 -e_error_code | LEAF_AUTO | Enabled | 2% | 135 -e_error_code | try_handle_some | Disabled | 2% | 1407 -e_error_code | try_handle_some | Enabled | 2% | 708 -e_error_code | LEAF_AUTO | Disabled | 50% | 916 -e_error_code | LEAF_AUTO | Enabled | 50% | 215 -e_error_code | try_handle_some | Disabled | 50% | 1467 -e_error_code | try_handle_some | Enabled | 50% | 851 -e_error_code | LEAF_AUTO | Disabled | 98% | 803 -e_error_code | LEAF_AUTO | Enabled | 98% | 204 -e_error_code | try_handle_some | Disabled | 98% | 1671 -e_error_code | try_handle_some | Enabled | 98% | 907 -e_system_error | LEAF_AUTO | Disabled | 2% | 1255 -e_system_error | LEAF_AUTO | Enabled | 2% | 204 -e_system_error | try_handle_some | Disabled | 2% | 1482 -e_system_error | try_handle_some | Enabled | 2% | 731 -e_system_error | LEAF_AUTO | Disabled | 50% | 1147 -e_system_error | LEAF_AUTO | Enabled | 50% | 362 -e_system_error | try_handle_some | Disabled | 50% | 1821 -e_system_error | try_handle_some | Enabled | 50% | 858 -e_system_error | LEAF_AUTO | Disabled | 98% | 977 -e_system_error | LEAF_AUTO | Enabled | 98% | 211 -e_system_error | try_handle_some | Disabled | 98% | 1585 -e_system_error | try_handle_some | Enabled | 98% | 952 -e_heavy_payload | LEAF_AUTO | Disabled | 2% | 1089 -e_heavy_payload | LEAF_AUTO | Enabled | 2% | 154 -e_heavy_payload | try_handle_some | Disabled | 2% | 1405 -e_heavy_payload | try_handle_some | Enabled | 2% | 824 -e_heavy_payload | LEAF_AUTO | Disabled | 50% | 1476 -e_heavy_payload | LEAF_AUTO | Enabled | 50% | 576 -e_heavy_payload | try_handle_some | Disabled | 50% | 2494 -e_heavy_payload | try_handle_some | Enabled | 50% | 1338 -e_heavy_payload | LEAF_AUTO | Disabled | 98% | 1644 -e_heavy_payload | LEAF_AUTO | Enabled | 98% | 931 -e_heavy_payload | try_handle_some | Disabled | 98% | 2646 -e_heavy_payload | try_handle_some | Enabled | 98% | 1676 - -#### 10000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096 (clang, Outcome): - -Error type | At each level | Inlining | Rate | μs -----------------|--------------------|----------|------:|-------: -e_error_code | OUTCOME_TRY | Disabled | 2% | 2129 -e_error_code | OUTCOME_TRY | Enabled | 2% | 2245 -e_error_code | Handle some errors | Disabled | 2% | 2102 -e_error_code | Handle some errors | Enabled | 2% | 2157 -e_error_code | OUTCOME_TRY | Disabled | 50% | 3994 -e_error_code | OUTCOME_TRY | Enabled | 50% | 4043 -e_error_code | Handle some errors | Disabled | 50% | 2958 -e_error_code | Handle some errors | Enabled | 50% | 2994 -e_error_code | OUTCOME_TRY | Disabled | 98% | 5533 -e_error_code | OUTCOME_TRY | Enabled | 98% | 5573 -e_error_code | Handle some errors | Disabled | 98% | 3754 -e_error_code | Handle some errors | Enabled | 98% | 3824 -e_system_error | OUTCOME_TRY | Disabled | 2% | 2144 -e_system_error | OUTCOME_TRY | Enabled | 2% | 2158 -e_system_error | Handle some errors | Disabled | 2% | 2130 -e_system_error | Handle some errors | Enabled | 2% | 2187 -e_system_error | OUTCOME_TRY | Disabled | 50% | 3954 -e_system_error | OUTCOME_TRY | Enabled | 50% | 3996 -e_system_error | Handle some errors | Disabled | 50% | 3210 -e_system_error | Handle some errors | Enabled | 50% | 3369 -e_system_error | OUTCOME_TRY | Disabled | 98% | 5422 -e_system_error | OUTCOME_TRY | Enabled | 98% | 5247 -e_system_error | Handle some errors | Disabled | 98% | 4278 -e_system_error | Handle some errors | Enabled | 98% | 4126 -e_heavy_payload | OUTCOME_TRY | Disabled | 2% | 25931 -e_heavy_payload | OUTCOME_TRY | Enabled | 2% | 26290 -e_heavy_payload | Handle some errors | Disabled | 2% | 26060 -e_heavy_payload | Handle some errors | Enabled | 2% | 26176 -e_heavy_payload | OUTCOME_TRY | Disabled | 50% | 43802 -e_heavy_payload | OUTCOME_TRY | Enabled | 50% | 42883 -e_heavy_payload | Handle some errors | Disabled | 50% | 42538 -e_heavy_payload | Handle some errors | Enabled | 50% | 42590 -e_heavy_payload | OUTCOME_TRY | Disabled | 98% | 58263 -e_heavy_payload | OUTCOME_TRY | Enabled | 98% | 57450 -e_heavy_payload | Handle some errors | Disabled | 98% | 58248 -e_heavy_payload | Handle some errors | Enabled | 98% | 57814 - -#### 10000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096 (gcc, LEAF): - -Error type | At each level | Inlining | Rate | μs -----------------|--------------------|----------|------:|-------: -e_error_code | LEAF_AUTO | Disabled | 2% | 1973 -e_error_code | LEAF_AUTO | Enabled | 2% | 76 -e_error_code | try_handle_some | Disabled | 2% | 2381 -e_error_code | try_handle_some | Enabled | 2% | 356 -e_error_code | LEAF_AUTO | Disabled | 50% | 1741 -e_error_code | LEAF_AUTO | Enabled | 50% | 235 -e_error_code | try_handle_some | Disabled | 50% | 2681 -e_error_code | try_handle_some | Enabled | 50% | 555 -e_error_code | LEAF_AUTO | Disabled | 98% | 1296 -e_error_code | LEAF_AUTO | Enabled | 98% | 234 -e_error_code | try_handle_some | Disabled | 98% | 2389 -e_error_code | try_handle_some | Enabled | 98% | 599 -e_system_error | LEAF_AUTO | Disabled | 2% | 1796 -e_system_error | LEAF_AUTO | Enabled | 2% | 76 -e_system_error | try_handle_some | Disabled | 2% | 2511 -e_system_error | try_handle_some | Enabled | 2% | 369 -e_system_error | LEAF_AUTO | Disabled | 50% | 1643 -e_system_error | LEAF_AUTO | Enabled | 50% | 249 -e_system_error | try_handle_some | Disabled | 50% | 2338 -e_system_error | try_handle_some | Enabled | 50% | 574 -e_system_error | LEAF_AUTO | Disabled | 98% | 1205 -e_system_error | LEAF_AUTO | Enabled | 98% | 263 -e_system_error | try_handle_some | Disabled | 98% | 2316 -e_system_error | try_handle_some | Enabled | 98% | 672 -e_heavy_payload | LEAF_AUTO | Disabled | 2% | 1907 -e_heavy_payload | LEAF_AUTO | Enabled | 2% | 91 -e_heavy_payload | try_handle_some | Disabled | 2% | 2069 -e_heavy_payload | try_handle_some | Enabled | 2% | 392 -e_heavy_payload | LEAF_AUTO | Disabled | 50% | 1982 -e_heavy_payload | LEAF_AUTO | Enabled | 50% | 831 -e_heavy_payload | try_handle_some | Disabled | 50% | 2579 -e_heavy_payload | try_handle_some | Enabled | 50% | 1139 -e_heavy_payload | LEAF_AUTO | Disabled | 98% | 2032 -e_heavy_payload | LEAF_AUTO | Enabled | 98% | 1579 -e_heavy_payload | try_handle_some | Disabled | 98% | 2806 -e_heavy_payload | try_handle_some | Enabled | 98% | 1682 - -#### 10000 iterations, call depth 32, sizeof(e_heavy_payload) = 4096 (gcc, Outcome): - -Error type | At each level | Inlining | Rate | μs -----------------|--------------------|----------|------:|-------: -e_error_code | OUTCOME_TRY | Disabled | 2% | 2261 -e_error_code | OUTCOME_TRY | Enabled | 2% | 1699 -e_error_code | Handle some errors | Disabled | 2% | 2333 -e_error_code | Handle some errors | Enabled | 2% | 1738 -e_error_code | OUTCOME_TRY | Disabled | 50% | 3754 -e_error_code | OUTCOME_TRY | Enabled | 50% | 2466 -e_error_code | Handle some errors | Disabled | 50% | 3004 -e_error_code | Handle some errors | Enabled | 50% | 2303 -e_error_code | OUTCOME_TRY | Disabled | 98% | 4745 -e_error_code | OUTCOME_TRY | Enabled | 98% | 2897 -e_error_code | Handle some errors | Disabled | 98% | 3482 -e_error_code | Handle some errors | Enabled | 98% | 2663 -e_system_error | OUTCOME_TRY | Disabled | 2% | 2591 -e_system_error | OUTCOME_TRY | Enabled | 2% | 2053 -e_system_error | Handle some errors | Disabled | 2% | 2591 -e_system_error | Handle some errors | Enabled | 2% | 1969 -e_system_error | OUTCOME_TRY | Disabled | 50% | 4793 -e_system_error | OUTCOME_TRY | Enabled | 50% | 3375 -e_system_error | Handle some errors | Disabled | 50% | 3662 -e_system_error | Handle some errors | Enabled | 50% | 2883 -e_system_error | OUTCOME_TRY | Disabled | 98% | 6653 -e_system_error | OUTCOME_TRY | Enabled | 98% | 4602 -e_system_error | Handle some errors | Disabled | 98% | 4550 -e_system_error | Handle some errors | Enabled | 98% | 3711 -e_heavy_payload | OUTCOME_TRY | Disabled | 2% | 31509 -e_heavy_payload | OUTCOME_TRY | Enabled | 2% | 17713 -e_heavy_payload | Handle some errors | Disabled | 2% | 28230 -e_heavy_payload | Handle some errors | Enabled | 2% | 17252 -e_heavy_payload | OUTCOME_TRY | Disabled | 50% | 44864 -e_heavy_payload | OUTCOME_TRY | Enabled | 50% | 25929 -e_heavy_payload | Handle some errors | Disabled | 50% | 43575 -e_heavy_payload | Handle some errors | Enabled | 50% | 26238 -e_heavy_payload | OUTCOME_TRY | Disabled | 98% | 63491 -e_heavy_payload | OUTCOME_TRY | Enabled | 98% | 35355 -e_heavy_payload | Handle some errors | Disabled | 98% | 61875 -e_heavy_payload | Handle some errors | Enabled | 98% | 34624 +> `leaf::result`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 729 | 562 +> e_system_error | 732 | 1134 +> e_heavy_payload | 901 | 5203 +> +> `tl::expected`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 830 | 459 +> e_system_error | 791 | 3957 +> e_heavy_payload | 1224 | 15512 +> +> `outcome::result`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 661 | 589 +> e_system_error | 743 | 1786 +> e_heavy_payload | 10453 | 15487 +> +> `outcome::outcome`: +> +> Error type | 2% (?s) | 98% (?s) +> ----------------|---------:|--------: +> e_int | 781 | 1811 +> e_system_error | 876 | 2493 +> e_heavy_payload | 10055 | 22527 ## Charts -The charts below are generated from the results from the previous section, converted from elapsed time in microseconds to millions of calls per second (so, bigger is better). Outcome performance is plotted in grey, LEAF performance is plotted in purple for clang, orange for gcc. +The charts below are generated from the results from the previous section, converted from elapsed time in microseconds to millions of calls per second. -| Error rate | `e_error_code` (clang) | `e_system_error` (clang) | `e_heavy_payload` (clang) | -|:---:|:---:|:---:|:---:| -| 2% | ![](clang_e_error_code_2.png) | ![](clang_e_system_error_2.png) | ![](clang_e_heavy_payload_2.png) | -| 50% | ![](clang_e_error_code_50.png) | ![](clang_e_system_error_50.png) | ![](clang_e_heavy_payload_50.png) | -| 98% | ![](clang_e_error_code_98.png) | ![](clang_e_system_error_98.png) | ![](clang_e_heavy_payload_98.png) | +### gcc: -| Error rate | `e_error_code` (gcc) | `e_system_error` (gcc) | `e_heavy_payload` (gcc) | -|:---:|:---:|:---:|:---:| -| 2% | ![](gcc_e_error_code_2.png) | ![](gcc_e_system_error_2.png) | ![](gcc_e_heavy_payload_2.png) | -| 50% | ![](gcc_e_error_code_50.png) | ![](gcc_e_system_error_50.png) | ![](gcc_e_heavy_payload_50.png) | -| 98% | ![](gcc_e_error_code_98.png) | ![](gcc_e_system_error_98.png) | ![](gcc_e_heavy_payload_98.png) | +> ![](gcc_e_int.png) +> +> ![](gcc_e_system_error.png) +> +> ![](gcc_e_heavy_payload.png) + +### clang: + +> ![](clang_e_int.png) +> +> ![](clang_e_system_error.png) +> +> ![](clang_e_heavy_payload.png) + +## Thanks + +Thanks for the valuable feedback: Peter Dimov, Glen Fernandes, Sorin Fetche, Niall Douglas, Ben Craig, Vinnie Falco, Jason Dictos diff --git a/benchmark/clang_e_error_code_2.png b/benchmark/clang_e_error_code_2.png deleted file mode 100644 index 338e901..0000000 Binary files a/benchmark/clang_e_error_code_2.png and /dev/null differ diff --git a/benchmark/clang_e_error_code_50.png b/benchmark/clang_e_error_code_50.png deleted file mode 100644 index 74c0025..0000000 Binary files a/benchmark/clang_e_error_code_50.png and /dev/null differ diff --git a/benchmark/clang_e_error_code_98.png b/benchmark/clang_e_error_code_98.png deleted file mode 100644 index 30f8aaa..0000000 Binary files a/benchmark/clang_e_error_code_98.png and /dev/null differ diff --git a/benchmark/clang_e_heavy_payload.png b/benchmark/clang_e_heavy_payload.png new file mode 100644 index 0000000..d2022ff Binary files /dev/null and b/benchmark/clang_e_heavy_payload.png differ diff --git a/benchmark/clang_e_heavy_payload_2.png b/benchmark/clang_e_heavy_payload_2.png deleted file mode 100644 index c6ce913..0000000 Binary files a/benchmark/clang_e_heavy_payload_2.png and /dev/null differ diff --git a/benchmark/clang_e_heavy_payload_50.png b/benchmark/clang_e_heavy_payload_50.png deleted file mode 100644 index 339b115..0000000 Binary files a/benchmark/clang_e_heavy_payload_50.png and /dev/null differ diff --git a/benchmark/clang_e_heavy_payload_98.png b/benchmark/clang_e_heavy_payload_98.png deleted file mode 100644 index eab2e1b..0000000 Binary files a/benchmark/clang_e_heavy_payload_98.png and /dev/null differ diff --git a/benchmark/clang_e_int.png b/benchmark/clang_e_int.png new file mode 100644 index 0000000..71dd857 Binary files /dev/null and b/benchmark/clang_e_int.png differ diff --git a/benchmark/clang_e_system_error.png b/benchmark/clang_e_system_error.png new file mode 100644 index 0000000..fa7598e Binary files /dev/null and b/benchmark/clang_e_system_error.png differ diff --git a/benchmark/clang_e_system_error_2.png b/benchmark/clang_e_system_error_2.png deleted file mode 100644 index 209248d..0000000 Binary files a/benchmark/clang_e_system_error_2.png and /dev/null differ diff --git a/benchmark/clang_e_system_error_50.png b/benchmark/clang_e_system_error_50.png deleted file mode 100644 index 0a29ae5..0000000 Binary files a/benchmark/clang_e_system_error_50.png and /dev/null differ diff --git a/benchmark/clang_e_system_error_98.png b/benchmark/clang_e_system_error_98.png deleted file mode 100644 index 68257e4..0000000 Binary files a/benchmark/clang_e_system_error_98.png and /dev/null differ diff --git a/benchmark/deep_stack_leaf.cpp b/benchmark/deep_stack_leaf.cpp index 9f148e1..b78d1fd 100644 --- a/benchmark/deep_stack_leaf.cpp +++ b/benchmark/deep_stack_leaf.cpp @@ -17,12 +17,24 @@ # error Please disable diagnostics. #endif +#ifdef _MSC_VER +# define NOINLINE __declspec(noinline) +# define ALWAYS_INLINE __forceinline +#else +# define NOINLINE __attribute__((noinline)) +# define ALWAYS_INLINE __attribute__((always_inline)) inline +#endif + #include #include +#include #include #include #include #include +#include +#include +#include namespace boost { @@ -37,63 +49,47 @@ namespace boost namespace leaf = boost::leaf; -#ifdef _MSC_VER -# define NOINLINE __declspec(noinline) -#else -# define NOINLINE __attribute__((noinline)) -#endif +#define USING_RESULT_TYPE "leaf::result" ////////////////////////////////////// -enum class e_error_code +struct e_int { - ec1=1, - ec2 + int value; }; -namespace boost { namespace leaf { - - template <> struct is_e_type: std::true_type { }; - -} } - struct e_system_error { - e_error_code value; + e_int value; std::string what; }; -// Note: in LEAF, handling of error objects is O(1) no matter how many stack frames. struct e_heavy_payload { - char value[4096]; - - e_heavy_payload() noexcept - { - std::memset(value, std::rand(), sizeof(value)); - } + std::array value; }; template E make_error() noexcept; template <> -inline e_error_code make_error() noexcept +NOINLINE e_int make_error() noexcept { - return (std::rand()%2) ? e_error_code::ec1 : e_error_code::ec2; + return { std::rand() }; } template <> -inline e_system_error make_error() noexcept +NOINLINE e_system_error make_error() noexcept { - e_error_code ec = make_error(); - return { ec, std::string(ec==e_error_code::ec1 ? "ec1" : "ec2") }; + return { make_error(), std::string(std::rand()%32, ' ') }; } template <> -inline e_heavy_payload make_error() noexcept +NOINLINE e_heavy_payload make_error() noexcept { - return e_heavy_payload(); + e_heavy_payload e; + std::fill(e.value.begin(), e.value.end(), std::rand()); + return e; } inline bool should_fail( int failure_rate ) noexcept @@ -103,151 +99,88 @@ inline bool should_fail( int failure_rate ) noexcept return (std::rand()%100) < failure_rate; } +inline int handle_error(e_int e ) noexcept +{ + return e.value; +} + +inline int handle_error( e_system_error const & e ) noexcept +{ + return handle_error(e.value) + e.what.size(); +} + +inline int handle_error( e_heavy_payload const & e ) noexcept +{ + return std::accumulate(e.value.begin(), e.value.end(), 0); +} + ////////////////////////////////////// -template -struct benchmark_check_error_noinline +// This is used to change the "success" type at each level. +// Generally, functions return values of different types. +template +struct select_result_type; + +template +struct select_result_type { - NOINLINE static leaf::result f( int failure_rate ) noexcept + using type = leaf::result; // Does not depend on E +}; + +template +struct select_result_type +{ + using type = leaf::result; // Does not depend on E +}; + +template +using select_result_t = typename select_result_type::type; + +////////////////////////////////////// + +template +struct benchmark +{ + using e_type = E; + + NOINLINE static select_result_t f( int failure_rate ) noexcept { - LEAF_AUTO(x, (benchmark_check_error_noinline::f(failure_rate))); + LEAF_AUTO(x, (benchmark::f(failure_rate))); return x+1; } }; -template -struct benchmark_check_error_noinline<1, T, E> +template +struct benchmark<1, E> { - NOINLINE static leaf::result f( int failure_rate ) noexcept + using e_type = E; + + NOINLINE static select_result_t<1, E> f( int failure_rate ) noexcept { if( should_fail(failure_rate) ) return leaf::new_error(make_error()); else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_check_error_inline -{ - static leaf::result f( int failure_rate ) noexcept - { - LEAF_AUTO(x, (benchmark_check_error_inline::f(failure_rate))); - return x+1; - } -}; - -template -struct benchmark_check_error_inline<1, T, E> -{ - static leaf::result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return leaf::new_error(make_error()); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_handle_some_noinline -{ - NOINLINE static leaf::result f( int failure_rate ) noexcept - { - if( N%4 ) - { - LEAF_AUTO(x, (benchmark_handle_some_noinline::f(failure_rate))); - return x+1; - } - else - return leaf::try_handle_some( - [=]() -> leaf::result - { - LEAF_AUTO(x, (benchmark_handle_some_noinline::f(failure_rate))); - return x+1; - }, - []( leaf::match ) - { - return 2; - } ); - } -}; - -template -struct benchmark_handle_some_noinline<1, T, E> -{ - NOINLINE static leaf::result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return leaf::new_error(make_error()); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_handle_some_inline -{ - static leaf::result f( int failure_rate ) noexcept - { - if( N%4 ) - { - LEAF_AUTO(x, (benchmark_handle_some_inline::f(failure_rate))); - return x+1; - } - else - return leaf::try_handle_some( - [=]() -> leaf::result - { - LEAF_AUTO(x, (benchmark_handle_some_inline::f(failure_rate))); - return x+1; - }, - []( leaf::match ) - { - return 2; - } ); - } -}; - -template -struct benchmark_handle_some_inline<1, T, E> -{ - static leaf::result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return leaf::new_error(make_error()); - else - return T{ }; + return std::rand(); } }; ////////////////////////////////////// template -int runner( int failure_rate ) noexcept +NOINLINE int runner( int failure_rate ) noexcept { return leaf::try_handle_all( [=] { return Benchmark::f(failure_rate); }, - []( e_error_code const & ) + []( typename Benchmark::e_type const & e ) { - return -1; - }, - []( e_heavy_payload const & ) - { - return -2; + return handle_error(e); }, [] { - return -3; + return -1; } ); } @@ -263,7 +196,7 @@ std::fstream append_csv() else { std::fstream fs("benchmark.csv", std::fstream::out | std::fstream::app); - fs << ",\"Check, noinline\",\"Check, inline\",\"Handle some, noinline\", \"Handle some, inline\",\"Error rate\"\n"; + fs << "\"Result Type\",2%,98%\n"; return fs; } } @@ -277,7 +210,7 @@ int print_elapsed_time( int iteration_count, F && f ) val += std::forward(f)(); auto stop = std::chrono::steady_clock::now(); int elapsed = std::chrono::duration_cast(stop-start).count(); - std::cout << std::right << std::setw(8) << elapsed; + std::cout << std::right << std::setw(9) << elapsed; append_csv() << ',' << elapsed; return val; } @@ -287,43 +220,34 @@ int print_elapsed_time( int iteration_count, F && f ) template int benchmark_type( char const * type_name, int iteration_count ) { - int const test_rates[ ] = { 2, 50, 98 }; int x=0; - std::cout << "----------------|--------------------|----------|-------|--------"; - for( auto fr : test_rates ) - { - append_csv() << "LEAF"; - std::cout << '\n' << std::left << std::setw(16) << type_name << "| LEAF_AUTO | Disabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| LEAF_AUTO | Enabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| try_handle_some | Disabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| try_handle_some | Enabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - append_csv() << ',' << fr << '\n'; - }; - std::cout << '\n'; + append_csv() << "\"" USING_RESULT_TYPE "\""; + std::cout << '\n' << std::left << std::setw(16) << type_name << '|'; + std::srand(0); + x += print_elapsed_time( iteration_count, [] { return runner>(2); } ); + std::cout << " |"; + std::srand(0); + x += print_elapsed_time( iteration_count, [] { return runner>(98); } ); + append_csv() << '\n'; return x; } ////////////////////////////////////// -int main( int argc, char const * argv[] ) +int main() { - int const depth = 32; + int const depth = 10; int const iteration_count = 10000; std::cout << iteration_count << " iterations, call depth " << depth << ", sizeof(e_heavy_payload) = " << sizeof(e_heavy_payload) << "\n" - "LEAF\n" - " | | Function | Error | Elapsed\n" - "Error type | At each level | inlining | rate | (μs)\n"; - return - benchmark_type("e_error_code", iteration_count) + - benchmark_type("e_system_error", iteration_count) + - benchmark_type("e_heavy_payload", iteration_count); + USING_RESULT_TYPE "\n" + "Error type | 2% (μs) | 98% (μs)\n" + "----------------|----------|---------"; + int r = 0; + r += benchmark_type("e_int", iteration_count); + r += benchmark_type("e_system_error", iteration_count); + r += benchmark_type("e_heavy_payload", iteration_count); + std::cout << '\n'; + // std::cout << std::rand() << '\n'; + return r; } diff --git a/benchmark/deep_stack_other.cpp b/benchmark/deep_stack_other.cpp new file mode 100644 index 0000000..1a6e6a3 --- /dev/null +++ b/benchmark/deep_stack_other.cpp @@ -0,0 +1,281 @@ +// Copyright (c) 2018-2019 Emil Dotchevski and Reverge Studios, Inc. + +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) + +// See benchmark.md + +#ifndef BENCHMARK_WHAT +# define BENCHMARK_WHAT 0 +#endif + +#if BENCHMARK_WHAT == 0 + +# ifndef TL_EXPECTED_HPP +# include "tl/expected.hpp" +# endif +# define ERROR(e) tl::unexpected(e) +# define BENCHMARK_TRY(v,r)\ + auto && _r_##v = r;\ + if( !_r_##v )\ + return ERROR(_r_##v.error());\ + auto && v = _r_##v.value() + +#else + +# include +# include +# define ERROR(e) e +# define BENCHMARK_TRY BOOST_OUTCOME_TRY +# ifndef BOOST_NO_EXCEPTIONS +# error Please disable exception handling. +# endif + +#endif + +#ifdef _MSC_VER +# define NOINLINE __declspec(noinline) +# define ALWAYS_INLINE __forceinline +#else +# define NOINLINE __attribute__((noinline)) +# define ALWAYS_INLINE __attribute__((always_inline)) inline +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace boost +{ + void throw_exception( std::exception const & e ) + { + std::cerr << "Terminating due to a C++ exception under BOOST_NO_EXCEPTIONS: " << e.what(); + std::terminate(); + } +} + +////////////////////////////////////// + +#if BENCHMARK_WHAT == 0 // tl::expected + +# define USING_RESULT_TYPE "tl::expected" + + template + using result = tl::expected; + +#elif BENCHMARK_WHAT == 1 // outcome::result + +# define USING_RESULT_TYPE "outcome::result" + + template + using result = boost::outcome_v2::std_result; + +#elif BENCHMARK_WHAT == 2 // outcome::outcome + +# define USING_RESULT_TYPE "outcome::outcome" + + template + using result = boost::outcome_v2::std_outcome; + +#else +# error Benchmark what? +#endif + +////////////////////////////////////// + +struct e_int +{ + int value; +}; + +struct e_system_error +{ + e_int value; + std::string what; +}; + +struct e_heavy_payload +{ + std::array value; +}; + +template +E make_error() noexcept; + +template <> +NOINLINE e_int make_error() noexcept +{ + return { std::rand() }; +} + +template <> +NOINLINE e_system_error make_error() noexcept +{ + return { make_error(), std::string(std::rand()%32, ' ') }; +} + +template <> +NOINLINE e_heavy_payload make_error() noexcept +{ + e_heavy_payload e; + std::fill(e.value.begin(), e.value.end(), std::rand()); + return e; +} + +inline bool should_fail( int failure_rate ) noexcept +{ + assert(failure_rate>=0); + assert(failure_rate<=100); + return (std::rand()%100) < failure_rate; +} + +inline int handle_error(e_int e ) noexcept +{ + return e.value; +} + +inline int handle_error( e_system_error const & e ) noexcept +{ + return handle_error(e.value) + e.what.size(); +} + +inline int handle_error( e_heavy_payload const & e ) noexcept +{ + return std::accumulate(e.value.begin(), e.value.end(), 0); +} + +////////////////////////////////////// + +// This is used to change the "success" type at each level. +// Generally, functions return values of different types. +template +struct select_result_type; + +template +struct select_result_type +{ + using type = result; +}; + +template +struct select_result_type +{ + using type = result; +}; + +template +using select_result_t = typename select_result_type::type; + +////////////////////////////////////// + +template +struct benchmark +{ + using e_type = E; + + NOINLINE static select_result_t f( int failure_rate ) noexcept + { + BENCHMARK_TRY(x, (benchmark::f(failure_rate))); + return x+1; + } +}; + +template +struct benchmark<1, E> +{ + using e_type = E; + + NOINLINE static select_result_t<1, E> f( int failure_rate ) noexcept + { + if( should_fail(failure_rate) ) + return ERROR(make_error()); + else + return std::rand(); + } +}; + +////////////////////////////////////// + +template +NOINLINE int runner( int failure_rate ) noexcept +{ + if( auto r = Benchmark::f(failure_rate) ) + return r.value(); + else + return handle_error(r.error()); +} + +////////////////////////////////////// + +std::fstream append_csv() +{ + if( FILE * f = fopen("benchmark.csv","rb") ) + { + fclose(f); + return std::fstream("benchmark.csv", std::fstream::out | std::fstream::app); + } + else + { + std::fstream fs("benchmark.csv", std::fstream::out | std::fstream::app); + fs << "\"Result Type\",2%,98%\n"; + return fs; + } +} + +template +int print_elapsed_time( int iteration_count, F && f ) +{ + auto start = std::chrono::steady_clock::now(); + int val = 0; + for( int i = 0; i!=iteration_count; ++i ) + val += std::forward(f)(); + auto stop = std::chrono::steady_clock::now(); + int elapsed = std::chrono::duration_cast(stop-start).count(); + std::cout << std::right << std::setw(9) << elapsed; + append_csv() << ',' << elapsed; + return val; +} + +////////////////////////////////////// + +template +int benchmark_type( char const * type_name, int iteration_count ) +{ + int x=0; + append_csv() << "\"" USING_RESULT_TYPE "\""; + std::cout << '\n' << std::left << std::setw(16) << type_name << '|'; + std::srand(0); + x += print_elapsed_time( iteration_count, [] { return runner>(2); } ); + std::cout << " |"; + std::srand(0); + x += print_elapsed_time( iteration_count, [] { return runner>(98); } ); + append_csv() << '\n'; + return x; +} + +////////////////////////////////////// + +int main() +{ + int const depth = 10; + int const iteration_count = 10000; + std::cout << + iteration_count << " iterations, call depth " << depth << ", sizeof(e_heavy_payload) = " << sizeof(e_heavy_payload) << "\n" + USING_RESULT_TYPE "\n" + "Error type | 2% (μs) | 98% (μs)\n" + "----------------|----------|---------"; + int r = 0; + r += benchmark_type("e_int", iteration_count); + r += benchmark_type("e_system_error", iteration_count); + r += benchmark_type("e_heavy_payload", iteration_count); + std::cout << '\n'; + // std::cout << std::rand() << '\n'; + return r; +} diff --git a/benchmark/deep_stack_outcome.cpp b/benchmark/deep_stack_outcome.cpp deleted file mode 100644 index 9d499b0..0000000 --- a/benchmark/deep_stack_outcome.cpp +++ /dev/null @@ -1,312 +0,0 @@ -// Copyright (c) 2018-2019 Emil Dotchevski and Reverge Studios, Inc. - -// Distributed under the Boost Software License, Version 1.0. (See accompanying -// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) - -// See benchmark.md - -#include -#include - -#ifndef BOOST_NO_EXCEPTIONS -# error Please disable exception handling. -#endif - -#include -#include -#include -#include -#include -#include - -namespace boost -{ - void throw_exception( std::exception const & e ) - { - std::cerr << "Terminating due to a C++ exception under BOOST_NO_EXCEPTIONS: " << e.what(); - std::terminate(); - } -} - -////////////////////////////////////// - -namespace outcome = boost::outcome_v2; - -template -using result = outcome::std_outcome; - -#ifdef _MSC_VER -# define NOINLINE __declspec(noinline) -#else -# define NOINLINE __attribute__((noinline)) -#endif - -////////////////////////////////////// - -enum class e_error_code -{ - ec1=1, - ec2 -}; - -struct e_system_error -{ - e_error_code value; - std::string what; -}; - -struct e_heavy_payload -{ - char value[4096]; - - e_heavy_payload() noexcept - { - std::memset(value, std::rand(), sizeof(value)); - } -}; - -template -E make_error() noexcept; - -template <> -inline e_error_code make_error() noexcept -{ - return (std::rand()%2) ? e_error_code::ec1 : e_error_code::ec2; -} - -template <> -inline e_system_error make_error() noexcept -{ - e_error_code ec = make_error(); - return { ec, std::string(ec==e_error_code::ec1 ? "ec1" : "ec2") }; -} - -template <> -inline e_heavy_payload make_error() noexcept -{ - return e_heavy_payload(); -} - -inline bool should_fail( int failure_rate ) noexcept -{ - assert(failure_rate>=0); - assert(failure_rate<=100); - return (std::rand()%100) < failure_rate; -} - -bool check_handle_some_value( e_error_code e ) noexcept -{ - return e==e_error_code::ec2; -} - -bool check_handle_some_value( e_system_error const & e ) noexcept -{ - return check_handle_some_value(e.value); -} - -bool check_handle_some_value( e_heavy_payload const & ) noexcept -{ - return false; -} - -////////////////////////////////////// - -template -struct benchmark_check_error_noinline -{ - NOINLINE static result f( int failure_rate ) noexcept - { - BOOST_OUTCOME_TRY(x, (benchmark_check_error_noinline::f(failure_rate))); - return x+1; - } -}; - -template -struct benchmark_check_error_noinline<1, T, E> -{ - static result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return make_error(); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_check_error_inline -{ - static result f( int failure_rate ) noexcept - { - BOOST_OUTCOME_TRY(x, (benchmark_check_error_inline::f(failure_rate))); - return x+1; - } -}; - -template -struct benchmark_check_error_inline<1, T, E> -{ - static result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return make_error(); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_handle_some_noinline -{ - NOINLINE static result f( int failure_rate ) noexcept - { - if( N%4 ) - { - BOOST_OUTCOME_TRY(x, (benchmark_handle_some_noinline::f(failure_rate))); - return x+1; - } - else if( auto r = benchmark_handle_some_noinline::f(failure_rate) ) - return r.value()+1; - else if( check_handle_some_value(r.error()) ) - return 1; - else - return r.as_failure(); - } -}; - -template -struct benchmark_handle_some_noinline<1, T, E> -{ - static result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return make_error(); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -struct benchmark_handle_some_inline -{ - static result f( int failure_rate ) noexcept - { - if( N%4 ) - { - BOOST_OUTCOME_TRY(x, (benchmark_handle_some_inline::f(failure_rate))); - return x+1; - } - else if( auto r = benchmark_handle_some_inline::f(failure_rate) ) - return r.value()+1; - else if( check_handle_some_value(r.error()) ) - return 1; - else - return r.as_failure(); - } -}; - -template -struct benchmark_handle_some_inline<1, T, E> -{ - static result f( int failure_rate ) noexcept - { - if( should_fail(failure_rate) ) - return make_error(); - else - return T{ }; - } -}; - -////////////////////////////////////// - -template -int runner( int failure_rate ) noexcept -{ - if( auto r = Benchmark::f(failure_rate) ) - return r.value(); - else - return -1; -} - -////////////////////////////////////// - -std::fstream append_csv() -{ - if( FILE * f = fopen("benchmark.csv","rb") ) - { - fclose(f); - return std::fstream("benchmark.csv", std::fstream::out | std::fstream::app); - } - else - { - std::fstream fs("benchmark.csv", std::fstream::out | std::fstream::app); - fs << ",\"Check, noinline\",\"Check, inline\",\"Handle some, noinline\", \"Handle some, inline\",\"Error rate\"\n"; - return fs; - } -} - -template -int print_elapsed_time( int iteration_count, F && f ) -{ - auto start = std::chrono::system_clock::now(); - int val = 0; - for( int i = 0; i!=iteration_count; ++i ) - val += std::forward(f)(); - auto stop = std::chrono::system_clock::now(); - int elapsed = std::chrono::duration_cast(stop-start).count(); - std::cout << std::right << std::setw(8) << elapsed; - append_csv() << ',' << elapsed; - return val; -} - -////////////////////////////////////// - -template -int benchmark_type( char const * type_name, int iteration_count ) -{ - int const test_rates[ ] = { 2, 50, 98 }; - int x=0; - std::cout << "----------------|--------------------|----------|-------|--------"; - for( auto fr : test_rates ) - { - append_csv() << "Outcome"; - std::cout << '\n' << std::left << std::setw(16) << type_name << "| OUTCOME_TRY | Disabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| OUTCOME_TRY | Enabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| Handle some errors | Disabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - std::cout << '\n' << std::left << std::setw(16) << type_name << "| Handle some errors | Enabled | " << std::right << std::setw(4) << fr << "% |"; - std::srand(0); - x += print_elapsed_time( iteration_count, [=] { return runner>(fr); } ); - append_csv() << ',' << fr << '\n'; - }; - std::cout << '\n'; - return x; -} - -////////////////////////////////////// - -int main() -{ - int const depth = 32; - int const iteration_count = 10000; - std::cout << - iteration_count << " iterations, call depth " << depth << ", sizeof(e_heavy_payload) = " << sizeof(e_heavy_payload) << "\n" - "Outcome\n" - " | | Function | Error | Elapsed\n" - "Error type | At each level | inlining | rate | (μs)\n"; - return - benchmark_type("e_error_code", iteration_count) + - benchmark_type("e_system_error", iteration_count) + - benchmark_type("e_heavy_payload", iteration_count); -} diff --git a/benchmark/gcc_e_error_code_2.png b/benchmark/gcc_e_error_code_2.png deleted file mode 100644 index da1d0e9..0000000 Binary files a/benchmark/gcc_e_error_code_2.png and /dev/null differ diff --git a/benchmark/gcc_e_error_code_50.png b/benchmark/gcc_e_error_code_50.png deleted file mode 100644 index 49bd0ca..0000000 Binary files a/benchmark/gcc_e_error_code_50.png and /dev/null differ diff --git a/benchmark/gcc_e_error_code_98.png b/benchmark/gcc_e_error_code_98.png deleted file mode 100644 index 6b56173..0000000 Binary files a/benchmark/gcc_e_error_code_98.png and /dev/null differ diff --git a/benchmark/gcc_e_heavy_payload.png b/benchmark/gcc_e_heavy_payload.png new file mode 100644 index 0000000..59cb68c Binary files /dev/null and b/benchmark/gcc_e_heavy_payload.png differ diff --git a/benchmark/gcc_e_heavy_payload_2.png b/benchmark/gcc_e_heavy_payload_2.png deleted file mode 100644 index 4ed68cf..0000000 Binary files a/benchmark/gcc_e_heavy_payload_2.png and /dev/null differ diff --git a/benchmark/gcc_e_heavy_payload_50.png b/benchmark/gcc_e_heavy_payload_50.png deleted file mode 100644 index c97ac2a..0000000 Binary files a/benchmark/gcc_e_heavy_payload_50.png and /dev/null differ diff --git a/benchmark/gcc_e_heavy_payload_98.png b/benchmark/gcc_e_heavy_payload_98.png deleted file mode 100644 index 4068ebd..0000000 Binary files a/benchmark/gcc_e_heavy_payload_98.png and /dev/null differ diff --git a/benchmark/gcc_e_int.png b/benchmark/gcc_e_int.png new file mode 100644 index 0000000..a4e66d8 Binary files /dev/null and b/benchmark/gcc_e_int.png differ diff --git a/benchmark/gcc_e_system_error.png b/benchmark/gcc_e_system_error.png new file mode 100644 index 0000000..7f70a1a Binary files /dev/null and b/benchmark/gcc_e_system_error.png differ diff --git a/benchmark/gcc_e_system_error_2.png b/benchmark/gcc_e_system_error_2.png deleted file mode 100644 index 96268c2..0000000 Binary files a/benchmark/gcc_e_system_error_2.png and /dev/null differ diff --git a/benchmark/gcc_e_system_error_50.png b/benchmark/gcc_e_system_error_50.png deleted file mode 100644 index 7e0359c..0000000 Binary files a/benchmark/gcc_e_system_error_50.png and /dev/null differ diff --git a/benchmark/gcc_e_system_error_98.png b/benchmark/gcc_e_system_error_98.png deleted file mode 100644 index a11b438..0000000 Binary files a/benchmark/gcc_e_system_error_98.png and /dev/null differ diff --git a/meson.build b/meson.build index e888479..9e0e2ee 100644 --- a/meson.build +++ b/meson.build @@ -3,7 +3,7 @@ # Distributed under the Boost Software License, Version 1.0. (See accompanying # file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) -project('leaf', 'cpp', default_options : ['cpp_std=c++11','b_pch=false'], license : 'boost') +project('leaf', 'cpp', default_options : ['cpp_std=c++17','b_pch=false'], license : 'boost') compiler = meson.get_compiler('cpp') compiler_id = compiler.get_id() @@ -14,12 +14,14 @@ if not meson.is_subproject() '-Wno-non-virtual-dtor', '-Wno-dangling-else', '-Wno-delete-non-virtual-dtor', + '-std=c++17', language:'cpp' ) elif compiler_id=='gcc' add_global_arguments( '-Wno-non-virtual-dtor', '-Wno-dangling-else', '-Wno-parentheses', + '-std=c++17', language:'cpp' ) endif endif @@ -163,6 +165,8 @@ if not exceptions executable('deep_stack_leaf', 'benchmark/deep_stack_leaf.cpp', dependencies: [leaf], override_options: ['cpp_std=c++17']) endif if get_option('boost_examples') - executable('deep_stack_outcome', 'benchmark/deep_stack_outcome.cpp', dependencies: [leaf,boost_headers], override_options: ['cpp_std=c++17'] ) + executable('deep_stack_tl', 'benchmark/deep_stack_other.cpp', override_options: ['cpp_std=c++17'], cpp_args: '-DBENCHMARK_WHAT=0' ) + executable('deep_stack_result', 'benchmark/deep_stack_other.cpp', dependencies: [boost_headers], override_options: ['cpp_std=c++17'], cpp_args: '-DBENCHMARK_WHAT=1' ) + executable('deep_stack_outcome', 'benchmark/deep_stack_other.cpp', dependencies: [boost_headers], override_options: ['cpp_std=c++17'], cpp_args: '-DBENCHMARK_WHAT=2' ) endif endif