diff --git a/Makefile.am b/Makefile.am index 06075f93..313788ec 100644 --- a/Makefile.am +++ b/Makefile.am @@ -88,8 +88,8 @@ EXTRA_DIST += $(top_srcdir)/doc/DoxygenLayout.xml EXTRA_DIST += $(top_srcdir)/doc/aedis.css EXTRA_DIST += $(top_srcdir)/doc/htmlfooter.html EXTRA_DIST += $(top_srcdir)/doc/htmlheader.html -EXTRA_DIST += $(top_srcdir)/benchmarks/cpp/libuv/echo_server_direct.c -EXTRA_DIST += $(top_srcdir)/benchmarks/cpp/libuv/README.md +EXTRA_DIST += $(top_srcdir)/benchmarks/c/libuv/echo_server_direct.c +EXTRA_DIST += $(top_srcdir)/benchmarks/c/libuv/README.md EXTRA_DIST += $(top_srcdir)/benchmarks/go/echo_server_direct.go EXTRA_DIST += $(top_srcdir)/benchmarks/nodejs/echo_server_direct.js EXTRA_DIST += $(top_srcdir)/benchmarks/nodejs/echo_server_over_redis.js diff --git a/benchmarks/benchmarks.md b/benchmarks/benchmarks.md index 1c6d0093..6e456dd6 100644 --- a/benchmarks/benchmarks.md +++ b/benchmarks/benchmarks.md @@ -1,67 +1,84 @@ # TCP echo server performance. -The main motivations for choosing an echo server to benchmark a redis -client were +This document describes how I benchmarked a TCP echo server +implemented in different languages and frameworks. + +## Motivation + +The main motivations for choosing a TCP echo server as a benchmark +program are * Simple to implement in most languages. * Does not require expertise level get decent performance. - * Excelent to measure ability to server concurrent requests. - * It must test the ability to deal with concurrency. - * I/O bound: In echo servers CPU consumption is very low as a result - the performance is more related to how well concurrency is - implemented. + * I/O bound: Echo servers have very low CPU consumption since they + don't compute anything. It is therefore an excelent measure of + the ability of a program to server concurrent requests. + * It simulates very well a typical backend in regard to concurrency. I also imposed some constraints on the implementations - * Favor simple implementations that use standard idioms of their - specific language or framework. - * Avoid of optimization that makes it too complex e.g. connection - pool or that requires expert level. + * It should not require me to write too much code, in other words, + it must be simple. + * Favor the use standard idioms. + * Avoid optimizations that requires expert level or makes the + code too complex e.g. connection and thread pool. ## Without Redis -Before we can compare the Redis clients implementation we must have a -rough view about how different libraries compare in terms of -performance. +First I tested a pure TCP echo server, i.e. that sends the messages +directly to the client without interacting with Redis. This is the +result -Remarks: +### Remarks: - * I was not expecting Asio to perform so much better than the alternatives that Tokio and libuv. - * I did expect nodejs to come a little behind given it is - implemented in javascript. Otherwise I expected it to have - similar performance to libuv since it is the framework behind - nodejs. + * I was not expecting Asio to perform so much better than the alternatives like Tokio and libuv. + * I did expect nodejs to come a little behind given it is is + javascript code. Otherwise I did expect it to have similar + performance to libuv since it is the framework behind it. + * The go performance was no surprise: decent and not some much far behind nodejs. -The code can be found at +The tests were performed on the localhost where latency is 0.07ms on +average. On higher latency networks the difference among libraries +will decrease. - * Asio: A variation of the asio example. - * Libuv: Copy and pasted of the libuv example. - * Tokio: Copy and pasted of the example provided here. - * Go: Found in the internet and make adjustmenst. +The code used in the benchmarks can be found at + + * [Asio](https://github.com/mzimbres/aedis/blob/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/cpp/asio/echo_server_direct.cpp): A variation of [this](https://github.com/chriskohlhoff/asio/blob/4915cfd8a1653c157a1480162ae5601318553eb8/asio/src/examples/cpp20/coroutines/echo_server.cpp) Asio example. + * [Libuv](): A variation of [this](https://github.com/libuv/libuv/blob/06948c6ee502862524f233af4e2c3e4ca876f5f6/docs/code/tcp-echo-server/main.c) Libuv example . + * [Tokio](https://github.com/mzimbres/aedis/tree/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/rust/echo_server_direct): This was taken from [here](https://docs.rs/tokio/latest/tokio/). + * [Nodejs](https://github.com/mzimbres/aedis/tree/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/nodejs/echo_server_direct). + * [Go](https://github.com/mzimbres/aedis/blob/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/go/echo_server_direct.go). ## With Redis -In this case the message is not echoed directly to the client but sent -to the server with the Ping command which will echo it back to the -server, only after that it is sent to the client. +This is similar to the echo server described above but the message is +echoed by Redis and the Echo server works as a proxy between the +client and the Redis server. The result can be seen below -the set is. +### Remarks -The most remarkable thing here is that the Rust Redis client comes so -far behind that it can't even be represented together with the other -benchmarks. +As the reader can see, the Libuv and the Rust test are not depicted +above, reasons are -The Reason why the Aedis and the nodejs client are so much faster than -the alternatives is that they implement pipeline. + * [redis-rs](https://github.com/redis-rs/redis-rs): This client + comes so far behind that it can't even be represented together + with the other benchmarks without making them insignificant. + I don't know for sure why it is so slow, I suppose however it is + not implementing pipelines properly. -The code I used is a variation of the list given above, that I -implemented myself. The Redis client for each language were chosen -based on their number of stars as given here. + * Libuv: I let this out because it would require too much work to + make it have a good performance. More specifically, I would have + to use hiredis and implement support for pipelines manually. + + +The code used in the benchmarks can be found at + + * [Aedis](https://github.com/mzimbres/aedis/blob/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/examples/echo_server.cpp). + * [Nodejs](https://github.com/mzimbres/aedis/tree/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/nodejs/echo_server_over_redis). + * [Go](https://github.com/mzimbres/aedis/blob/3fb018ccc6138d310ac8b73540391cdd8f2fdad6/benchmarks/go/echo_server_over_redis.go). ## Conclusion: -The main conclusion is that pipelines is fundamental, much more -fundamental than the language performance per se. +The main conclusion is that pipelines is fundamental, much more fundamental than the language performance per se. -If your spot any performance improvement in any of the example, please -open a PR and I will gladly merge it. +If your spot any performance improvement in any of the example, please open a PR and I will gladly merge it. diff --git a/benchmarks/cpp/libuv/README.md b/benchmarks/c/libuv/README.md similarity index 100% rename from benchmarks/cpp/libuv/README.md rename to benchmarks/c/libuv/README.md diff --git a/benchmarks/cpp/libuv/echo_server_direct.c b/benchmarks/c/libuv/echo_server_direct.c similarity index 100% rename from benchmarks/cpp/libuv/echo_server_direct.c rename to benchmarks/c/libuv/echo_server_direct.c