Simple framework to test HTTP servers, inspired by Simple Web Benchmark but focused on dlang frameworks and libraries.

It measures achievable RPS (requests per second) in a simple plaintext response test scenario.

Tests were gathered or modified from various places (including TechEmpower).

It uses docker container to build and host services on and can run locally or use load tester from remote host.

wrk is used as a default load generator and requests statistics collector, but hey is supported too (just use --tool switch).

Tests can be run without docker too, one just needs to have installed tested language compilers and wrk/hey workload generator (but this has been tested on linux only).

Note for io_uring tests:

• At least Linux kernel 5.7 is needed for tests to work.
• Problems with user limits on locked memory (ulimit -l) are possible too when run with regular user.
• There is some performance regression starting from Kernel 5.7.16. See also these: #189, #8.

Tests are divided to two types:

• singleCore - services are started in single core mode to measure performance without multiple threads / processes (default)
• multiCore - services are started to use all hosts CPU cores

Some of the tests are written just with a single thread usage. To be comparable with frameworks that can utilize more threads, prefork.d was added to the suite, that just forks multiple processes and pin them to separate CPUs (tests are named prefork or has (PF) in the name).

• make build - build execution container
• make shell - enter container

Note: Performance governor is set to performance with this command.

For simplicity, one can just run one of these commands (in the container shell):

make all        # runs all tests
make single     # runs tests limited to single CPU core usage
make multi      # runs tests limited to multiple CPU cores usage

Main entry point to more advanced tests is in _suite/runner.d which is a runnable CLI script.

• _suite/runner.d list - list available benchmarks
• _suite/runner.d bench - runs benchmarks
• _suite/runner.d responses - prints out sampled response from each benchmark service
• _suite/runner.d versions - prints out used language versions in Markdown table format

Use _suite/runner.d -h to print out CLI interface help.

Sample:

_suite/runner.d bench --type singleCore dlang rust # runs all dlang and rust tests

As localhost only benchmarking is discouraged (see ie https://www.mnot.net/blog/2011/05/18/http_benchmark_rules), CLI supports executing of load tester from the remote host.

Steps:

• on a host that would run servers, enter the container shell
• from that run something like _suite/runner.d bench --type singleCore -r foo@192.168.0.3 --host 192.168.0.2 dlang

Where -r or --remote specifies username and hostname used for executing load tester through ssh. --host is not in most cases necessary as CLI determines host IP from default route, but it's added for cases when it's needed anyway.

It's easier to generate ssh key and copy it's identity to the load generator host as otherwise underlying ssh command'll ask for password twice for each test (warmup and test itself).

Load tester (hey) must be installed on the load tester host.

Host that generates load should be ideally more performant (in a way that workload generator doesn't saturate the used CPUs).

Some of the top of the Techempower frameworks were added as a reference point.

Many of the tests there are using various tweaks unusable in a real life scenarios.

• no router (ie only match on path length, etc.)
• no HTTP request parser
• no Date header in the response
• prebuilt static text for response
• etc.

I've tried at least make response sizes to be of the same size for all the tests to be more fair.

These are added to determine the potential of the test environment configuration. They don't try to work as a generic HTTP servers, but just utilize the eventloop at the top speed.

Epoll and io_uring are added. Both named as raw.

New http server framework using async io (epoll, kqueue, ...).

I've wanted to add this popular library in the mix just for comparison. Currently three configurations of internal http servers are used:

• process - forked process, each serving one request
• threads - threadpool to handle connected clients
• hybrid - an experimental Linux-only hybrid implementation of forked processes, worker threads, and fibers in an event loop

They are the same in both type tests as they don't use (at the moment) some eventloop so we can compare this traditional way against the others.

See Adam's description for more.

• raw - test that tries to be as fast as possible to make a theoretical limit of the used system facility (so no parsers, routers, ... - just plain event loop)

Using new asynchronous I/O io_uring it's interesting to compare against mostly used epoll on Linux systems.

Not a library, but just an underlying polling mechanism used by most frameworks. Added to test theoretical limit of the system we measure on - same as during/raw

Library that is a basis for vibe-d framework. It generalizes event loop against epoll on linux (iocp and kqueue on windows and MacOS).

It's a microbenchmark as it has no proper http parser, router or response writer and currently only shows event loop potential of the library.

• callbacks - uses just callbacks to handle socket events
• fibers - uses fibers to emulate sync behavior on async events

Asynchronous I/O library used by archttp to show event base loop performance (micro benchmark).

• hunt-http - idiomatic use of the framework (HTTP router, parser and all)
• hunt-pico - highly customized and optimized test that uses picohttpparser and tweaked handlers for just the test purpose (prebuilt responses, no router, ...) - no wonder that it's relatively high in Techempower

Found this on code.dlang.org so I've added it to the mix too.

It has parser, router, and response writer.

Note: Currently it's disabled due to the etcimon/libasync#90

Core library from Weka.

It uses it's own low level fibers implementation.

Note: Uses some hacks to access druntime internals so it'd be able to switch fibers. But it won't compile with current release and some compilers. I've tried to patch it, but it still can have some problems.

It's not on code.dlang.org but is an interesting library that rewrites glibc syscalls and emulates them via epoll eventloop and fibers underneath.

Test uses nodejs http-parser (not that fast as pico) and doesn't use router.

New simple to use HTTP server that uses managed subprocesses as a worker pool to handle individual requests. Single dependency library, using just standard phobos features.

It falls to the same category as arsd.

There are two variants of tests, one that uses same number if worker processes as is available CPUs and one that uses pool of 16 workers.

Higher level library that uses eventcore and adds fiber backed tasks framework to handle event callbacks.

Still a microbenchmark as it only uses TCPConnection and reads request line by line and then just writes static response text.

No router or http parser used.

Finally most popular dlang web framework that has it all.

ASP.Net Core is used as a reference.

It has multiple tweaks mentioned above (simplistic router, prepared plaintext responses, ...). It does some magic when HTTP pipelining is used.

fasthttp is used as a reference.

Test uses HTTP parser, but no router.

Actix is used for comparison in two variants:

• actix-web - simple generic usage of the library
• actix-raw - more tweaked version with less generically used features and static responses

Currently test runner outputs results in a Markdown formatted table.

Column description:

• Res[B] - size of the sample response in bytes - to check responses are of the same size ideally
• Req - total number of requests load generator generated
• Err - number of responses with other than 200 OK results
• RPS - requests per second
• BPS - bytes per second
• min - minimal request time in [ms]
• 25% - 25% of requests has been completed within this time in [ms]
• 50% - 50% of requests has been completed within this time in [ms] (median)
• 75% - 75% of requests has been completed within this time in [ms]
• 99% - 99% of requests has been completed within this time in [ms]
• max - maximal request time in [ms]

• Host: AMD Ryzen 7 3700X 8-Core (16 threads), kernel 5.17.9, Fedora 36
• Workload generator: KVM VM with 14 pinned CPUs, Ubuntu 22.04 LTS, kernel 5.15.0-33
• Service runner: KVM VM with 2 pinned CPUs, Ubuntu 22.04 LTS, kernel 5.15.0-33
• Network: bridged network on the host
• wrk version: 4.1.0
• hey version: 0.1.4

Note: Only 2 cores for server and 14 for clients as otherwise wrk will saturate the CPUs, affecting the results (we need to utilize http server cores and not client workers) - should've bought a threadripper instead ;-)

• Test command: for i in 8 64 128 256; do _suite/runner.d bench --type multiCore -r tomas@192.168.122.176 --host 192.168.122.175 --tool wrk -d 30 -c $i; done

For an idea of how HTTP pipelining is handled in the frameworks, here are some results.

Command: _suite/runner.d bench --type multiCore -r tomas@192.168.122.176 --host 192.168.122.175 --tool wrk -d 30 -c 256 --pipeline 10

Note: arsd, geario and serverino have problems with this and doesn't finish

As arsd (excluding arsd/hybrid) and serverino doesn't use async IO, requests can be blocked for some time when keep-alive connections are used (on by default). With wrk it's not visible much, as it just bursts requests from any client that communicates and some of the clients doesn't receive any response.

To make it stand out more, here are some results with hey with number of clients equal to available CPU cores (so they should be completed without waiting in queue) and a multiple of that.

Command: for i in 2 16; do _suite/runner.d bench --type multiCore -r tomas@192.168.122.176 --host 192.168.122.175 --tool hey -b 2 -c $i arsd serverino actix-web; done