Benchmark of multiple HTTP client variants implemented with Linux system calls.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Smoolak e2d105116e Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
.gitignore Adding extension-less files to the .gitignore file 1 year ago
1.c Initial public release 1 year ago
2.c Removed explicit keep-alive headers in 2, 5, 8 and 12 because it's HTTP/1.1 default behavior (RFC 2616, section 8.1.2) 1 year ago
3.c Initial public release 1 year ago
4.c Initial public release 1 year ago
5.c Removed explicit keep-alive headers in 2, 5, 8 and 12 because it's HTTP/1.1 default behavior (RFC 2616, section 8.1.2) 1 year ago
6.c Initial public release 1 year ago
7.c Initial public release 1 year ago
8.c Removed explicit keep-alive headers in 2, 5, 8 and 12 because it's HTTP/1.1 default behavior (RFC 2616, section 8.1.2) 1 year ago
9.c Initial public release 1 year ago
10.c Initial public release 1 year ago
11.c Initial public release 1 year ago
12.c Removed explicit keep-alive headers in 2, 5, 8 and 12 because it's HTTP/1.1 default behavior (RFC 2616, section 8.1.2) 1 year ago
13.c Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
14.c Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
15.c Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
16.c Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
LICENSE.txt Initial public release 1 year ago
README.md Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
benchmark.sh Adding version of the keep-alive client with pipelining and updating the README and the benchmark script accordingly 1 year ago
common.h Initial public release 1 year ago
errmacro.h Initial public release 1 year ago
file_generator.c Initial public release 1 year ago

README.md

The Linux HTTP Client Benchmark

This project contains a series of variants of HTTP clients implemented with Linux system calls. The goal of this project is to provide an easy way to compare implementations against each other and to generate data that allows you to choose the best implementation for a specific task.

Context

This benchmark was developed for a distributed computing project where each computing node had to download a large number of small files from a local HTTP server. The computing task was performed in cascades. Stage 1 consisted of downloading a database of raw images (~6MB/image) and computing diverse operations on them. The result, a small binary blob, was then uploaded back to the server for reuse in stage 2. In our specific context, it was not practical to combine the images in one large file because they were extracted from a video sequence which needed to be sampled at different rates. For example, a computation task could need one image per second and another (potentially running at the same time) could need an image every 100 milliseconds. Therefore, to avoid data duplication on the server we had to keep the files separate.

That lead us to ask ourselves, “What would be the fastest and the most efficient way to download a large batch of small files?”. Our solution was to develop a benchmark containing multiple HTTP client implementations each one using a different approach to download the files. Then, we decided to strip the sections specific to our distributed computing project out of the source code and to release it under the AGPLv3 license for everyone to enjoy. :D

The implementations

Since we needed performance and fine control over the code, we have chosen to implement the benchmark in C. Instead of coming up with a complicated naming convention for the C files we just named them sequentially 1,2,3, etc. Each file contains a complete implementation meant to be compiled individually. We adopted this architecture to make it easy to download a single implementation without the need to dissect a large code base. Redundant code is in a separated common.h file and macro definitions for error reporting are in errmacro.h.

Usage

Note: See Utilities for the benchmark script.

Each client can be compiled individually as follows:

gcc -Ofast -lm -lcrypto [-lpthread] -o implementation implementation.c

Note: The -lpthread is only necessary for the multithreaded implementations.

A client can then be executed as follows:

./implementation server port path size nb_file [nb_conn]

Note: The nb_conn argument is only necessary for some of the implementations.

Execution flow of the clients

  1. Resolve the server IP address with getaddrinfo (this is done only once)
  2. Download a file named checksums containing the md5 hash of each file to be downloaded (see Utilities for the file generator)
  3. Execute the main download task and for each downloaded file, perform an integrity check with the checksum to detect potential programming errors that could lead to corrupt downloads and skewed execution times

Characteristics of the clients

File Kernel interface Multithreading Number of connections Type of connection Pipelining
1.c Blocking sockets No 1 Close N/A
2.c Blocking sockets No 1 Keep-alive No
13.c Blocking sockets No 1 Keep-alive Yes
3.c Blocking sockets Yes (1 thread per file) 1 per file Close N/A
4.c Blocking sockets Yes (N threads) N Close N/A
5.c Blocking sockets Yes (N threads) N Keep-alive No
14.c Blocking sockets Yes (N threads) N Keep-alive Yes
6.c epoll ET No 1 per file Close N/A
7.c epoll ET No N Close N/A
8.c epoll ET No N Keep-alive  No
15.c epoll ET No N Keep-alive  Yes
9.c epoll ET Yes (N threads, 1 epoll per thread) 1 per file Close N/A
10.c epoll ET Yes (N threads) 1 per file Close  N/A
11.c epoll ET Yes (N threads) N Close N/A
12.c epoll ET Yes (N threads) N Keep-alive No
16.c epoll ET Yes (N threads) N Keep-alive Yes

Details

Kernel interface

There is multiple system calls available in the Linux kernel to facilitate network connections. The socket and the connect system calls are necessary to all client connections, and thus used in each implementation. Once a socket is open and connected, we can use write (or send) and read (or recv) to send and receive data. By default sockets are blocking, which means that a thread (or process) will have to wait for the socket operations to complete before the kernel return control. A read would return control if at least one byte of data is ready to be processed and a write would return control only if all the data have been sent. Clients 1 to 5 implement such blocking sockets.

That being said, the O_NONBLOCK flag can be enabled on a file descriptor with fcntl to allow for non-blocking I/O. With non-blocking sockets, instead of blocking, read or write will return -1 and errno will be set to EAGAIN indicating that no more data is available or ready to be consumed. This feature allows a single thread (or process) to watch and manage multiple sockets at the same time. To perform this task, the Linux kernel makes available three interfaces for I/O multiplexing: select, poll and epoll. These interfaces work in a similar fashion by watching a list of file descriptors and by waking up the calling thread (or process) when some of them are ready to read, write or have encountered an error. So far, we have only implemented clients with epoll since it’s the most recent and the most advanced interface available. In epoll, the system call epoll_wait is used to wait for socket events and can work in two different modes: Level-Triggered (LT) or Edge-Triggered (ET). In the level-triggered mode, epoll_wait will keep returning control as long there is an active event (e.g. data is available). In the edge-triggered mode, the calling thread (or process) is only woke up one time when the event first occurs and then it’s the responsibility of the user to fully consume the event (e.g. read or write until EAGAIN). Clients 6 to 12 implement epoll in edge-triggered mode.

Multithreading and number of connections

Multithreading with pthreads (POSIX threads) have been implemented in 3 to 5 and 9 to 12.

  • 3 have one connection per downloaded file and one thread for each connection
  • 4 and 5 have N connections and also N threads, one per connection
  • 9 have one connection per downloaded file and N threads. In that case, each thread has its own epoll managing (number of files)/N sockets.
  • 10 have one connection per downloaded file and N threads that consume the events of a single global epoll
  • 11 and 12 have N connections and N threads that consume the events of a single global epoll

Note: For maximum efficiency, N should be set to the number of core available on the system that run the program.

Type of connection

So far the only protocol implemented is HTTP/1.1. Where applicable, we have implemented two variants of each client, one with the Connection header set to close and one with the header set to keep-alive. With close, a connection is closed after one file is downloaded and another connection must be open for the next file to be downloaded. In the implementations that use one connection per downloaded file, the connection type is naturally set to close. With keep-alive, the connection stays open after the initial GET request is completed, allowing for another GET request to be sent through the same connection. This allows for multiple files to be downloaded through the same connection without the need to reopen a socket to the server after each download. Therefore, using keep-alive connections save a significant amount of round trips and prevent unnecessary latency especially in the context of downloading a large number of small files.

Pipelining

Pipelining is an HTTP/1.1 feature that allow a client to send multiple requests in a row when using keep-alive connections. For each keep-alive client we implemented a version with pipelining and a version without pipelining. The versions without pipelining wait until the requested file is completely downloaded to send the request for the next file. The version with pipelining send all the requests at once and then download all the files sequentially without the need for additional requests, therefore avoiding unnecessary latency.

Utilities

File generator

The source file file_generator.c contain a file generator that sequentially generate nb_file files filled with size bytes of random data extracted from /dev/urandom with the getrandom system call. The files are numbered from 0 to nb_file-1 and are written in path. The generated files should be transferred to a local or a remote HTTP server before running the benchmark. The file generator will also create a file named checksums containing the md5 hash of each generated file.

Compilation:

gcc -Ofast -lm -lcrypto -o file_generator file_generator.c

Command line usage:

./file_generator path size nb_file

Benchmarking script

The source file benchmark.sh is a bash script provided to perform the benchmark. This script sequentially compiles and then execute each implementation with the perf command line tool.

Command line usage:

./benchmark.sh server port path size nb_file nb_conn nb_rep

The first 6 arguments are forwarded to the clients. The clients will ignore the nb_conn argument if it’s not relevant to their implementation. The nb_rep argument is the number of times the execution is repeated. perf will output the average execution time and the standard deviation along with other statistics.

License

This project source code is licensed under the GNU Affero General Public License v3 (AGPLv3). AGPLv3 is a strong copyleft license and a close variant of GPLv3 with only one additional requirement related to remote network interactions: if you modify this project and allow remote network interaction with its software, you must prominently offer the opportunity to all users to receive the source code of that modified version at no charge. Furthermore, we grant additional permission, under section 7 of the license, to link or to combine any part of this project with the OpenSSL library or a modified version of that library containing parts covered by the terms of the OpenSSL or SSLeay licenses.

Here’s a summary of the AGPLv3 license extracted from choosealicense.com (CC BY 3.0):

Permissions Conditions Limitations
Commercial use Disclose source Liability
Distribution License and copyright notice Warranty
Modification Network use is distribution
Patent use Same license
Private use State changes

The choice of AGPLv3 is motivated by the fact that this project provides client side network software and will eventually provide server side network software. The client software does not provide a possibility for remote interactions but the server software will provide such a possibility and will, therefore, provide a way for anyone to directly access its own source code.

Citation

Please cite our work when using our software or part of our software in your own software, research or publication.

BibTeX entry:

@misc{couturier_linux_2018,
    title = {The {Linux} {HTTP} {Client} {Benchmark}},
    url = {https://smoolak.com/git/Smoolak/TheLinuxHTTPClientBenchmark},
    author = {Couturier, Andy},
    publisher = {Smoolak's git server},
    year = {2018}
}

IEEE:

A. Couturier, The Linux HTTP Client Benchmark. Smoolak’s git server, 2018.

APA:

Couturier, A. (2018). The Linux HTTP Client Benchmark. Smoolak’s git server. Retrieved from https://smoolak.com/git/Smoolak/TheLinuxHTTPClientBenchmark

TO DO

  • Add comments to code and more documentation
  • Add an option to disable the checksums
  • Add select, poll and epoll level-triggered implementations
  • Add HTTP/1.0 and HTTP/2 implementations
  • Add support for encrypted connections
  • Add custom server implementations and custom protocols
  • Add a real example of a benchmark with each step explained and the execution results