Project Information Environment

Simulating Production Environments

For this project, we aim for each framework to be configured according to its best practices for production deployments. Because the specifics vary by framework, we especially appreciate review from maintainers of each framework to confirm that the implementation(s) we have are indeed compliant with best practices.

The goal is to approximate and measure a sensible production-like deployment of each framework.

Environment Details

For language and framework specific details, view their README within their respective directory.

Hardware

Citrine (Self-hosted)

• BENCHMARK_ENV=Citrine
• Switched 10-gigabit ethernet

All 3 machines: tfb-server, tfb-database, tfb-client

• 28 HT Cores Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz
• 32GB RAM
• Ubuntu 18.04.3 4.15.0-88-generic #88-Ubuntu

Note: Hardware details used for the various rounds can be found on the results website.

The sysctl was edited with the following values:

net.ipv4.tcp_max_syn_backlog = 65535
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1
vm.max_map_count = 1048576

Azure

• BENCHMARK_ENV=Azure

All 3 machines: tfb-app-vm, tfb-data-vm, tfb-load-vm

• Standard_D3_v2 (4 vcpus, 14 GB memory)
• Default 30 GB of managed disk size (Standard HDD LRS)
• TFB run and Docker images both utilize the temporary storage (under /mnt) of the Standard_D3_v2, which has 200 GB of SSD for the temporary storage
• The network interface for each of the 3 VMs has the accelerated networking option enabled
• The 3 VMs communicate via a virtual internal network

More information coming soon...

Load simulator

• Wrk

Historical Environments

Cloud Environments

AWS (rounds 1 through 12)

Amazon EC2 c3.large instances (2 vCPU each); switched gigabit Ethernet (m1.large was used through Round 9).

Physical Environments

ServerCentral (rounds 13 through 15)

Dell R910 (4x 10-Core Intel Xeon E7-4850 CPUs) application server; Dell R710 (2x 4-Core Intel Xeon E5520 CPUs) database server; switched 10-gigabit Ethernet. Provided by ServerCentral.

Peak (rounds 9 through 12)

Dell R720xd dual Intel Xeon E5-2660 v2 (40 HT cores) with 32 GB memory; database servers equipped with SSDs in RAID; switched 10-gigabit Ethernet. Provided by Peak Hosting.

i7 (rounds 1 through 8)

In-house Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory (early 2011 vintage); database server equipped with Samsung 840 Pro SSD; switched gigabit Ethernet.