| This directory contains code uses to measure the performance of the Veyron IPC |
| stack. |
| |
| ================================================================================ |
| |
| The ipc_test.go file uses GO's testing package to run benchmarks. Each |
| benchmark involves one server and one client. The server has two very simple |
| methods that echo the data received from the client back to the client. |
| |
| client ---- Echo(payload) ----> server |
| client <--- return payload ---- server |
| |
| There are two versions of the Echo method: |
| - Echo(payload []byte) ([]byte], error) |
| - EchoStream() <[]byte,[]byte> error |
| |
| The first benchmarks use the non-streaming version of Echo with a varying |
| payload size. The second benchmarks use the streaming version with varying |
| number of chunks and payload sizes. The third one is for measuring the |
| performance with multiple clients hosted in the same process. |
| |
| All these benchmarks create a VC before measurements begin. So, the VC creation |
| overhead is excluded. |
| |
| $ veyron go test -test.bench=. -timeout=1h -test.cpu=1 -test.benchtime=5s \ |
| veyron/runtimes/google/ipc/benchmarks |
| |
| Benchmark____1B 2000 5144219 ns/op 0.00 MB/s |
| Benchmark___10B 2000 5526448 ns/op 0.00 MB/s |
| Benchmark___1KB 2000 4528221 ns/op 0.44 MB/s |
| Benchmark_100KB 1000 7569096 ns/op 26.42 MB/s |
| Benchmark____1_chunk_____1B 1000 8945290 ns/op 0.00 MB/s |
| Benchmark____1_chunk____10B 1000 9711084 ns/op 0.00 MB/s |
| Benchmark____1_chunk____1KB 1000 8541689 ns/op 0.23 MB/s |
| Benchmark____1_chunk___10KB 1000 8972995 ns/op 2.23 MB/s |
| Benchmark___10_chunks____1B 1000 13114807 ns/op 0.00 MB/s |
| Benchmark___10_chunks___10B 1000 13219493 ns/op 0.02 MB/s |
| Benchmark___10_chunks___1KB 1000 13292236 ns/op 1.50 MB/s |
| Benchmark___10_chunks__10KB 500 15733197 ns/op 12.71 MB/s |
| Benchmark__100_chunks____1B 500 45078939 ns/op 0.00 MB/s |
| Benchmark__100_chunks___10B 200 49113585 ns/op 0.04 MB/s |
| Benchmark__100_chunks___1KB 100 57982457 ns/op 3.45 MB/s |
| Benchmark__100_chunks__10KB 100 81632487 ns/op 24.50 MB/s |
| Benchmark__per_chunk____1B 50000 357880 ns/op 0.01 MB/s |
| Benchmark__per_chunk___10B 20000 476941 ns/op 0.04 MB/s |
| Benchmark__per_chunk___1KB 10000 806491 ns/op 2.48 MB/s |
| Benchmark__per_chunk__10KB 10000 1185081 ns/op 16.88 MB/s |
| Benchmark____1B_mux___10_chunks___10B 1000 20235386 ns/op 0.00 MB/s |
| Benchmark____1B_mux___10_chunks___1KB 500 21346428 ns/op 0.00 MB/s |
| Benchmark____1B_mux__100_chunks___10B 100 72942436 ns/op 0.00 MB/s |
| Benchmark____1B_mux__100_chunks___1KB 100 81538481 ns/op 0.00 MB/s |
| |
| 'Benchmark___1KB' shows that it takes an average of 4.528 ms to |
| execute a simple Echo RPC with a 1 KB payload. |
| |
| 'Benchmark___10_chunks___1KB' shows that a streaming RPC with the |
| same payload (i.e. 10 chunks of 1 KB) takes about 13.292 ms on average. |
| |
| 'Benchmark__per_chunk___1KB' shows that sending a stream of 1 KB chunks |
| takes an average of 0.806 ms per chunk. |
| |
| 'Benchmark____1B_mux___10_chunks___1KB' shows that it takes an average |
| of 21.346 ms to execute a simple Echo RPC with a 1 B payload while streaming |
| 10 chunks of 1 KB payloads continuously in the same process. |
| |
| bm/main.go does the same benchmarks as ipc_test.go but with more varying |
| configurations and optional histogram outputs. |
| |
| $ veyron go run veyron/runtimes/google/ipc/benchmarks/bm/main.go \ |
| -test.cpu=1,2 -test.benchtime=5s -histogram |
| |
| RESULTS.txt has the latest benchmark results with main.go |
| |
| ================================================================================ |
| |
| bmserver/main.go and bmclient/main.go are simple command-line tools to run the |
| benchmark server and client as separate processes. Unlike the benchmarks above, |
| this test includes the startup cost of name resolution, creating the VC, etc. in |
| the first RPC. |
| |
| $ veyron go run veyron/runtimes/google/ipc/benchmarks/bmserver/main.go \ |
| -veyron.tcp.address=localhost:8888 -acl='{"In":{"...":"R"}}' |
| |
| (In a different shell) |
| $ veyron go run veyron/runtimes/google/ipc/benchmarks/bmclient/main.go \ |
| -server=/localhost:8888 -iterations=100 -chunk_count=0 -payload_size=10 |
| iterations: 100 chunk_count: 0 payload_size: 10 |
| elapsed time: 619.75741ms |
| Histogram (unit: ms) |
| Count: 100 Min: 4 Max: 54 Avg: 5.65 |
| ------------------------------------------------------------ |
| [ 4, 5) 42 42.0% 42.0% #### |
| [ 5, 6) 32 32.0% 74.0% ### |
| [ 6, 7) 8 8.0% 82.0% # |
| [ 7, 9) 13 13.0% 95.0% # |
| [ 9, 11) 3 3.0% 98.0% |
| [ 11, 14) 1 1.0% 99.0% |
| [ 14, 18) 0 0.0% 99.0% |
| [ 18, 24) 0 0.0% 99.0% |
| [ 24, 32) 0 0.0% 99.0% |
| [ 32, 42) 0 0.0% 99.0% |
| [ 42, 55) 1 1.0% 100.0% |
| [ 55, 72) 0 0.0% 100.0% |
| [ 72, 94) 0 0.0% 100.0% |
| [ 94, 123) 0 0.0% 100.0% |
| [123, 161) 0 0.0% 100.0% |
| [161, 211) 0 0.0% 100.0% |
| [211, inf) 0 0.0% 100.0% |
| |
| |
| On a Raspberry Pi, everything is much slower. The same tests show the following |
| results: |
| |
| $ ./benchmarks.test -test.bench=. -test.cpu=1 -test.benchtime=5s 2>/dev/null |
| PASS |
| Benchmark____1B 500 21316148 ns/op |
| Benchmark___10B 500 23304638 ns/op |
| Benchmark__100B 500 21860446 ns/op |
| Benchmark___1KB 500 24000346 ns/op |
| Benchmark__10KB 200 37530575 ns/op |
| Benchmark_100KB 100 136243310 ns/op |
| Benchmark_N_RPCs____1_chunk_____1B 500 19957506 ns/op |
| Benchmark_N_RPCs____1_chunk____10B 500 22868392 ns/op |
| Benchmark_N_RPCs____1_chunk___100B 500 19635412 ns/op |
| Benchmark_N_RPCs____1_chunk____1KB 500 22572190 ns/op |
| Benchmark_N_RPCs____1_chunk___10KB 500 37570948 ns/op |
| Benchmark_N_RPCs___10_chunks___1KB 100 51670740 ns/op |
| Benchmark_N_RPCs__100_chunks___1KB 50 364938740 ns/op |
| Benchmark_N_RPCs_1000_chunks___1KB 2 3586374500 ns/op |
| Benchmark_1_RPC_N_chunks_____1B 10000 1034042 ns/op |
| Benchmark_1_RPC_N_chunks____10B 5000 1894875 ns/op |
| Benchmark_1_RPC_N_chunks___100B 5000 2857289 ns/op |
| Benchmark_1_RPC_N_chunks____1KB 5000 6465839 ns/op |
| Benchmark_1_RPC_N_chunks___10KB 100 80019430 ns/op |
| Benchmark_1_RPC_N_chunks__100KB Killed |
| |
| The simple 1 KB RPCs take an average of 24 ms. The streaming equivalent takes |
| about 22 ms, and streaming many 1 KB chunks takes about 6.5 ms per chunk. |
| |
| |
| $ ./bmserver --address=localhost:8888 --acl='{"...":"A"}' |
| |
| $ ./bmclient --server=/localhost:8888 --count=10 --payload_size=1000 |
| CallEcho 0 2573406000 |
| CallEcho 1 44669000 |
| CallEcho 2 54442000 |
| CallEcho 3 33934000 |
| CallEcho 4 47985000 |
| CallEcho 5 61324000 |
| CallEcho 6 51654000 |
| CallEcho 7 47043000 |
| CallEcho 8 44995000 |
| CallEcho 9 53166000 |
| |
| On the pi, the first RPC takes ~2.5 sec to execute. |
