| This directory contains code uses to measure the performance of the Vanadium IPC |
| stack. |
| |
| ================================================================================ |
| |
| The ipc_test.go file uses GO's testing package to run benchmarks. Each |
| benchmark involves one server and one client. The server has two very simple |
| methods that echo the data received from the client back to the client. |
| |
| client ---- Echo(payload) ----> server |
| client <--- return payload ---- server |
| |
| There are two versions of the Echo method: |
| - Echo(payload []byte) ([]byte], error) |
| - EchoStream() <[]byte,[]byte> error |
| |
| The first benchmarks use the non-streaming version of Echo with a varying |
| payload size. The second benchmarks use the streaming version with varying |
| number of chunks and payload sizes. The third one is for measuring the |
| performance with multiple clients hosted in the same process. |
| |
| This test creates a VC before the benchmark begins. So, the VC creation |
| overhead is excluded. |
| |
| $ v23 go test -bench=. -timeout=1h -cpu=1 -benchtime=5s \ |
| v.io/x/ref/profiles/internal/ipc/benchmark |
| PASS |
| Benchmark____1B 1000 8301357 ns/op 0.00 MB/s |
| --- Histogram (unit: ms) |
| Count: 1000 Min: 7 Max: 17 Avg: 7.89 |
| ------------------------------------------------------------ |
| [ 7, 8) 505 50.5% 50.5% ##### |
| [ 8, 9) 389 38.9% 89.4% #### |
| [ 9, 10) 38 3.8% 93.2% |
| [ 10, 11) 12 1.2% 94.4% |
| [ 11, 12) 4 0.4% 94.8% |
| [ 12, 14) 19 1.9% 96.7% |
| [ 14, 16) 23 2.3% 99.0% |
| [ 16, 18) 10 1.0% 100.0% |
| [ 18, 21) 0 0.0% 100.0% |
| [ 21, 24) 0 0.0% 100.0% |
| [ 24, inf) 0 0.0% 100.0% |
| Benchmark___10B 1000 8587341 ns/op 0.00 MB/s |
| ... |
| |
| RESULTS.txt has the full benchmark results. |
| |
| ================================================================================ |
| |
| benchmarkd/main.go and benchmark/main.go are simple command-line tools to run the |
| benchmark server and client as separate processes. Unlike the benchmarks above, |
| this test includes the startup cost of name resolution, creating the VC, etc. in |
| the first RPC. |
| |
| $ v23 go run benchmarkd/main.go \ |
| -v23.tcp.address=localhost:8888 -v23.permissions.literal='{"Read": {"In": ["..."]}}' |
| |
| (In a different shell) |
| $ v23 go run benchmark/main.go \ |
| -server=/localhost:8888 -iterations=100 -chunk_count=0 -payload_size=10 |
| iterations: 100 chunk_count: 0 payload_size: 10 |
| elapsed time: 1.369034277s |
| Histogram (unit: ms) |
| Count: 100 Min: 7 Max: 94 Avg: 13.17 |
| ------------------------------------------------------------ |
| [ 7, 8) 1 1.0% 1.0% |
| [ 8, 9) 4 4.0% 5.0% |
| [ 9, 10) 17 17.0% 22.0% ## |
| [ 10, 12) 24 24.0% 46.0% ## |
| [ 12, 15) 24 24.0% 70.0% ## |
| [ 15, 19) 28 28.0% 98.0% ### |
| [ 19, 24) 1 1.0% 99.0% |
| [ 24, 32) 0 0.0% 99.0% |
| [ 32, 42) 0 0.0% 99.0% |
| [ 42, 56) 0 0.0% 99.0% |
| [ 56, 75) 0 0.0% 99.0% |
| [ 75, 101) 1 1.0% 100.0% |
| [101, 136) 0 0.0% 100.0% |
| [136, 183) 0 0.0% 100.0% |
| [183, 247) 0 0.0% 100.0% |
| [247, 334) 0 0.0% 100.0% |
| [334, inf) 0 0.0% 100.0% |
| |
| |
| On a Raspberry Pi, everything is much slower. The same tests show the following |
| results: |
| |
| $ ./benchmarks.test -test.bench=. -test.cpu=1 -test.benchtime=5s 2>/dev/null |
| PASS |
| Benchmark____1B 500 21316148 ns/op |
| Benchmark___10B 500 23304638 ns/op |
| Benchmark__100B 500 21860446 ns/op |
| Benchmark___1KB 500 24000346 ns/op |
| Benchmark__10KB 200 37530575 ns/op |
| Benchmark_100KB 100 136243310 ns/op |
| Benchmark_N_RPCs____1_chunk_____1B 500 19957506 ns/op |
| Benchmark_N_RPCs____1_chunk____10B 500 22868392 ns/op |
| Benchmark_N_RPCs____1_chunk___100B 500 19635412 ns/op |
| Benchmark_N_RPCs____1_chunk____1KB 500 22572190 ns/op |
| Benchmark_N_RPCs____1_chunk___10KB 500 37570948 ns/op |
| Benchmark_N_RPCs___10_chunks___1KB 100 51670740 ns/op |
| Benchmark_N_RPCs__100_chunks___1KB 50 364938740 ns/op |
| Benchmark_N_RPCs_1000_chunks___1KB 2 3586374500 ns/op |
| Benchmark_1_RPC_N_chunks_____1B 10000 1034042 ns/op |
| Benchmark_1_RPC_N_chunks____10B 5000 1894875 ns/op |
| Benchmark_1_RPC_N_chunks___100B 5000 2857289 ns/op |
| Benchmark_1_RPC_N_chunks____1KB 5000 6465839 ns/op |
| Benchmark_1_RPC_N_chunks___10KB 100 80019430 ns/op |
| Benchmark_1_RPC_N_chunks__100KB Killed |
| |
| The simple 1 KB RPCs take an average of 24 ms. The streaming equivalent takes |
| about 22 ms, and streaming many 1 KB chunks takes about 6.5 ms per chunk. |
| |
| |
| $ ./benchmarkd --address=localhost:8888 --v23.permissions.literal='{"...":"A"}' |
| |
| $ ./benchmark --server=/localhost:8888 --count=10 --payload_size=1000 |
| CallEcho 0 2573406000 |
| CallEcho 1 44669000 |
| CallEcho 2 54442000 |
| CallEcho 3 33934000 |
| CallEcho 4 47985000 |
| CallEcho 5 61324000 |
| CallEcho 6 51654000 |
| CallEcho 7 47043000 |
| CallEcho 8 44995000 |
| CallEcho 9 53166000 |
| |
| On the pi, the first RPC takes ~2.5 sec to execute. |
