tree eac76dfc24520620abd8e0775cf5084907768ff8
parent 23e585a74faf2137d9b7f8f1abfa18928f1f981b
author Todd Wang <toddw@google.com> 1464238142 -0700
committer Todd Wang <toddw@google.com> 1464243304 -0700

Change vom decoder benchmarks to allocate new value to fill.

I got bored of making the numbers better, so I made them worse.:)

The old vom decoder benchmarks repeatedly decoded into the same
value.  While that's nice because it avoids allocation of the
value on each decode operation, it's also unrealistic.  More
common is to decode into new values, e.g. for RPC in and out
args, or syncbase deserialization.

It also makes the numbers incomparable to gob, which has the
behavior that if you decode into a slice, it'll re-use the slice
if it has enough capacity, and overwrite existing elements of the
underlying array.  We've explicitly chosen not to implement this
behavior, since it makes it easier for users to introduce subtle
bugs.  We've encountered these bugs in practice; a very old
version of vom had the same behavior as gob, and our code
generated iterators for streaming RPCs had this bug.

The new decoder benchmarks allocate a new value to fill in on
every decode operation.  Some numbers below.

+ Compare gob to vom reflect (expect similar or better performance)
+ String: gob uses package unsafe to avoid an allocation
+ List,Map,Struct: need to investigate
benchmark                     GOB ns/op   VOM ns/op      delta
DecodeMany_XNumber                 567         512      -9.70%
DecodeMany_XStringSmall            667         737     +10.49%
DecodeMany_XStringLarge          19989       32438     +62.28%
DecodeMany_XByteListSmall          797         814      +2.13%
DecodeMany_XByteListLarge        23315       19047     -18.31%
DecodeMany_XByteArray              777         840      +8.11%
DecodeMany_XArray                  806         850      +5.46%
DecodeMany_XListSmall             1168        1148      -1.71%
DecodeMany_XListLarge          2021911     3383709     +67.35%
DecodeMany_XMap                   2343        3149     +34.40%
DecodeMany_XSmallStruct            865        2530    +192.49%
DecodeMany_XLargeStruct           2538       30767   +1112.25%
DecodeMany_XLargeStructZero        643         876     +36.24%
+ VOM has a specific optimization for the single-shot cases below
Decode_____XNumber                2469        1250     -49.37%
Decode_____XStringSmall           2627        1418     -46.02%
Decode_____XStringLarge          39046       32903     -15.73%
Decode_____XByteListSmall         2942        1572     -46.57%
Decode_____XByteListLarge        42076       20364     -51.60%
Decode_____XByteArray            31054        1712     -94.49%
Decode_____XArray                31005        1821     -94.13%
Decode_____XListSmall            31588        2110     -93.32%
Decode_____XListLarge          2091515     3358565     +60.58%
Decode_____XMap                  32873        4102     -87.52%
Decode_____XSmallStruct          33665        3533     -89.51%
Decode_____XLargeStruct          83570       32130     -61.55%
Decode_____XLargeStructZero      81377        2102     -97.42%

+ Compare gob to vom codegen (expect vom better)
+ String: gob uses package unsafe to avoid an allocation
+ List,Struct,Time: need to investigate
benchmark                     GOB ns/op   VOM ns/op      delta
DecodeMany_VNumber                 567         287     -49.38%
DecodeMany_VStringSmall            667         409     -38.68%
DecodeMany_VStringLarge          19989       31256     +56.37%
DecodeMany_VByteListSmall          797         472     -40.78%
DecodeMany_VByteListLarge        23315       17734     -23.94%
DecodeMany_VByteArray              777         388     -50.06%
DecodeMany_VArray                  806         479     -40.57%
DecodeMany_VListSmall             1168         567     -51.46%
DecodeMany_VListLarge          2021911     2458774     +21.61%
DecodeMany_VMap                   2343        1305     -44.30%
DecodeMany_VSmallStruct            865         855      -1.16%
DecodeMany_VLargeStruct           2538        5914    +133.02%
DecodeMany_VLargeStructZero        643         423     -34.21%
DecodeMany_Time                    715        1733    +142.38%
+ VOM has a specific optimization for the single-shot cases below
Decode_____VNumber                2469         876     -64.52%
Decode_____VStringSmall           2627        1058     -59.73%
Decode_____VStringLarge          39046       32050     -17.92%
Decode_____VByteListSmall         2942        1073     -63.53%
Decode_____VByteListLarge        42076       18806     -55.30%
Decode_____VByteArray            31054         963     -96.90%
Decode_____VArray                31005        1222     -96.06%
Decode_____VListSmall            31588        1283     -95.94%
Decode_____VListLarge          2091515     2437164     +16.53%
Decode_____VMap                  32873        2140     -93.49%
Decode_____VSmallStruct          33665        1669     -95.04%
Decode_____VLargeStruct          83570        7061     -91.55%
Decode_____VLargeStructZero      81377        1403     -98.28%
Decode_____Time                  30370        2639     -91.31%

+ Compare old to new (expect new worse, with 1 extra allocation)
+ Only provided for reference
benchmark                          old ns/op  new ns/op    delta
Vom___Decode_____XNumber               1204       1250    +3.82%
Vom___DecodeMany_XNumber                489        512    +4.70%
Vom___Decode_____VNumber                798        876    +9.77%
Vom___DecodeMany_VNumber                254        287   +12.99%
Vom___Decode_____XStringSmall          1304       1418    +8.74%
Vom___DecodeMany_XStringSmall           610        737   +20.82%
Vom___Decode_____VStringSmall           928       1058   +14.01%
Vom___DecodeMany_VStringSmall           344        409   +18.90%
Vom___Decode_____XStringLarge         33214      32903    -0.94%
Vom___DecodeMany_XStringLarge         32505      32438    -0.21%
Vom___Decode_____VStringLarge         32223      32050    -0.54%
Vom___DecodeMany_VStringLarge         31378      31256    -0.39%
Vom___Decode_____VEnum                  839        917    +9.30%
Vom___DecodeMany_VEnum                  273        330   +20.88%
Vom___Decode_____XByteListSmall        1421       1572   +10.63%
Vom___DecodeMany_XByteListSmall         719        814   +13.21%
Vom___Decode_____VByteListSmall         991       1073    +8.27%
Vom___DecodeMany_VByteListSmall         381        472   +23.88%
Vom___Decode_____XByteListLarge       19752      20364    +3.10%
Vom___DecodeMany_XByteListLarge       18975      19047    +0.38%
Vom___Decode_____VByteListLarge       18768      18806    +0.20%
Vom___DecodeMany_VByteListLarge       18022      17734    -1.60%
Vom___Decode_____XByteArray            1409       1712   +21.50%
Vom___DecodeMany_XByteArray             742        840   +13.21%
Vom___Decode_____VByteArray             874        963   +10.18%
Vom___DecodeMany_VByteArray             333        388   +16.52%
Vom___Decode_____XArray                1685       1821    +8.07%
Vom___DecodeMany_XArray                 752        850   +13.03%
Vom___Decode_____VArray                1122       1222    +8.91%
Vom___DecodeMany_VArray                 407        479   +17.69%
Vom___Decode_____XListSmall            2092       2110    +0.86%
Vom___DecodeMany_XListSmall            1052       1148    +9.13%
Vom___Decode_____VListSmall            1179       1283    +8.82%
Vom___DecodeMany_VListSmall             460        567   +23.26%
Vom___Decode_____XListLarge         3348542    3358565    +0.30%
Vom___DecodeMany_XListLarge         3439946    3383709    -1.63%
Vom___Decode_____VListLarge         2398669    2437164    +1.60%
Vom___DecodeMany_VListLarge         2412183    2458774    +1.93%
Vom___Decode_____XListAnySmall         4081       4324    +5.95%
Vom___DecodeMany_XListAnySmall         2820       2897    +2.73%
Vom___Decode_____VListAnySmall         2594       2695    +3.89%
Vom___DecodeMany_VListAnySmall         1562       1629    +4.29%
Vom___Decode_____XListAnyLarge     36501787   36624993    +0.34%
Vom___DecodeMany_XListAnyLarge     35729354   34987568    -2.08%
Vom___Decode_____VListAnyLarge     25786589   25716237    -0.27%
Vom___DecodeMany_VListAnyLarge     24609779   24009331    -2.44%
Vom___Decode_____VSet                  1824       1939    +6.30%
Vom___DecodeMany_VSet                  1059       1144    +8.03%
Vom___Decode_____XMap                  3879       4102    +5.75%
Vom___DecodeMany_XMap                  2920       3149    +7.84%
Vom___Decode_____VMap                  2052       2140    +4.29%
Vom___DecodeMany_VMap                  1244       1305    +4.90%
Vom___Decode_____XSmallStruct          3267       3533    +8.14%
Vom___DecodeMany_XSmallStruct          2303       2530    +9.86%
Vom___Decode_____VSmallStruct          1573       1669    +6.10%
Vom___DecodeMany_VSmallStruct           735        855   +16.33%
Vom___Decode_____XLargeStruct         32316      32130    -0.58%
Vom___DecodeMany_XLargeStruct         28746      30767    +7.03%
Vom___Decode_____VLargeStruct          6978       7061    +1.19%
Vom___DecodeMany_VLargeStruct          5662       5914    +4.45%
Vom___Decode_____XLargeStructZero      1896       2102   +10.86%
Vom___DecodeMany_XLargeStructZero       724        876   +20.99%
Vom___Decode_____VLargeStructZero      1233       1403   +13.79%
Vom___DecodeMany_VLargeStructZero       298        423   +41.95%
Vom___Decode_____VSmallUnion           2266       2424    +6.97%
Vom___DecodeMany_VSmallUnion           1230       1337    +8.70%
Vom___Decode_____Time                  2592       2639    +1.81%
Vom___DecodeMany_Time                  1680       1733    +3.15%
Vom___Decode_____Blessings            55686      56661    +1.75%
Vom___DecodeMany_Blessings            53738      54714    +1.82%
Vom___Decode_____RPCRequestZero        1656       1862   +12.44%
Vom___DecodeMany_RPCRequestZero         313        487   +55.59%
Vom___Decode_____RPCRequestFull       57124      58312    +2.08%
Vom___DecodeMany_RPCRequestFull       54537      55910    +2.52%
Vom___Decode_____RPCResponseZero       1667       1856   +11.34%
Vom___DecodeMany_RPCResponseZero        340        487   +43.24%
Vom___Decode_____RPCResponseFull      44018      44654    +1.44%
Vom___DecodeMany_RPCResponseFull      40320      40772    +1.12%

Change-Id: I151dfe9a83bd066405aa395a03723c25c2d358c2