node: unix sockets are slow in case of lots of small buffers

[BridgeAR]

While running the node_redis benchmarks with unix sockets I see a huge performance decrease opposed to TCP while pipelining lots of small buffers (4byte || 4kb) to the stream. The performance will drop to 50% of TCP while benchmarking the small buffers and increase by about 50% by using 4mb buffers opposed to TCP.
Writing strings to the stream stays about the same or increases the performance a tiny bit.

I tried to work out where the difference came from but this is handled in C and I have no clue about C.

As a side note: node_redis mixes buffers and strings while writing buffers to the stream. Strictly only using buffers does not change the outcome though.

clients: 1, NodeJS: 5.5.0, Redis: 3.0.5, parser: hiredis, connected by: socket
   SET 4B str,         1/1 min/max/avg:   0.01/  2.77/  0.02   2501ms total,   54347 ops/sec
   SET 4B str,  batch 50/1 min/max/avg:   0.09/  3.41/  0.11   2501ms total,  430688 ops/sec
   SET 4B buf,         1/1 min/max/avg:   0.02/  3.11/  0.03   2501ms total,   37721 ops/sec
   SET 4B buf,  batch 50/1 min/max/avg:   0.49/  4.43/  0.59   2501ms total,   83826 ops/sec
 SET 4KiB str,         1/1 min/max/avg:   0.02/  3.52/  0.02   2501ms total,   39837 ops/sec
 SET 4KiB str,  batch 50/1 min/max/avg:   0.25/  1.96/  0.29   2501ms total,  172891 ops/sec
 SET 4KiB buf,         1/1 min/max/avg:   0.02/  2.79/  0.03   2501ms total,   35443 ops/sec
 SET 4KiB buf,  batch 50/1 min/max/avg:   0.53/  6.28/  0.67   2501ms total,   73870 ops/sec
 SET 4MiB str,         1/1 min/max/avg:   3.54/ 20.85/  4.41   2501ms total,     226 ops/sec
 SET 4MiB str,  batch 20/1 min/max/avg:  67.20/ 89.91/ 75.32   2561ms total,     266 ops/sec
 SET 4MiB buf,         1/1 min/max/avg:   1.25/  4.78/  1.37   2501ms total,     724 ops/sec
 SET 4MiB buf,  batch 20/1 min/max/avg:  24.51/ 36.13/ 26.51   2520ms total,     754 ops/sec

clients: 1, NodeJS: 5.5.0, Redis: 3.0.5, parser: hiredis, connected by: tcp
   SET 4B str,         1/1 min/max/avg:   0.02/  5.21/  0.03   2501ms total,   34392 ops/sec
   SET 4B str,  batch 50/1 min/max/avg:   0.10/  3.37/  0.11   2501ms total,  439584 ops/sec
   SET 4B buf,         1/1 min/max/avg:   0.04/  3.61/  0.05   2501ms total,   19350 ops/sec
   SET 4B buf,  batch 50/1 min/max/avg:   0.25/  6.18/  0.29   2501ms total,  170292 ops/sec
 SET 4KiB str,         1/1 min/max/avg:   0.03/  5.49/  0.03   2501ms total,   29333 ops/sec
 SET 4KiB str,  batch 50/1 min/max/avg:   0.27/  6.38/  0.32   2501ms total,  152879 ops/sec
 SET 4KiB buf,         1/1 min/max/avg:   0.04/  3.63/  0.06   2501ms total,   17617 ops/sec
 SET 4KiB buf,  batch 50/1 min/max/avg:   0.31/  3.92/  0.36   2501ms total,  136026 ops/sec
 SET 4MiB str,         1/1 min/max/avg:   4.01/ 20.96/  5.29   2501ms total,     189 ops/sec
 SET 4MiB str,  batch 20/1 min/max/avg:  78.62/117.93/ 87.70   2544ms total,     228 ops/sec
 SET 4MiB buf,         1/1 min/max/avg:   1.82/ 10.09/  2.28   2502ms total,     436 ops/sec
 SET 4MiB buf,  batch 20/1 min/max/avg:  38.21/ 51.92/ 41.36   2523ms total,     484 ops/sec

Ping @trevnorris

[mscdex]

What if you explicitly use .cork()/.uncork() in node_redis to batch writes?

[BridgeAR]

@mscdex this is already in use but it does not change anything by removing it.

[GeorgeBailey]

Thanks for bringing this up. I had a similar experience in Node 0.10.36 about 11 months ago, but opted to use STDIO for my project.

[mcollina]

The way currently cork it's being currently done in node_redis is not the most efficient (as it is right now). Basically it calls cork/uncork multiple times during the same synchronous flow, effectively crossing the C++ barrier multiple times. @BridgeAR have you tried calling uncork in nextTick and see if there is any difference? That will likely batch up all your writes as they were a single one.

[BridgeAR]

Using nextTick does not change the performance in my tests. It stays in the tolerance range after each test run. There might be a increase that I can't measure well because of the noise but it won't be significant.

[mcollina]

Run the same thing with http://npm.im/0x, and upload the results somewhere, so we can have a look.

It might even be OS specific (maybe some tuning in the kernel). Be careful of not running it on OS X, I had some surprises when running stuff on Linux for production.