Quick Update: Capacity Testing
Posted by Dave
Another late-night coding session, another learning experience... It seems that the Linux per-process open file descriptor limit that had me a bit concerned has a bark that's worse than its bite:
dave@server:~$ ps aux | grep meteord
meteor 10613 76.2 6.6 11156 8356 pts/1 R Feb23 13:11 meteord daemon
ddr4179 11732 0.0 0.6 3004 768 pts/2 R+ 00:00 0:00 grep meteord
dave@server:~$ ulimit -n
1024
dave@server:~$ sudo lsof -p 10613 | wc -l
1884
As it turns out, some limits in the meteord.conf file were preventing persistent connections from many of the Deacon instances in my previous test, and so only a limited number were able to hold a connection at any given time. I've fixed that now, and the tests seem to be running a bit more smoothly. More results as I get them...
Testing: Epilogue to Chapter 1
Posted by Dave
Outside the delightful task of cleaning and organizing our basement, this weekend offered some prime coding time - and I spent it chewing through a new and improved Deacon+Meteor test suite.
Unfortunately, the results have been more cause for confusion than celebration. The new suite uses a pair of dedicated machines - one server, one client - on an isolated network with PTP-synchronized clocks and a socket-based side channel for control. It successfully addresses the error introduced by including push send time in the round-trip duration, but exposes many more weaknesses in my previous tests.
While I didn't seem to encounter this limit before, I'm now having great difficulty getting either the client or server machines past 1,024 connections - an upper-bound imposed by Linux's per-process file descriptor limit. This limit is configurable, but still seems to be leading me on a merry chase. I also discovered that the previous test didn't account for Deacon instances that weren't actually connected to the server - so I've added some extra accessors and state tracking to Deacon to prevent the test from advancing before all the clients are ready.
I'm still making progress - so stay tuned and hopefully I'll have some better news to share soon. In the mean time, suggestions are always welcome via the comments or mailing list...