Hi Martin,
To answer your questions (and again, note this only applies to
high-frequency algo-type trading and microsecond/millisecond analysis)
"Sure, latency monitoring using Wireshark wouldn't do, however, AFAIU,
monitoring of the throughput - say with a window 1 sec long - may work, no?"
1 sec these days is the equivalent of 10 seconds polling a few years ago.
If your polling rate is every 10seconds or 30seconds, what happens to peaks
of traffics within those seconds? They are not reported.
Similarly, with 1sec polling, you don't see a spike that lasts
100milliseconds or 100microseconds. On a particular service we use, spikes
at this time scale are told to go from ~6mbps to +110mbps.
"As for the cards, I recall Endace claiming microsecond (or even lower)
granularities. Does it mean that when using Wireshark for the analysis
of the capured data, microseconds are simply stripped off?"
In fact, I recall some partnership between CACE and Endace recently been
announced (I'm sure some people on this list know more about this). I
wonder if this relationship has advantages for CACE's capture card which
seems very good for the asking price.
I'm assuming that with a capture card (such as endace's) providing a very
low time granularity, results are more accurate. However, I am not sure at
which point within the OS wireshark makes a record of the packets.
Kernel, server load, disk access etc. all play a part when trying to capture
details at this level.
The solutions that promise to provide 100% accuracy at this level are
unfortunately very expensive at this stage. For those of us with a budget
to consider we try and come up with a compromise or alternative solution.
Considering it's free, wireshark has been, and still is, a fantastic tool.
Again, I think your article serves a good purpose, just keep in mind the
outcome you are looking for. The limitation is not within your explanation,
but rather within the tools available.
Cheers,
------------------------------
Message: 8
Date: Mon, 09 Feb 2009 11:57:06 +0100
From: Martin Sustrik <sustrik@xxxxxxxxxx>
Subject: Re: [Wireshark-users] Wireshark & monitoring in the
enterprise environment
To: Community support list for Wireshark
<wireshark-users@xxxxxxxxxxxxx>
Message-ID: <49900C02.1040604@xxxxxxxxxx>
Content-Type: text/plain; charset=us-ascii; format=flowed
Mario,
Thanks for comments, find my replies inlined.
mv652@xxxxxxxxxxxx wrote:
In my opinion, I think you've written a well documented introduction for
anyone looking to monitor Financial Trading (or similar) Data. It gives a
very nice and simple methodology to get a 'general feel' for the traffic
passing over the network.
One caveat I'd add with regards to monitoring Financial Trading activity
(and I'm happy if anyone can explain any different), is that this is good
for a general network data analysis. For high-frequency algo-type trading
activity, wireshark seems to have found it's limitation (at least for the
moment).
The granularity of the wireshark I/O is at a minimum of 0.001sec (1
millisecond). In high-frequency trading we are now monitoring at the depths
of microseconds (0.000001 sec) as opposed to milliseconds. There are two
problems as a consequence:
1) The NIC used for 'monitoring' is unable to poll at these levels (CACE's
TurboCap only provides 3-5 microsecond granularity, and this is a
'specialized' monitoring card)
Sure, latency monitoring using Wireshark wouldn't do, however, AFAIU,
monitoring of the throughput - say with a window 1 sec long - may work, no?
As for the cards, I recall Endace claiming microsecond (or even lower)
granularities. Does it mean that when using Wireshark for the analysis
of the capured data, microseconds are simply stripped off?
2) Wireshark does provide microsecond timsetamps for the data, but the
accuracy of the data is questionable, dependable on the hardware used and
other processes occuring at the time on the monitoring system.
Please don't get me wrong. I think wireshark is a fantastic network
analysis tool and a lot of good work has been done by the authors and the
community to make it the great piece of software it is today. Just to point
out that I think you should indicate there is a limitation to the analysis
of data that can be captured.
I only say this purely due to the fact the document been written analysing
Financial Trading traffic.
Yup, understood.
Martin
