Wireshark-users: [Wireshark-users] bug in sub-allocator emem.c - guard pages cause unnecessary 'out of memory' condition

From: bob frazier <bfrazier@xxxxxxxxxxx>
Date: Fri, 08 Aug 2008 09:09:05 -0700

for very large files (1 hour streaming video capture) tshark will crash when doing analysis of RTP packets (let's say you're extracting specific data and filtering the output and generating text). On FreeBSD there is an unhandled 'out of memory' exception that causes the crash. This crash can be prevented (at least to some extent) by disabling "guard pages" and "canaries" in emem.c, then doing a full re-build (incremental build was insufficient).

The problem appears to be a serious virtual address fragmentation problem, since when
the crash happens the virtual address space is >2Gb, while actual memory used is in the
neighborhood of 200Mb.  Removing guard pages and canaries seems to resolve the problem.

tshark command line looked like this (on a >17Gb capture of RTP/RTSP/RTCP data)

tshark -r test.pcap -p -R"(ip.dst==192.168.1.100 && rtp.p_type==96)" -Tfields
-eframe.number -eframe.time_relative -ertp.p_type -ertp.extseq

Similar problems also exist in the WIN32 version. Modifying the code to compile the "no guard page" 'malloc' sections (in lieu of 'VirtualAlloc' + 'VirtualProtect') for WIN32 also resolves THAT problem.