On Sep 18, 2011, at 8:45 PM, Ye Deng wrote:
> I have a serious issue when using "mergecap" and "editcap" tools for my project.
> e.g. If I try to merge many pcap files captured at my home, I sometimes got errors saying, "mergecap: Error reading my_pcap_file12: File contains a record that's not valid (pcap: File has 16793778-byte packet, bigger than maximum of 65535)".
>
> My question is:
> Is there any existing tool (e.g. an "improved mergecap") that can skip the unrecognizable packets, and process the resting valid packets?
None that I know of. The program would have to use something other than either libpcap or Wireshark's Wiretap code to read the capture file, because (as you've discovered) both of them regard packets with a size bigger than 65535 as invalid.
> After I did some researches online, I found it may be caused by file transfers using HTTP/FTP in some text mode.
> Please search "corrupt" on this webpage below.
> http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html
> Therefore, I think the pcap-next-generation-dump-file can deal with this issue.
Yes, it can deal with this issue.
It deals with it by having a field in the file that will be changed if you transfer a file in text mode between systems with different line ending conventions (for example, between Windows and UN*X) and by treating a file with a wrong value in that field as being damaged.
It does *NOT* deal with it by doing anything more than that. In particular, it does *NOT*, and cannot, magically undo the damage done to the file by transferring it in text mode.
> But I tried it in Wireshark, and got an assertion failure, which shows that it is still unfinished...
No, that shows that there's a bug somewhere. What was the assertion failure? We'd like to fix the bug, but we'd need to know the assertion failure.
However, even if we fix that bug, and any other bugs you run into:
1) it will not magically be able to read pcap-ng files that have been damaged by being transferred in text mode;
2) even if it could (which it can't, as there's no way for it to figure out where, in the file, the pair of bytes 0x0d 0x0a was turned into the single byte 0x0a, or the single byte 0x0a was turned into the pair of bytes 0x0d 0x0a - the first of those would happen if a file were transferred in text mode from Windows to UN*X, the second of those would happen if a file were transferred in text mode from UN*X to Windows), it wouldn't help you, because your file is in pcap format, not pcap-ng format.
Furthermore, if that was the cause of the problem, there is no guarantee that a packet that happens to have a size <= 65535 wasn't damaged, so even if you *were* to get a program that skips the bad packets, you'll have something that's usable. If the packet is claimed to have a captured length of 16793778 bytes, the only thing a program could do with it is to skip it, which means skipping 16793778 bytes, so it could end up skipping over other packets as well.
