Wireshark · Ethereal-dev: Re: [Ethereal-dev] Re: NTAR - PCAP next generation dump file formatimplementation

[ronnie sahlberg <ronniesahlberg@xxxxxxxxx>]

> In my opinion the marker should contain the following informations
> - number of packets up to the next "marker block"
> - number of bytes up to the next "marker block"
> - timestamp of the first and last packet up to the next "marker block"
> - offset to the next "marker block"
> - other??

Well,   isnt that exactly what i proposed apart from some semantics.
I vote YES to the marker blocks.

The design should enforce and require one such marker block every x
number of bytes or packets. Even if writing to a stream and one can
just not fseek back and fill the fields in, the writer should still
write the marker blocks but specify all unknown fields  such as the
timestamps and the number of packets until next marker block  as 0
(==unknown, run some offline tool to massage the capture file if these
fields are important to you).

This in order to allow writing to a pipe (can not seek), the design
should also allow a writer to specify
these fields as all zero (except the bytes until next marker block).


An offline tool could then later be used to go and fill in all these
zero  timestamp, packet count fields  by just writing over the zero
fields,   WITHOUT rewriting the entire file.

But i think,  the marker blocks MUST be mandatory (or else no one will
use them) and they MUST occur at least every x number of k/M bytes.
So that one knows there WILL be marker blocks there and that one can
do things like :


  The capture is 10GByte,   show me the start stop  timestamp for all
marker blocks
  then    only load  the packets between marker block  5927  and 5984
  and not the entire 10GByte file.




On 6/27/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxx> wrote:
> In general, I prefer your solution over Ronnie's one: the basic idea of
> pcap-ng was to create not too much nesting in the file, so at the moment we
> basically have two levels, sections and blocks. In the future it's
> possible/probable that we will have some nesting, especially to support
> compressed and cyphered blocks.
> 
> The task of grouping packets in 1/10/whatever MB chunks can be achieved
> using sections (a file can contains an indefinite number of sections), and
> define new options in the section block that tell the number of packets and
> bytes (and other infos) for that section. An architectural drawback of this
> approach is that we are putting infos related to specific blocks (the packet
> blocks) in the section header, it's not so "elegant"...
> 
> As you suggested, another solution is defining a new "marker" block. However
> I'm not completely sure about its content (maybe I'm missing something in
> you mail). You suggest to write the number of packets/bytes/whatever written
> so far, I don't see a great advantage while reading packets back: you end up
> reading n packets, then a "marker" block tells you "ok, you have read n
> packets so far": I know that... Again, maybe I'm missing something...
> 
> In my opinion the marker should contain the following informations
> - number of packets up to the next "marker block"
> - number of bytes up to the next "marker block"
> - timestamp of the first and last packet up to the next "marker block"
> - offset to the next "marker block"
> - other??
> 
> Have a nice day
> GV
> 
> 
> 
> 
> ----- Original Message -----
> From: "LEGO" <luis.ontanon@xxxxxxxxx>
> To: "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx>; "Ethereal development"
> <ethereal-dev@xxxxxxxxxxxx>
> Sent: Saturday, June 25, 2005 7:03 PM
> Subject: Re: [Ethereal-dev] Re: NTAR - PCAP next generation dump file
> formatimplementation
> 
> 
> While reverse engineering tektronix rf5 file format I discovered they
> insert 4 words (4 bytes each) every 0x2000 bytes (even in the middle
> of a record).
> 
> I think these contain the number of records so far, the number of
> records in the following block and a pointer to the first whole record
> in the following block (preety much like aal2 does).
> 
> While I find simply obnoxious cutting the packet in the middle, I
> think that "marking" the file this way makes seeking much simpler and
> faster.
> 
> Whithout the need of creating a "frame"  for the "blocks" once a
> record would reach the marker one could zero pad the current block
> until the right position of the "block marker"  and continue writing
> after that.
> 
> If the block marker contains just the number of packets so far writing
> can be done without going backwards to fill the previous marker.
> 
> L
> 
> On 6/26/05, ronnie sahlberg <ronniesahlberg@xxxxxxxxx> wrote:
> > The draft describes two different blocks for holding a captured packet:
> >
> > Packet Block: it contains a single captured packet, or a portion of it.
> > Simple Packet Block: it contains a single captured packet, or a
> > portion of it, with only a minimal set of information about it.
> >
> >
> > I often work with very very large capture files and often want to only
> > extract a very small subset (packets captured between time X and time
> > Y).
> > This is very very slow with the current fileformats doe to the massive
> > amount of data that has to be processed.
> >
> >
> > I would propose to introduce a new type of block  and encapsulate both
> > PacketBlock and SimplePacketBlock in it :
> >
> >
> >
> > Packet
> > ======
> > This the block you refer to in the draft as PacketBlock. A Packet MUST
> > be encapsulated inside a PacketBlock.
> >
> >
> > SimplePacket
> > ============
> > This the block you refer to in the draft as SimplePacketBlock. A
> > SimplePacket MUST be encapsulated inside a PacketBlock.
> >
> >
> > Then introduce a new PacketBlock :
> >
> > PacketBlock
> > ===========
> > A block that contains one or (preferrably) more Packet or SimplePacket
> > structures.
> > Format of PacketBlock (in units of 4 bytes):
> >
> > Version: 1 byte,  version of packetblock structure.
> >
> > Num packets: 3 bytes,  number of Packet or SimplePacket structures in
> > this PacketBlock, or zero (==use offline tool to calculate this and
> > populate the field)
> >
> > Length : 8 bytes,  total length of this block in bytes.
> >
> > TSofFirstPacket: timestamp of the first packet or simplepacket in this
> > PacketBlock or zero (==run some offline tool to populate this field if
> > you need it)
> >
> > TSofLastPacket: timestamp of last packet or simplepacket in this
> > PaketBlock or zero.
> >
> >
> >
> > A PCAP writer should aggregate PacketBlocks/SimplePacketBlocks into
> > blocks of approximately 10MByte.
> > This default approximate block-size should be controllable by the user
> > but i think 10MByte should be reasonable.
> >
> >
> >
> > Assuming we have a 10GByte capture file, then this would make it possible
> > to :
> > 1, parse the cap file and collect all PacketBlock headers, timestamps
> > and number of packets in each block.  fseek() to the next block
> > instead of parsing it packet by packet.
> > 2, present to the user a list :
> > Block 1: start:x:y stop:x:y  packets:z
> > Block 2: ...
> > ...
> > 3, Allow the user to ONLY read/extract a subset of the PacketBlocks in
> > teh capture file.
> >
> >
> > Having this kind of encapsulation structure would allow users to use
> > large captures (and skip the portions that are of no interest) while
> > the current design would require the user to read the entire 10GB
> > file.
> >
> > 1GB and larger capture files are becoming more and more standard to
> > many users and reading through this massive amount of data is slow and
> > painful.
> >
> >
> >
> > On 6/25/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxx> wrote:
> > > Hi all.
> > >
> > > This mail is to announce the birth of the NTAR project. NTAR stands for
> > > Network Trace Archival and Retrieval library, and is an implementation
> > > of
> > > the PCAP next generation dump file format, that was proposed and
> > > discussed
> > > last year by several folks on the libpcap/tcpdump and WinPcap mailing
> > > lists.
> > > The library is released under the 3-clause/BSD license.
> > >
> > > The URL of the project is:
> > >
> > > http://www.winpcap.org/ntar
> > >
> > > On this website you can find
> > > - the source file of the library (both for windows and *nix)
> > > - the HTML documentation of the API (generated with doxygen from the
> > >   commented source files), both for the user  and for someone wanting to
> > >   extend it. The docs contain some tests/examples that you can look to
> > >   get an idea of how the library works.
> > > - An updated version of the PCAP draft specifying the file format.
> > >   The original draft of the file format is available at
> > >      http://www.tcpdump.org/pcap/pcap.html
> > >
> > >
> > >
> > > I'm seeking contributors to improve the library in terms of
> > > - testing the library on different platforms and operating systems
> > > - reviewing the API
> > > - implementing new extensions to the library.
> > >
> > > A mailing list, ntar-workers@xxxxxxxxxxx, has been created for
> > > NTAR-related
> > > discussions. People interested in this project are welcome to join it,
> > > the
> > > mailman web interface to subscribe is available at
> > >
> > > https://www.winpcap.org/mailman/listinfo/ntar-workers
> > >
> > > Have a nice day
> > > Gianluca Varenni
> > >
> > > _______________________________________________
> > > Ethereal-dev mailing list
> > > Ethereal-dev@xxxxxxxxxxxx
> > > http://www.ethereal.com/mailman/listinfo/ethereal-dev
> > >
> >
> > _______________________________________________
> > Ethereal-dev mailing list
> > Ethereal-dev@xxxxxxxxxxxx
> > http://www.ethereal.com/mailman/listinfo/ethereal-dev
> >
> 
> 
> --
> This information is top security. When you have read it, destroy yourself.
> -- Marshall McLuhan
> 
> _______________________________________________
> Ethereal-dev mailing list
> Ethereal-dev@xxxxxxxxxxxx
> http://www.ethereal.com/mailman/listinfo/ethereal-dev
> 
>