Hi LEGO.
----- Original Message -----
From: "LEGO" <luis.ontanon@xxxxxxxxx>
To: "Gianluca Varenni" <gianluca.varenni@xxxxxxxxxxxx>
Cc: "Ethereal development" <ethereal-dev@xxxxxxxxxxxx>;
<ntar-workers@xxxxxxxxxxx>
Sent: Tuesday, June 28, 2005 4:26 AM
Subject: Re: [ntar-workers] Re: [Ethereal-dev] Re: NTAR - PCAP
nextgenerationdump file formatimplementation
Comments marked with --LEGO--
On 6/28/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxxxxx> wrote:
> Comments marked with --GV--
>
> Have a nice day
> GV
>
> ----- Original Message -----
> From: "LEGO" <luis.ontanon@xxxxxxxxx>
> To: "Gianluca Varenni" <gianluca.varenni@xxxxxxxxx>
> Cc: "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx>; "Ethereal development"
> <ethereal-dev@xxxxxxxxxxxx>; "ntar-workers" <ntar-workers@xxxxxxxxxxx>
> Sent: Monday, June 27, 2005 4:21 AM
> Subject: [ntar-workers] Re: [Ethereal-dev] Re: NTAR - PCAP next
> generationdump file formatimplementation
>
>
> I was not clear at all in my last mail so I'll re-expose myself.
>
> Please note that I'll refer to "chunks" for segments of data
> containing more records and to "records" to refer to atoms. I find the
> term block confusing (in my forma-mentis a block seems something that
> contains more records).
>
> The essential point I tried to make is to have the marker-records at
> predictable locations in the file.
> That would be N*chunk_size bytes.
>
> --GV--
> That's clear.
> --GV--
>
>
> To archieve that we need:
> A padding record to fill in the space between the last packet and the
> location of the next marker record.
> And, a marker record that tell us what we have so far.
>
> --GV--
> Well, I don't see why you cannot use the sections for the same purpose.
> You
> can jump between jumps easily, and we can easily add some information
> (either in the interface description block or by means of a sort of
> marker
> block) that tells number of packets/bytes, timestamps... In my opinion,
> this
> is one of the main uses of sections. Any other opinion about that?
--LEGO--
You might use hsbs as "chunk markers" I find that more than OK. As far
as the hsb contains the offset up to the next one (if any) [see my
note about padding bellow].
Yes, it does. It contains the length of the section (or 0xFF...., if
unknown) to allow easily jumps between sections.
Quoting from the pcap-ng spec:
a.. Section Length: 64-bit value specifying the lenght in bytes of the
following section, excluding the Section Header Block itself. This field can
be used to skip the section, for faster navigation inside large files.
That means that we will not have to have a fixed chunk size for all
the file but still we could skip from one hsb to the next. As far as
enough info to know where we are and where to go its conveyed in the
hsb that's ok.
The problem, I did not consider up to now is how while skipping do I
find where the interface descriptions are?
Once we sought forwrad several chunks most probably we've had skipped
some. This needs a solution.
We could replicate all the known interface descriptions after every
HSB. I don't like it... any other ideas?
The validity of an IDB is the section; the reason is quite simple: a section
represents the minimal trace file (i.e. a file is at least formed by one
section), and a trace file can contain multiple sections, either all written
by the same tool, or possibly obtained by *physically* cat'ting multiple
trace files together (i.e. "cat trace1.ntar trace2.ntar > trace3.ntar").
Moreover the IDB is mandatory in every section (or better, at least, at
least one IDB should be present in section *before* any packet that refers
this IDB), but the same section can contain multiple IDBs.
This is documented in the spec here:
https://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#sectionidb
This makes me think: what happens if you create a section that does NOT
contain packets?
--LEGO--
> Moreover, the idea of having a padding block is troublesome: by
> definition,
> the smallest block in pcap-ng is 12 bytes. If we create a padding block
> because we want to support these markers at fixed locations, we need to
> be
> *very* careful of where to put these padding blocks (i.e. we need to
> take
> care of having either 0 bytes left before a new marker, or >= 12 bytes,
> otherwise we are screwed up).
--LEGO--
That's just a little more waste, the solution is somewhat trivial.
if (packet_len + 12 > remaining && packet_len != remaining) {
write_padding(remaining);
write_hsb();
}
write_packet();
BTW. I believe that zero padding would be a better solution.
--LEGO--
> Finally, as I said before having these markers at fixed locations is a
> real
> pain in the neck when you have a file with multiple sections *and* the
> SHB
> does not contain the length of the section.
---LEGO---
The SHB is the right place for the chunk lenght option.
It's just a matter whether it could, should or MUST be there.
I believe it must be there. Even more, I would say it's value MUST be
between 2Mb and 16Mb.
Well, it's there by spec :-).
Regarding the "could/should/must be there" problem, we do have the section
length, *but* its value can be "unknown" (0xFFF...), and this is needed in
order to support pipes (i.e. when you cannot backwards seek and write the
section length when you close a section). So I vote for "it should be there,
if we create the trace file through pipe, it's *strongly* suggested to
reprocess the trace file and write the correct section lengths in the SHBs".
Have a nice day
GV
---LEGO---
> --GV--
>
> Another point is not to have to seek backwards to fill in fields while
> dumping. Neither I like the Idea of having to keep a whole chunk in
> memory nor to have to keep more than few very essential state
> variables.
>
> --GV--
> I totally agree with you.
> --GV--
>
> I think either the file header or the HSB should tell which chunk-size
> we are using.
> The whole dump-file should use one an only one chunk-size.
>
> --GV--
> No. A file contains multiple sections (possibly coming from multiple
> cat'ted
> files). So *if* we really want to use this chunks approach, the chunk
> size
> should be saved in the SHB.
> --GV--
--LEGO--
I volunteer to write a ntar-cat that either takes the chunk_size of
the first hsb or allows the user to set it.
Uhm... what do you mean? Two trace files can be binary cat'ted, and the
result is still a valid trace file. Do you mean an utility able to
backpatch/rewrite the trace file and organize packets in sections of equal
chunk_size? This could be interesting, and *will* be easily done with the
ntar APIs (I mean file-rewriting, in-place patching is not possible at the
moment, it's in my todo list). "will" because this feature needs backwards
seeking in ntar, and currently disabled (due to the hell of having a
portable fseek working with large files); I'm working on this stuff, I hope
to have a new version of ntar on the website in the weekend.
--LEGO--
>
> On 6/27/05, Gianluca Varenni <gianluca.varenni@xxxxxxxxx> wrote:
> > In general, I prefer your solution over Ronnie's one: the basic idea
> > of
> > pcap-ng was to create not too much nesting in the file, so at the
> > moment
> > we
> > basically have two levels, sections and blocks. In the future it's
> > possible/probable that we will have some nesting, especially to
> > support
> > compressed and cyphered blocks.
>
> I like it that way. Nesting stuff would mean either to seek back to
> write the length or to keep in memory a whole meta-record before
> writing. I do not like either solution.
>
> [snip]
> > In my opinion the marker should contain the following informations
> > - number of packets up to the next "marker block"
> This is a forward reference
> > - number of bytes up to the next "marker block"
> my point is we should know this before even reading the first
> marker-record.
> > - timestamp of the first and last packet up to the next "marker block"
> I would put those before the marker record.
> > - offset to the next "marker block"
> In my scenario we know it beforehand
> > - other??
>
> _______________________________________________
> ntar-workers mailing list
> ntar-workers@xxxxxxxxxxx
> https://www.winpcap.org/mailman/listinfo/ntar-workers
>
>
--
This information is top security. When you have read it, destroy yourself.
-- Marshall McLuhan
_______________________________________________
Ethereal-dev mailing list
Ethereal-dev@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-dev
