Wireshark · Wireshark-dev: Re: [Wireshark-dev] hf_http_response_code in packet-http.c

["Sultan, Hassan" <sultah@xxxxxxxxxx>]

> -----Original Message-----
> From: Pascal Quantin [mailto:pascal.quantin@xxxxxxxxx]
> Sent: Thursday, July 13, 2017 1:16 PM
> To: Developer support list for Wireshark <wireshark-dev@xxxxxxxxxxxxx>
> Cc: Sultan, Hassan <sultah@xxxxxxxxxx>
> Subject: Re: [Wireshark-dev] hf_http_response_code in packet-http.c
> 
> 
> 
> Le 13 juil. 2017 22:00, "Sultan, Hassan via Wireshark-dev" <wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > a écrit :
> 
> 
> 
> 
> 	> -----Original Message-----
> 	> From: Wireshark-dev [mailto:wireshark-dev-bounces@xxxxxxxxxxxxx
> <mailto:wireshark-dev-bounces@xxxxxxxxxxxxx> ] On Behalf
> 	> Of Erik de Jong
> 
> 	> Sent: Thursday, July 13, 2017 11:58 AM
> 	> To: Developer support list for Wireshark <wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> >
> 	> Subject: Re: [Wireshark-dev] hf_http_response_code in packet-http.c
> 	>
> 	>
> 	>
> 
> 	> On Thu, Jul 13, 2017 at 8:47 PM, Sultan, Hassan via Wireshark-dev
> <wireshark-
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > > wrote:
> 	>
> 	>
> 	>
> 	>
> 	>       > -----Original Message-----
> 	>       > From: Pascal Quantin [mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx>
> 	> <mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx> > ]
> 
> 	>       > Sent: Thursday, July 13, 2017 11:26 AM
> 	>       > To: Sultan, Hassan <sultah@xxxxxxxxxx
> <mailto:sultah@xxxxxxxxxx>
> 
> 	> <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx> > >
> 	>       > Cc: Developer support list for Wireshark <wireshark-
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > >
> 
> 	>       > Subject: RE: [Wireshark-dev] hf_http_response_code in packet-
> http.c
> 	>       >
> 	>       >
> 	>       >
> 	>       > Le 13 juil. 2017 19:54, "Sultan, Hassan" <sultah@xxxxxxxxxx
> <mailto:sultah@xxxxxxxxxx>
> 	> <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx> >
> 
> 	>       > <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx>
> <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx> > > > a
> 
> 	> écrit :
> 	>       >
> 	>       >
> 	>       >
> 	>       >
> 	>       >       > -----Original Message-----
> 	>       >       > From: Pascal Quantin [mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx>
> 	> <mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx> >
> 	>       > <mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx>
> 	> <mailto:pascal.quantin@xxxxxxxxx
> <mailto:pascal.quantin@xxxxxxxxx> > > ]
> 	>       >       > Sent: Thursday, July 13, 2017 10:39 AM
> 	>       >       > To: Developer support list for Wireshark <wireshark-
> 
> 	>       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >  <mailto:wireshark-
> <mailto:wireshark->
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > > >
> 	>       >
> 
> 	>       >       > Cc: Sultan, Hassan <sultah@xxxxxxxxxx
> <mailto:sultah@xxxxxxxxxx>
> 	> <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx> >
> 
> 	>       > <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx>
> <mailto:sultah@xxxxxxxxxx <mailto:sultah@xxxxxxxxxx> > > >
> 
> 	>       >       > Subject: Re: [Wireshark-dev] hf_http_response_code in
> packet-
> 	> http.c
> 	>       >       >
> 	>       >
> 	>       >       > Hi Hassan,
> 	>       >       >
> 	>       >       >
> 	>       >       > 2017-07-13 19:25 GMT+02:00 Sultan, Hassan via Wireshark-
> dev
> 	>       > <wireshark-
> 	>       >
> 	>       >       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >
> 
> 	> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> > >
> <mailto:wireshark- <mailto:wireshark->
> 	> <mailto:wireshark- <mailto:wireshark-> >
> 	>       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >  <mailto:wireshark-
> <mailto:wireshark->
> 
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > > > >:
> 	>       >
> 	>       >       >
> 	>       >       >
> 	>       >       >
> 	>       >       >
> 	>       >       >       > -----Original Message-----
> 	>       >       >       > From: Wireshark-dev [mailto:wireshark-dev-
> <mailto:wireshark-dev->
> 
> 	> <mailto:wireshark-dev- <mailto:wireshark-dev-> >
> 	>       > bounces@xxxxxxxxxxxxx <mailto:bounces@xxxxxxxxxxxxx>
> <mailto:bounces@xxxxxxxxxxxxx <mailto:bounces@xxxxxxxxxxxxx> >
> 	> <mailto:wireshark-dev-bounces@xxxxxxxxxxxxx <mailto:wireshark-
> dev-bounces@xxxxxxxxxxxxx>  <mailto:wireshark-dev- <mailto:wireshark-dev->
> 	> bounces@xxxxxxxxxxxxx <mailto:bounces@xxxxxxxxxxxxx> > >
> 	>       >       > <mailto:wireshark-dev-bounces@xxxxxxxxxxxxx
> <mailto:wireshark-dev-bounces@xxxxxxxxxxxxx>
> 	> <mailto:wireshark-dev-bounces@xxxxxxxxxxxxx <mailto:wireshark-
> dev-bounces@xxxxxxxxxxxxx> >  <mailto:wireshark- <mailto:wireshark->
> 
> 	> <mailto:wireshark- <mailto:wireshark-> >
> 	>       > dev-bounces@xxxxxxxxxxxxx <mailto:dev-
> bounces@xxxxxxxxxxxxx>  <mailto:dev-bounces@xxxxxxxxxxxxx <mailto:dev-
> bounces@xxxxxxxxxxxxx> > >
> 	> > ] On Behalf
> 	>       >       >       > Of Erik de Jong
> 	>       >       >       > Sent: Wednesday, July 12, 2017 10:12 PM
> 	>       >       >       > To: Developer support list for Wireshark <wireshark-
> 	>       >
> 	>       >       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >
> 
> 	> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> > >
> <mailto:wireshark- <mailto:wireshark->
> 	> <mailto:wireshark- <mailto:wireshark-> >
> 	>       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >  <mailto:wireshark-
> <mailto:wireshark->
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > > > >
> 	>       >       >       > Subject: Re: [Wireshark-dev] hf_http_response_code in
> 	> packet-
> 	>       > http.c
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       > On Thu, Jul 13, 2017 at 1:12 AM, Sultan, Hassan via
> 	> Wireshark-
> 	>       > dev
> 	>       >       > <wireshark-
> 	>       >
> 	>       >       >       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >
> 	> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> > >
> 
> 	>       > <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >
> 
> 	> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> > > >
> 	> <mailto:wireshark- <mailto:wireshark->  <mailto:wireshark-
> <mailto:wireshark-> >
> 	>       > <mailto:wireshark- <mailto:wireshark->  <mailto:wireshark-
> <mailto:wireshark-> > >
> 	>
> 	>       >
> 	>       >       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >
> 	> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> > >
> <mailto:wireshark- <mailto:wireshark->
> 	> <mailto:wireshark- <mailto:wireshark-> >
> 	>       > dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>
> <mailto:dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx> >  <mailto:wireshark-
> <mailto:wireshark->
> 	> dev@xxxxxxxxxxxxx <mailto:dev@xxxxxxxxxxxxx>  <mailto:wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx> > > > > > wrote:
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >       Hi,
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >       I am starting to learn the Wireshark code base, and
> one
> 	> thing
> 	>       >       > puzzles
> 	>       >       >       > me…
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >       Why is hf_http_response_code defined as a
> FT_UINT16
> 	> with
> 	>       >       > BASE_DEC
> 	>       >       >       > rather than an FT_STRING ?
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       >       It’s a text field… not an integer.
> 	>       >       >       >
> 	>       >       >       >
> 	>       >       >       > Presenting it as a number allows for filtering like:
> 	>       >       >       > http.response.code > 200
> 	>       >       >       >
> 	>       >       >       > Which would not be possible when presented as a
> string.
> 	>       >       >
> 	>       >       >       Thanks for the info, but in that case would it not be more
> 	>       > appropriate to
> 	>       >       > have the normal field as an FT_STRING and add a generated
> field
> 	> as
> 	>       > FT_UINT16
> 	>       >       > ? My understanding of generated fields is that this is their
> 	> purpose :
> 	>       > represent
> 	>       >       > data that doesn't exactly correspond to the packet data.
> 	>       >       >       We could still keep the field named as is today (hence
> 	> ensuring all
> 	>       >       > existing filters still work), but simply make it a generated
> field,
> 	> and add
> 	>       > an
> 	>       >       > FT_STRING to represent the actual data as it is in the packet.
> 	>       >       >
> 	>       >       >       Thoughts ?
> 	>       >       >
> 	>       >       >
> 	>       >       >
> 	>       >       >
> 	>       >       > Does having the field as it is designed today generating you
> any
> 	> issue?
> 	>       > For fields
> 	>       >       > having numerical values only, it makes much more sense to
> have
> 	> the
> 	>       > digits
> 	>       >       > directly instead of the string and that's already what we are
> doing
> 	> in
> 	>       > numerous
> 	>       >       > places. And personally I do not see any valid reason to
> change
> 	> that.
> 	>       > Your
> 	>       >       > suggestion implies that many fields would start to be
> duplicated
> 	>       > without any
> 	>       >       > added value.
> 	>       >
> 	>       >
> 	>       >       My reason is that anyone wanting to build automation based
> on
> 	>       > Wireshark's parsing will not be able to rely on Wireshark's
> reporting of
> 	> field type
> 	>       > & format to interpret the packet data.
> 	>       >       The automation would walk through parsed fields and be told
> that
> 	> the
> 	>       > http response code is a 3-bytes UINT16 with value say 200, so
> bytes 00
> 	> 00 c8,
> 	>       > which is wrong. Right now the way these fields are setup works
> for
> 	> human
> 	>       > interpretation, by people looking at the rendered value on a
> screen,
> 	> but it is far
> 	>       > from ideal for automation.
> 	>       >
> 	>       >
> 	>       > Thanks fact that HTTP is text based does not change the fact that
> the
> 	> code is
> 	>       > supposed to be a number. So I'm not sure to get your point.
> 	>       > Moreover tons of people use tshark for automation and I do not
> 	> remember
> 	>       > someone complaining about the current output. So maybe I
> simply did
> 	> not
> 	>       > understand your use case.
> 	>       >
> 	>       >
> 	>       >       One could argue it's the main purpose of Wireshark, but I
> think it
> 	>       > needlessly prevent a whole class of use-cases from being
> achieved
> 	> with
> 	>       > Wireshark's technology.
> 	>       >
> 	>       >
> 	>       >
> 	>       > Could you elaborate? What use case do you have in mind
> exactly?
> 	>
> 	>
> 	>       Imagine automation that attempts to perform modification on
> packets
> 	> using the parsed information provided by Wireshark, or somehow
> goes and
> 	> directly reads fields in the packet data based on what Wireshark's
> parsed
> 	> information says (type/offset/length/...). Wireshark reporting the field
> as
> 	> FT_UINT16 instead of say a UTF-8 FT_STRING throws off the
> automation as it
> 	> would either attempt to write an FT_UINT16 at the offset of the field,
> writing
> 	> the wrong format, or be confused by seeing a 3 bytes FT_UINT16 and
> give up.
> 	>
> 	>
> 	>
> 	>
> 	> But in that case it's more like you're doing a part of the parsing in
> another
> 	> application, why not implement a full parser in that application? The
> whole point
> 	> of an analyzer is to make sense of the data passed into it for the user.
> And what
> 	> about protocols that could be send with different encodings (BER,
> XER, etc)
> 	> there a field type should still be the same - eg UINT16 - but the
> representation
> 	> on the wire is totally different.
> 
> 
> 	The beauty of this is specifically that the other app does not need to do
> any parsing at all. It fully relies on Wireshark's parsing engine to get the tree of
> fields, the offset/length/type/name/encoding/... of each field and instead of
> having to write custom code for each protocol (you know, a parser), the app
> simply has to implement generic handling for FT_STRING, FT_UINT16, etc... and
> the various encodings. So for your BER encoded integer, having an FT_UINT16
> with an ENC_BER encoding flag for example and a size of 4 (type+length+2 bytes
> for most short ints) would convey all the info needed to handle the field.
> 
> 
> 
> That would require having an encoder able to understand Wireshark output
> syntax and the structure of the protocol you are using to perform the encoding
> (text based, TLV, CSN1, ASN1, etc): all the encoding rules related to the protocol
> you are playing with. Having a decoder / parser is far from giving you an
> encoder.

Absolutely, but this is a lot, lot, less work than writing parsers for all the protocols. That's where Wireshark shines, it provides 70-80% of the work (all the parsing), and only the handling of encoding / shifting fields (in case user decides to write a field to be larger/smaller than originally in the packet) needs to be done.


> As Eric explained, having the HTTP response code as a number simply gives you
> much more advantages (filtering, comparisons, inequalities...) than having it in
> text. So this is clearly not a bug as it was done on purpose.

I agree it's not a bug. For the existing purpose of Wireshark it does the job. It just makes some other use-cases a bit more difficult.
 
> 	Will such a thing work for absolutely every single protocol out there ?
> Maybe not, but it works for the vast majority of them.
> 
> 
> I have some doubts regarding this statement, for the reason I gave above.
> But as I'm eager to learn, if you can come up with a PoC I would be very
> interested to look at it :)
 
It might be possible to show something but it will take a bit of time and is a maybe. I'm still in the learning stages of how Wireshark works and there's quite a bit of work that will be needed before I can confirm if it's doable with Wireshark or what changes are needed in Wireshark to make it work (the example I just gave being one).
 
> 	I did a very similar thing on top of Network Monitor in a previous life at
> Microsoft (I had the code available being internal to the company) and it was a
> huge success, helping many people in various ways.
> 
> 
> 
> So what kind of encoding did it support?

Gzip compression, asn.1 BER & DER, utf-16/8/ucs2/ascii, little/big endian, base64, custom encodings for netbios strings, and more that I forget, it also had inference for finer grained datatypes (figure out that this string field is actually a url, domain name, numerical text, email, xml, json, ... instead of text and it did so in a recursive manner so that when finding xml for example, attributes in xml could be inferred as url, numerical text, domain name, etc...). It could also infer which fields were length/count fields for other fields (so it could update them automatically upon changes) or, in certain circumstances, checksum fields. Whenever users on a protocol encountered an encoding we didn't support, we'd add the encoder for it. Much simpler and quicker task than adding a protocol parser.
 
> 
> 
> 	>       How about having an #ifdef for such things so that Wireshark's
> use-case
> 	> today does not have to suffer the addition of generated fields, and
> such
> 	> automation cases could still get full fidelity parsing by building
> Wireshark's
> 	> engine with the specific #define ? Today they have to build directly
> against
> 	> Wireshark's code anyway so that shouldn't be a problem.
> 	>
> 	>
> 	>
> __________________________________________________________
> 	> _________________
> 	>       Sent via:    Wireshark-dev mailing list <wireshark-
> dev@xxxxxxxxxxxxx <mailto:wireshark-dev@xxxxxxxxxxxxx>
> 
> 	> <mailto:wireshark-dev@xxxxxxxxxxxxx <mailto:wireshark-
> dev@xxxxxxxxxxxxx> > >
> 	>       Archives:    https://www.wireshark.org/lists/wireshark-dev
> <https://www.wireshark.org/lists/wireshark-dev>
> 	> <https://www.wireshark.org/lists/wireshark-dev
> <https://www.wireshark.org/lists/wireshark-dev> >
> 	>       Unsubscribe:
> https://www.wireshark.org/mailman/options/wireshark-
> <https://www.wireshark.org/mailman/options/wireshark->
> 	> dev <https://www.wireshark.org/mailman/options/wireshark-dev
> <https://www.wireshark.org/mailman/options/wireshark-dev> >
> 	>                    mailto:wireshark-dev-request@xxxxxxxxxxxxx
> <mailto:wireshark-dev-request@xxxxxxxxxxxxx>  <mailto:wireshark-
> <mailto:wireshark->
> 	> dev-request@xxxxxxxxxxxxx <mailto:dev-request@xxxxxxxxxxxxx> >
> ?subject=unsubscribe
> 	>
> 
> 
> 	__________________________________________________________
> _________________
> 	Sent via:    Wireshark-dev mailing list <wireshark-dev@xxxxxxxxxxxxx
> <mailto:wireshark-dev@xxxxxxxxxxxxx> >
> 	Archives:    https://www.wireshark.org/lists/wireshark-dev
> <https://www.wireshark.org/lists/wireshark-dev>
> 	Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-
> dev <https://www.wireshark.org/mailman/options/wireshark-dev>
> 	             mailto:wireshark-dev-request@xxxxxxxxxxxxx <mailto:wireshark-
> dev-request@xxxxxxxxxxxxx> ?subject=unsubscribe
>