Hi!
How do users climbing the steep path of deep packet inspection extract files,
in HTTP/HTTPS protocols, i.e. the streams in SSL (and plain TCP) conversations?
Is there a program that can extract files from SSL- or plain- TCP streams
automatically?
I've long wished to gain that knowledge/that knowhow, since it's absolutely
likely the big surveillors have their tools to do that, and do that very
comfortably... Just, the public at large are not allowed nor access to nor the
knowledge about those programs... No proofs that it is so, but it's so stinking
likely that it is so.
And asked about extr here on this list:
https://www.wireshark.org/lists/wireshark-users/201604/msg00002.html
E.g.:
> Are there such scripts that could take a stream, and extract all the files
> from it? [...]
And I was recommended Chaosreader.
But Chaosreader does not, well at least currently (who knows about the future),
decrypt SSL and so can't extract from SSL streams, only from plain HTTP.
And so I've decided to spend some serious time debugging Chaosreader my own way
to figure out how it works, and learned a lot of Perl as I went.
And I've managed to put together a script that uses a few modified
subroutines from Chaosreader on already decrypted SSL TCP streams and extracts
files from them.
Judge for yourself how successful it is.
Here's the script, or the primitive program if I should call it:
https://github.com/miroR/stream-cont
In the script chread_tcp.pl in the program I explained how i debugged
Chaosreader, the commits can be followed from Chaosreader itself which is the
initial commit, to the final chread_tcp.pl version and creation of
stream-cont.pl itself.
And here's a hands on tutorial on the whole process needed to work out a PCAP
(or many PCAPs in the same session):
https://www.croatiafidelis.hr/foss/cap/cap-180505-schmoog-referendum/
I was able to extract all files automatically, so no right-mouse clicking,
chosing, selecting et cetera, no GUI stuff, repeated each time say, for every
single stream, or...
Or firing hexedit and manually navigating through the GET and HTTP headers, and
cutting and pasting and stuff...
And that is not the case just on those samples from that tutorial, but on most
other samples from a range of internet connections to different places.
stream-cont only rarely doesn't accomplish extractions properly, and I'm yet to
(some day, not immediately in the world, see below) figure out the details and
reasons of failures.
And it does all its extractions automatically, no clicking, no cutting and
pasting...
It's not completed. I want it to extract POST as well, and do other things...
But I just dropped dead tired once I started getting first great results that I
could only dream of till then... And still wouldn't be able to work on it not
even now and not for at least a few more days...
I hope other users struggling with similar issues will find my stream-cont
program useful.
So I decided to present my stream-cont to this list. Feedback welcome!
Regards!
--
Miroslav Rovis
Zagreb, Croatia
https://www.CroatiaFidelis.hr
Attachment:
signature.asc
Description: PGP signature
