Wireshark · Wireshark-users: Re: [Wireshark-users] Analyzing a "broken" FTP session

[Martin Visser <martinvisser99@xxxxxxxxx>]

There is definitely something interesting going on.

As you possibly know, for FTP, there is always two sessions between client and server. The control session is opened by the client on TCP port 21. Then for every subsequent data transfer another data session is required. The control session is always long-lived.

In your case a new data session is opened for each transfers (you are using Passive, PASV, mode). We see 4 STOR requests, and hence 4 new TCP sessions for each data transfer are established.

The first 3 all go through without incident. However on the 4th some trouble appears. What seems to happen is that the data transfer is finished, but the command channel still had not completed the transaction. It seems that the server has got confused.

Looking at made_on_server.pcap:-

51: Client issues STOR request for CAMBRID06_QT.CFLT.pgp
52: Server responds, asking the client to open a BINARY data channel
53-57 The transfer happens on server TCP port 47818 
58: The server sends another copy of the  of the server open BINARY data channel response that it sent in 52. Now the client did ACK the server response sent in frame 52 in good time (you can see this on the made_on_client.pcap), but the server hasn't seen this yet and got impatient (a 200ms timer went off). (Unfortunately your packet captures are truncated, so we assume it is the same data in the response)
59: The late ACK from the client arrives
60: The server tells the client that data connection is closed (presumably for CAMBRID06_QT.CFLT.pgp)
61: Client ACKs again Frame 52 and 58
62: Clients ACKs frame 60
63: Client requests a new PASV connection (for next STOR presumably)
64-65: The server ACKs packet 59 again
66: The server repeats the data connection closed response of 60 (even though it was already ACKed by the client)
67: The client ACKs again
68: The client retransmits the PASV request of 63
69: The server only ACKs to 62
70: The client retransmits the PASV request of 63
71: The server only ACKs to 62
and-so-on
Retransmission timers get longer and longer.
88: Keep-alives start kicking in 
89: Client responds to keep-alive
...
109: Server kills FTP session (with RST)

From what I can see, it seems that the server has got into a state where it won't accept a valid PASV request. I have a feeling it is because of the the fact that the data transfer in 53-57 is finished before it even thinks it should have started. In the earlier transfers the ACK from the client does come after the transfer is finiished but within the retransmission timer. 

The FTP protocol (and the implementations) should be able to handle this sort of scenario, so I am not quite sure what is going on there.

(Though one thing is slightly curious the packet sent by the server in frame 58 (the duplicate open BINARY channel command) never reaches the client. I'm not sure whether this is significant in things getting into the wrong state.)

It would be interesting to see whether this sort of sequence (with retransmissions) is occuring everytime your FTP session locks up.

Can I suggest for future captures you use a longer tcpdump snap length. The default is 96 bytes which means we don't see the full FTP command. "-s 0" would be nice, but at least "-s 256" would be OK.

Regards, Martin








Regards, Martin

MartinVisser99@xxxxxxxxx

On Sat, Aug 22, 2009 at 3:28 AM, Chivian, John <jchivian@xxxxxxxxxxxxxx> wrote:

Thanks Martin.   The clipped packet captures are attached.

Client info:
$ # the command line FTP client was used
$ uname -a
Linux hostname-masked 2.6.9-55.0.12.ELsmp #1 SMP Fri Nov 2 12:38:56 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 4.7 (Final)

Server info:
$ # a commercial FTP server daemon
$ uname -a
Linux hostname-masked 2.6.9-89.0.7.ELsmp #1 SMP Wed Aug 5 14:12:18 EDT 2009 i686 athlon i386 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux ES release 4 (Nahant Update 8)

Network information:
The systems are both operating at 100 Mb/s.
They are both in the same physical location.
Client <-> Switch <-> Router <-> Switch <-> Server
The problem is generally seen with FTP sessions involving hundreds of small files.


I understand that the issue may be network as opposed to server related, and I understand that the packet captures may not contain enough information to make a definitive judgment.

Thanks all for your help!

JC




From: wireshark-users-bounces@xxxxxxxxxxxxx [mailto:wireshark-users-bounces@xxxxxxxxxxxxx] On Behalf Of Martin Visser
Sent: Friday, August 21, 2009 2:49 AM
To: Community support list for Wireshark
Subject: Re: [Wireshark-users] Analyzing a "broken" FTP session

Posting snippets of packet captures with reasonable problem reports are generally welcome. A lot of us here enjoy the challenge of trying interpret a stream of bytes with the hope of actually diagnosing the higher level issue that may be causing the problem.

That said, some of us do this work for a living and get paid quite a bit of money to do it, so doing it for free is only viable if it doesn't take a lot of time. Any time I respond to such requests is out of a spirit of mentoring the community and also hopefully giving myself a little confidence that I actually sometimes know what I am talking about :-)

So please post, but also understand the context of the community.

Regards, Martin

MartinVisser99@xxxxxxxxx

On Fri, Aug 21, 2009 at 12:36 PM, Chivian, John <jchivian@xxxxxxxxxxxxxx> wrote:
Group:

  I'm not sure if this is the correct forum for this but I am hoping to get some help identifying a problem that sometimes occurs between an FTP client and server.   (If this isn't the right forum can someone point me in the right direction?)

  I have PCAP files made on both systems using tcpdump that have captured a recent failure, but I do not have enough expertise in packet analysis or the guts of the FTP protocol to read them and draw a definitive conclusion regarding why the connection "broke".

  If someone can help I am happy to provide more information regarding the systems themselves, the network topology between them, and the trimmed PCAP files for analysis.

  Thanks in advance, JC

---
 
John (JC) Chivian
Staff Software Engineer
Staff Unix/Linux Administrator
Corporate Information Systems
Photronics, Inc.
 
mailto:jchivian@xxxxxxxxxxxxxx
http://www.photronics.com
 
---
 
This e-mail and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain Photronics' confidential information. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, be advised that you have received this e-mail in error and that any use, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited.
 
---
 
Environmentalism is an ethic and a way of life.  Pass it on!


___________________________________________________________________________
Sent via:    Wireshark-users mailing list <wireshark-users@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-users
Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
            mailto:wireshark-users-request@xxxxxxxxxxxxx?subject=unsubscribe

___________________________________________________________________________
Sent via:    Wireshark-users mailing list <wireshark-users@xxxxxxxxxxxxx>
Archives:    http://www.wireshark.org/lists/wireshark-users
Unsubscribe: https://wireshark.org/mailman/options/wireshark-users
            mailto:wireshark-users-request@xxxxxxxxxxxxx?subject=unsubscribe