Well, it turns out that the customer had 4 lucent switches (SRC adn DST were
matched to MACs)that are somehow screwed up and do broadcast at 1 sec
intervals. Also we found some strange worm/trojan/virus that was doing ARPs
at 2-3 sec in 52 windows xp machines but the funny thing is that they kept
asking each other way too fast.
example:
windows xp(139.60.1.5)-->Who has 139.60.1.7
windowx xp(139.60.1.7)--> response to 139.60.1.5
windows xp(139.60.1.7)-->Who has 139.60.1.5
windowx xp(139.60.1.5)--> response to 139.60.1.7
and this was going and going and going EVERY second for all the 52
computers. This definitely is not normal and we found some weird program
listening(??) on port 9876 on every machine. Maybe a bug in this
worm/trojan/whatever caused it to arp without learning the response.
since the customer has no good policy about internet usage, the users are
able to donwload all they want. kazaa, overnet, edonkey, spywares,adwares,
etc were found to be running (im not blaming the p2p software, maybe
something sneaked in using the p2p connection).
the solution, since we are *not* AV people, was to do a clean install on all
machines and getr rid of all the garbage. One of the machines was even found
to be a open smtp relay (by means of a RAT) receiving a smtp message by
listening in tcp 2525 as well as a mailing list, and then sending it to
thousands of users.
So at the end, ARP was not the only culprit. But thanks to ethereal we
quickly found what is was.
They had a PIX firewall but it was completely open (just doing NATing) and
the logs showed management connections from the outside as well as forwarded
ports.
what about the netadmins? what for? this is a self-healing 200 computer
network... ;)
Firewall...what firewall? hehe.
thanks to all and thanks to ethereal.
Erick
-----Original Message-----
From: Wes [mailto:wes_r@xxxxxxxxx]
Sent: Thursday, May 27, 2004 8:43 AM
To: Ethereal user support
Subject: Re: [Ethereal-users] 78 percent of ARP packets on the network
Do you have a lot of switches in the network?
The problem I've seen is you will see all the other non broadcast traffic on
the switch you are connected to (with the port set to replicate traffic),
but will only see broadcasts from all the other switches in the network.
This can make it look like an ARP storm. The fact that it is 78 percent ARP
traffic may simply be because there isn't much Unicast traffic on the switch
you are monitoring compared to the broadcasts you are getting from the
entire company.
The real question is what is the Packet per second rate of the ARP traffic
and are they ARPs for known hosts or simply something looking for unknown
hosts to respond.
Wes
--- eperez@xxxxxxxxxxx wrote:
> Well, the network is a 139.60.0.0/255.255.0.0 doing natting to the
> outside via PIX that NATs to 64.116.x.x. The network has around 200
> machines.
> Yes, I know the 139.x.x.x is wrong but somehow they decided that was a
> good network (why do they want 65536 hosts in unknown to
> me..) But since they are
> doing NATting I see no problem related to this ARP storm. The net
> numbering can be fixed later.....
>
> A few minutes ago we also discover IPX traffic.
> About 10%. So they have a case
> of Lame sysadmins that do click-click-clik Windows installations.
>
> It has several jetdirect devices that are know to do broadcasts (will
> be checked and disabled if needed) but the devices are like a year old
> and the problem just started a few days ago.
>
> As per the validity of the ARP SRC and DST, I will check that tomorrow
> and do a repost to this list.The ARP list is so hughe and it was
> already closing time down here (gmt -5) that we were not able to
> verify it onsite.
>
> It cannot be nachy/welchia worm because thats for Win2k/XP and they
> have a lot of 95/98/NT machines. Only a few (less than 20) are XP Pro.
> Unless of course somehow they got a way to infect those older OSs.
>
> David: port mirroring is working fine.
> Peter: Ill check SRC and DST tomorrow and do a repost.
> Andrew: Well, Ill ditch MS technologies from my entire country if i
> could but i cant in this case. LONG LIVE *nix
> Brett: We also use static IPs. I will check for viruses using my
> personal laptop since I dont trust any of the customer's computer.
>
> Thanks to all, I will repost tomorrow.
>
> Erick.
>
> Quoting eperez@xxxxxxxxxxx:
>
> > My network started to slow down a few days ago. So
> I installed latest
> > ethereal
> > and winpcap for windows in a NT Server 4.0. All
> the network is switched and
> > I
> > was trying to find some cause of slowdown. I am
> aware of the limitations of
> > sniffing on a switched network so I set the
> switches to replicate traffic so
> > i
> > can see it with ethereal.
> > So far so good, but in the main ethereal windows
> where it shows how many
> > packets
> > per protocol has received during the sniffing
> session I found that after 1
> > hour
> > of sniffing 78% of my traffic was ARP and the rest
> was TCP(normal smb, tns,
> > etc).
> >
> > All the network has windows machines
> (95,98,NT,2000,XP) all servers are NT
> > 4.0
> > and the network has one PDC one BDC and one WINS
> server.
> >
> > I did a search on the mailing list but found no
> clue about it. Maybe this is
> > normal but I just dont know.
> >
> > Comments/Flames/Suggestions are welcomed.
> >
> > Erick.
> >
> >
> > _______________________________________________
> > Ethereal-users mailing list
> > Ethereal-users@xxxxxxxxxxxx
> >
>
http://www.ethereal.com/mailman/listinfo/ethereal-users
> >
>
>
>
> _______________________________________________
> Ethereal-users mailing list
> Ethereal-users@xxxxxxxxxxxx
>
http://www.ethereal.com/mailman/listinfo/ethereal-users
__________________________________
Do you Yahoo!?
Friends. Fun. Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/
_______________________________________________
Ethereal-users mailing list
Ethereal-users@xxxxxxxxxxxx
http://www.ethereal.com/mailman/listinfo/ethereal-users
