Basic Network Application Troubleshooting With Wireshark (Ethereal)

Network protocol analysis is a technique used to view, in real time, the raw data sent and received over a network interface. This is useful for troubleshooting network configuration and network application problems. It is also useful when developing new network protocols. In this article, I’ll go over the very basics of troubleshooting a network application with Wireshark (Ethereal).

Wireshark Overview

Ethereal is a open source protocol analyzer initially written by Gerald Combs. It was renamed Wireshark in 2006 because of trademarks'copyrights held on the name Ethereal. Wireshark is now maintained and enhanced by hundreds of people worldwide.

What can it do?

Several years ago I worked on a compliance project to archive all instant messaging (IM) traffic flowing through the company network for the Yahoo! IM, MSN and AOL IM networks. I was a member of the Messaging and Groupware team which was responsible for (among other things) email, IM and other collaborative software systems.
M&G, as the team was called, was not responsible for the network, dns servers and certainly not the firewall. We had to communicate our intentions and coordinate our efforts with the groups responsible for these other systems in order to complete the project. Here is a short (read incomplete) list of what needed to happen - or so we thought.

1. DNS servers needed to report the address of our IM archive server. The archive server works like a web proxy server and will open connections to the real IM server on behalf of the clients.
2. The firewall needed to allow the IM archive server to open connections to the internet on specific ports to a specific server.
3. IM clients needed to be reconfigured to “connect directly to the internet” as opposed to using a socks proxy or http tunneling. The IM client would send a query to resolve a DNS name of an IM server (for example login.oscar.aol.com) to an IP address. The company’s internal DNS servers would return the address of our IM proxy which would then open a connection to the real IM server out on the internet. And our proxy would log every conversation detail along the way. Big Brother would be alive and well.

After all of the details were worked out and implemented, it was time to test. But when we flipped the switch, it did not work! I queried our DNS server to check that the proper DNS A records were created, and they were. I checked that I could connect to the IM proxy server on the ports used by each of my IM clients, and I could. I then checked to see if the proxy server could connect to the internet IM servers on the required ports, and I could not.
So obviously the problem was with the firewall, right? I called up the firewall team and complained to them about how they had not opened up the ports I’d requested. However, they swore they had opened up all reports to the specified servers I’d requested. So what was happening?
Well, that’s when the product support guy for IM Manager (the IM proxy) suggested we use Ethereal to “see” what was going on. To shorten a long story, with Ethereal we were able to determine that the proxy server could not open a connection to the public IM server since the firewall rules in place only allowed it to open connection to the internet for a specific host but on any port. The hosts we’d given our firewall team were really a sort of traffic cop. It would instruct the IM client to connect to another machine. Since this host was not in our firewall rule set, the connection was not allowed. In the end we requested that the proxy servers be allowed to open connections to any host on the required ports. The required changes were made to the firewall rule sets and everything worked as expected.
This is a classic example of troubleshooting with Wireshark.

What can’t it do?

Just as with any tool, Wireshark can be used for some things and not others. Here is a list of some of the things Wireshark cannot do:

1. It cannot be used to map out a network. Take a look at the NMAP tool for that functionality.
2. It does not generate network data – it is a passive tool. Tools like NMAP, ping, and traceroute are examples of tools that generate network data. These tools are active.
3. It can only show detailed information about protocols it actually understands. The good news is that it understands a great many protocols. It is also extensible, so you can add protocol support for ones it doesn’t understand. Otherwise you will only be able to see a hexdump of data it has captured.
4. It can only capture data as well as the OS'Interface'Interface driver supports. An example of this is capturing data over wireless networks. This does not work well (or at all) for some software and hardware combinations.

Installing Wireshark

Wireshark is an open source application and may be downloaded for free fromwww.wireshark.org. Installation is straight forward. To install on Windows using the executable package:

1. Double click the installer file.
2. Click the ‘Next’ button at the Welcome screen. 
3. Click the ‘I Agree’ button to accept the licensing terms. 
4. Click the ‘Next’ button to accept the defaults at the Choose Components dialog box. 
5. Click the ‘Next’ button at the Select Additional Tasks dialog box. 
6. Click the ‘Next’ button at the Choose Install Location dialog box. 
7. At this point, the installer will ask if you want to install WinPcap. Ensure that the Install Winpcap checkbox is selected and click the ‘Next’ button. 
8. The Wireshark installation will now begin copying files to your system. 
9. The WinPcap installer will launch during Wireshark installation. Click the ‘Next’ button at the Welcome screen. 
10. Click the ‘Next’ button at the WinPcap Setup Wizard screen. 
11. Click the ‘I Agree’ at the License Agreement screen. 
12. Click the ‘Finish’ button to close the WinPcap installer. 
13. Click the ‘Next’ button on the Wireshark Installation Complete dialog box. 
14. Click the ‘Finish’ button to close the Wireshark installer.

Running Wireshark on Windows

Launching Wireshark

Running Wireshark on Windows is a simple matter of double clicking the shortcut on the start menu. This will open the Wireshark main screen.
    

Wireshark Interface

The Wireshark interface is fairly simple considering what it can do.

1. Title bar – this will contain different information depending on what Wireshark is doing. If it is capturing network data, it will show the interface that is in use. If it is displaying data from a previous capture, the name of the file containing the captured data will be shown (untitled is shown if a capture was performed, stopped and not saved). Otherwise it will show the application name: Wireshark Network Protocol Analyzer
2. Menu bar – Menu bar providing access to application features
   1. File – Functions for working with captured data such as saving and exporting to different file formats
   2. Edit – Functions for finding packets, setting the time reference, and setting preferences
   3. View - Functions for modifying how Wireshark displays information such as which windows are open
   4. Go – Functions for navigating to specific packets
   5. Capture – Functions for starting and stopping captures, saving filters and working with network interfaces
   6. Analyze – Functions for interpreting and filtering captured data
   7. Statistics – Functions to statistically analyze captured data
   8. Help – access to product help
3. Main tool bar – Shortcuts to frequently used functions in the menu bar
4. Filter tool bar – Quick access to filter functions
5. Packet list pane – Displays all the packets in the current capture file.
6. Packet details pane – Shows a more detailed view of the packet currently selected in the Packet List pane
7. Packet bytes pane – A hexdump view of the packet currently selected in the Packet List pane
8. Status bar – Provides informational messages and feedback to the user

Sample Wireshark Capture

In this example, I will start a Wireshark capture on my wired laptop interface. I will then launch Thunderbird to retrieve email from Comcast and GMail

1. First launch Wireshark.
2. Then select Capture->Interfaces from the menu bar. 
3. 3) This will bring up the Interfaces dialog box. Select the interface you want to use. This is important since Wireshark (as with any protocol analyzer) can only capture data from a network it is physically connected to. I will be using the wired Ethernet adapter in my laptop so I will choose the Intel adapter in the list. Click the ‘Start’ button. Capturing will now begin. After a short while, you will see the main Wireshark window (the packet list, detail and byte panes) fill with data. 
4. Now I will launch Thunderbird and login to both my GMail and Comcast mail accounts. At this point I will wait for all my mail to download and then I will stop the network capture by selecting Capture->Stop from the menu bar. Click File and Save to save this capture to disk after all data is captured.   
5. I have just captured two complete pop3 sessions with Wireshark. To single out the pop session information I will apply a filter. In the filter bar enter the following text and press the ‘apply’ button: tcp.port eq 110. This will limit the display to traffic on tcp port 110 (the POP port). Also notice that Wireshark “understands” the Post Office Protocol, so it will interpret bits of information such as POP commands and even authentication information. I do not connect to the Comcast mail server using SSL so my password is contained in the trace in clear text. I had to choose this screenshot wisely! I’ve actually used this to troubleshoot end user client connection problem to pop and imap servers. 
6. Scrolling through the filtered captured data only shows a conversation between two hosts; my laptop and the Comcast mail server. What happened to gmail? Well, I use SSL with my GMail account and SSL POP connections are associated with port 995 not 110. In the filter bar enter the following text and press the ‘apply’ button: tcp.port eq 995. This will show all the POP over SSL traffic. But notice that no other details are available about the application protocol. The protocols in use on port 995 are TCP, SSL and TLS. You will see some packets dealing with key exchange, but that is all to do with the security negotiations associated with SSL'TLS. All the Application data is encrypted.

Closing remarks
This is only the tip of the iceberg. I do not have a need to use this tool very often, but when I do need it, it is there and it can be a life saver – or a job saver.
For more information on this tool visit the Wireshark website at www.wireshark.org. The documentation on this site is quite extensive and should get you up to speed fairly quickly.
Also check out the documentation for whatever protocol you are sniffing. For the internet protocols, the RFC’s are a must read. You can find these at the Internet Engineering Task Force website at www.ietf.org.
Finally, if you are sniffing an undocumented protocol like nrpc (used by Lotus Notes'Domino) you may want to spend time researching a similar protocol that is documented. This may help you to understand what you are seeing in Wireshark related to the undocumented protocol.