Monitoring / Debugging web traffic
If you are a web developer, you would have encountered one of the following challenges.
- Ensure that your web page contents are not cached by the browser.
- Check if your clear text componenets are gzipped by the web server.
- Add additional request parameters / headers for testing.
- Determine if a stolen cookie can be used to login to the site.
- Analyze request / response times.
- Verify web server response status / headers.
Well, I could go on and on and the list is certainly large. Do you see anything common in the list mentioned above ? They all require a tcp monitor / web debugging proxy to solve. There are several that are available out there. If you have never used one before, its never too late. I was able to save several minutes of time when programming java web apps after marrying them with a debugging proxy / monitor.
I use a tool named charles. Charles is a commercial product and a trial version will let you debug for 30 minutes. It is pretty cheap to purchase and gives you a central dashboard where you can focus your efforts. The trial version also shows a pop up now and then that will wait for 5-10 seconds before it lets you use the tool again. I have found these limitations to be bearable, if you are using the tool for free.
Here are a few use cases where a debugging proxy can prove to be useful.
Cache control:
It is important that you mention an expires header or cache control policies for your web pages. This allows your browser to display cached content if the content has not yet expired. You can use Charles to track these headers and check if they are actually being returned in the response
Start the charles debugging proxy. If you are using firefox, there is a firefox plugin that integrates with this tool. Now simply visit the site whose expires headers you want to check. Here is an example.
Expires header:
The report from charles shows that the expires header for a resource was set to Sep 22 2009. If you want to tweak that value, you can always change the params in the web server, reload the configuration and check back here.
Response times:
Now, if you want to check the request response statistics of your web site and how well it is performing, simply visit a couple of links on your site. The left side bar should look like this
Charles:
Click on the Chart tab on the right side bar and this will give you detailed information about the site performance. The chart can display data by time line,size, duration, or resource type.
Chart:
Arguably, a tool like apache JMeter is more suited to perform load tests and visualize test results. But a TCP monitor / debugger can also help here.
The tool has many more advanced capabilities which I will be writing about in future posts. If you would like to keep up to date, subscribe to the RSS feed.
I hope that the tool saves you time. Have fun programming 
PS: I do not have any affiliation with Charles. I just think it is a cool tool. Other web proxies that you might be interested in are given below
Fiddler – From MS
Tcp Mon – A cool tool from java.net
Wireshark – A very advanced network monitoring tool that captures packets for analysis.