Let me begin this post by saying that I am not writing this so that you can read this and become a haCkEr. I am writing this post so you can learn to identify a vulnerability and try to avoid an embarrassment.
Google is an amazing search engine. The problem is that it is too good at what it does sometimes 
Here are some ways that google can reveal vulnerabilities on your website by mistake.
You allowed google to index a critical file:
This happens more often than you think. WordPress for example houses important files under the wp-* folders and it is no one’s business except yours to look at these files. Other files like .htaccess htpasswd are critical to your site’s security (if you are using apache and ‘allow overrides’). Do not allow google to index them. You can prevent that by placing a robots.txt file on the root path of your website. More on that here.
I have seen a few estimation nightmares in my time and have been unfortunate enough to be in some of them. Let me narrate a few anecdotes first
Anecdote 1:
I used to work with a re-insurer. This company had a legacy application that was written in fortran. Yes fortran. It did some very important things. It was capable of making estimations for a given market and it crunched a lot of numbers into meaningful data. Because this application was written in fortran, finding the right engineers to maintaining it was difficult. So they decided to shift the application to a better supported platform / language.
The work came to IT and a manager said ‘Lets convert it to VB’. This person, did not know fortran and was not a master of VB either. No developers or architects were asked for advice. It was simply decided that the application should be converted to VB from fortran.
I joke with a friend about this all the time. ‘We are surrounded by Google programmers‘ he said, and I couldn’t agree more.
There is a difference between a ‘Google programmer’ and a ‘programmer that works at Google’. Google programmers simply search Google for a piece of boiler plate code and stick it into their app. It wreaks havoc later and causes trouble for a lot of people.
I chanced upon this blog post a while back that was complaining about lazy developers at stackoverflow. The complaint being that some developers ask silly questions, the answer to which a Google search will easily reveal. It seems some people are so lazy to use google that whole websites are dedicated to Google something for you
If you are a web developer, you would have encountered one of the following challenges.
- Ensure that your web page contents are not cached by the browser.
- Check if your clear text componenets are gzipped by the web server.
- Add additional request parameters / headers for testing.
- Determine if a stolen cookie can be used to login to the site.
- Analyze request / response times.
- Verify web server response status / headers.
Well, I could go on and on and the list is certainly large. Do you see anything common in the list mentioned above ? They all require a tcp monitor / web debugging proxy to solve. There are several that are available out there. If you have never used one before, its never too late. I was able to save several minutes of time when programming java web apps after marrying them with a debugging proxy / monitor.
The RSS cloud and the pubsubhubbub (PuSH) way of notifying RSS feeds has gained some interest of late, thanks to wordpress supporting the former format. If you have never heard of these protocols, heres the low down. The RSS feed works by having a client ‘pull’ data from a feed. That is, clients keep nagging at the server every X minutes saying ‘Hey do you have something new for me ?’. The server responds with a no about 99% of the time. This leads to a lot of inefficiencies. Instead if the clients were to use a push model, the data is actually ‘pushed’ to a client automatically without the client asking for it. This is great since the client no longer makes unnecessary calls to the server. The diagrams below illustrate this.
Classic update:
Rss Cloud update:
