How would an email server go about identifying spam email ? The problem is an interesting one. The challenges towards identifying spam are…

1. Scaling any solution to thousands of emails
2. Identifying spam even when there are small changes to the spam content
3. Reducing false positives

One solution could be to identify a hash for the spam message and compare the hash with the hash of a new message. The problem with this approach is that minute changes in a message can result in a different hash. A fuzzy hash (Context triggered piecewise hash (CTPH)) solves this by calculating hashes based on a trigger point in the text. Hash values are calculated for pieces of the text, delimited by a trigger. For example the trigger for the following text could be ‘a’ and ‘or’

why would a lazy sunday be greeted with sleep. or was I wrong? You were not sleeping ?

Here are the MD5 sums for this text, split by the delimiters / triggers

why would = 3cbca4bc4bba85fd54f384867ff4fd3e

lazy sunday be greeted with sleep. = e007174b7ef850ccc4b68ae1db98d1fe

was I wrong? You were not sleeping ? = 17d36b21dc2dfe530ea677075519a265

Summing up these hashes gives a final hash. If this text were considered spam, and minor changes occur to the spam, the individual hashes can be compared to arrive at a ‘match score’.

why would a lazy sunday be greeted with sleep. or was I wrong? You were not sleeping ? Did you get the jIAgra pills I sent ?

why would = 3cbca4bc4bba85fd54f384867ff4fd3e

lazy sunday be greeted with sleep. = e007174b7ef850ccc4b68ae1db98d1fe

was I wrong? You were not sleeping ? Did you get the jIAgra pills I sent ? = 556ce7bf1c804330863e4d39755d5c58

Only one of the hashes has changed. This would give a good score. The ssdeep library calculates fuzzy hashes for you. Comparing similar spam messages is like comparing similar source code. Developers love to copy one piece of code from Project A and paste it into Project B. Eclipse plugins like ‘Google code pro analytix’ try to track similar code. I am not aware if the plugin uses fuzzy hashes to make this comparison, but it is a possibility.

The fuzzy hash concept can be be extended to solve other problems

• A DDOS tool issues the same type of request across different clients. If a fuzzy hash is applied to one request and it is identified as spam, the same can be done across all other requests. If the same request originates from the IP again, the fuzzy hash calculation is not necessary. Simply ban the IP for a few days.
• Environments that are affected by a common problem probably log the same error message. A network timeout could affect say 5 servers. Comparing the error message likeness across the server logs can help determine the problem quicker
• CTPH is also used in forensics.

The next time you encounter a problem that involves ‘text similarity’ give fuzzy hashes a thought.

When mock-ups and screenshots fail to deliver the idea that you are trying to convey, a prototype can deliver a strong impact. But why should you prototype with GreaseMonkey when you can check out code from source control and mock a web page with a few shabbily scribbled lines of javascript ?

I often find myself comparing GreaseMoney with the Netbeans swing builder for thick clients. Its great because you can deliver a quick UI without worrying about the underlying functionality. But where GreaseMoney differentiates itself is in its ability to mock a live web page. By live I mean one that is already deployed on production. Never underestimate the impact of mocking a feature on a live website. Changing the innerHTML of a few HTML elements and hacking out some JS can get you quick results.

How an idea is presented to a person that might green-light it is pretty critical.From my experience, here is how it usually plays out

Of course, I am not trying to suggest that all ideas that you mock on a screen-scrapped live web site will kick off. An idea is great only when your potential users realize its significance and importance. The way you can get them excited about it, is to actually show them what it could look like. Once they are wowed, they will be eating out of your hand.

Be wary of the risk of getting folks too excited. Explain carefully that you are screen scrapping a page before you present it. The last thing you want is a customer thinking that you already implemented an idea

If you have never used GreaseMonkey before, the hello world tutorial is a great place to start. As you get your way around the simple stuff, you will eventually perform relatively complex operations like changing HTML content, setting timers, making ajax calls. Some of those steps will not work due to limitations in the way GreaseMonkey works. Which is when this common pitfalls page will be of great help.

Good luck mocking your web page.

Deleting a file that has been opened by another process in linux does not free up disk space. Running the df or du commands will indicate conflicting results. Closing / killing the process that opened the files will release the space on the disk. The lsof command can help you track, say the top ten open files in your OS sorted by disk space. If you ever run into trouble with large open files, use the following command

Top ten open files:
lsof / | awk ‘{if($7 > 1048576) print $7/1048576 “MB” ” ” $9 }’ | sort -n -u | tail

Output:

3.8054MB /usr/lib/libgtk-x11-2.0.so.0.2200.0
4.28024MB /usr/share/icons/hicolor/icon-theme.cache
8.17912MB /usr/lib/locale/locale-archive
8.86022MB /var/lib/apt/lists/lk.archive.ubuntu.com_ubuntu_dists_maverick_main_binary-i386_Packages
11.4047MB /usr/lib/flashplugin-installer/libflashplayer.so
14.6893MB /usr/lib/firefox-3.6.10/libxul.so
15.6504MB /var/cache/apt/pkgcache.bin
27.4744MB /var/lib/apt/lists/lk.archive.ubuntu.com_ubuntu_dists_maverick_universe_binary-i386_Packages
34.6615MB /usr/share/icons/gnome/icon-theme.cache
44.1719MB /home/user/.mozilla/firefox/tnrqzpro.default/urlclassifier3.sqlite

You can also lookup open files based on pid / port number. I hope the script saves you some time, should you ever find yourself in this situation.

Having switched to Suse linux a while back, I am enjoying the ride but for a few gripes. The UI does not load as smoothly as it should under certain circumstances. The keyboard also acts all crazy without warning. The new ubuntu release 10.10 is here, so I wanted to give that a shot hoping the experience would be better than what Suse had to offer. May be this was a KDE Vs Gnome problem. Perhaps 32 bit installations are less troublesome than the 64 bit ones. Well I wont know unless I try.

And it begins:

After downloading the Ubuntu ISO, I went about writing it to a CD and started the install process. The installation was riddled with error messages. Selecting partition X on hard disk 1, made the installer hate me. It complained saying ‘Either the hard disk or the CD has some sort of media related problem’. ‘hmmm… its probably the hard disk since there were no errors when the CD was written’ I thought.

Selecting another partition fixed the problem. It was now time for GRUB, the bootloader to fail. For some unknown reason, GRUB failed to install correctly on any given partition. One of the installation choices allowed me to install a bootloader manually after the install process. So I ended up installing Ubuntu; booting to Suse linux; manually adding a menu selection into GRUB by selecting the appropriate partition. This also failed with GRUB spurting out the following message on boot ‘Error 1: Filename must be either an absolute pathname or blocklist’. Debugging this problem took forever. It turned out to be a missing parameter for the ‘configfile’ command which caused the error. But no luck in the end. Ubuntu refused to load.

Media corruption:

It finally dawned on me that I should have checked the ISO hash with the one on the CD that was burned. The Ubuntu help page on this is great ! And it turns out that the hash was different and the CD was corrupt. Firstly, I found it very hard to digest that this was even possible. May be because it had never happened before. Secondly, I never knew that pen drive linux was a very convenient solution for an error free bootable linux. I decided to take that route.

Buggy BIOS:

Fast forward a few minutes and I have my flash drive plugged in, ready to boot Ubuntu and reinstall. Now my computer hangs up and displays a black screen of…. err death ? The BIOS tells me that a USB ZIP drive has been recognized. Wait… what ? A zip drive ? I plugged in a flash drive. So may be the BIOS is buggy. I updated the BIOS and now… finally… I get to reinstall Ubuntu. The installation went through without a hitch.

Looking back on the experience I wonder if some of these hurdles prevent folks from adopting linux. Some are not really linux’s fault while others are. Corrupt CDs / Buggy BIOS / Use of technical language during the installation like ‘partitions’ ‘mount’ etc / bootloader configuration. Being someone that is technically inclined, I enjoy solving some of these interesting problems. To someone else, these are just annoyances. People just want to get things done. I can understand why even one of these problems could end up frustrating someone and make them give up.

Anyway, I hope the anecdote above saves someone a few hours. This was how I blew my Sunday

Many passwords in linux are encrypted using the crypt() utility. The user is usually not aware of the difference between a crypt and a MD5 encryption. Well it can turn out to be important, especially if crypt uses the default DES-based scheme to perform the encryption.

The problem with crypt() + Traditional DES is that it truncates the password length to 8 characters. Users are not usually aware of this and assume that the entire length of the password has been saved and encrypted. Take the apache tool htpasswd for example. It uses crypt() to encrypt passwords (It may also use its own MD5 routine) into a password file. The following command creates a new user in a password file

htpasswd password_file new_user

After this command is executed, you are prompted for a password. If the password is greater than 8 characters, for example – 123456789, it will still be accepted and no warning will be provided that it was truncated. So providing the password 12345678 will also allow you to be authenticated into the system. Why is this bad ?

• The time taken to crack 8 character passwords is shorter (in relative comparison).

• It is also likely that the password was truncated in these scenarios, so an attacker may well target passwords that are exactly 8 characters in length.

• Some people have the knack of prefixing the password with the username first. Bad idea if your username happens to be 8 characters long

• The user may not even be aware of the problem, since he/she assumes that the password is strong and greater than 8 characters.

So the next time you provide a password to a system, you might want to know how they get saved into a persistence store and what encryption is used.

I came across an interesting application today. Google has released an application named Jarlsberg that is full of security holes. The intent is to make developers learn how these holes work and put them in a position to combat the security vulnerabilities.

You can visit the app to learn more. Security flaws to be detected are classified under the following categories

• Black box. You dont know the code
• White box. Requires you to see the code to understand how to break it.
• Gray box. Some code will be made visible.

I also came across an instructor’s guide that has problems to be solved in the application, graded by their difficulty level.

What better way to learn an exploit than to perform it on a test system ? Some of the exploits involve

• XSS and related challenges
• Path traversal exploits
• DOS
• Buffer overflow
• SQL Injection

and so much more. Give it a try now

I use remote linux services often and exposing them as local services can be performed securely using SSH. For example you can access a tomcat server or email server hosted at IP 1.2.3.4 by opening a secure SSH tunnel between your local machine and the target address – 1.2.3.4.

The OpenSSH tool can be used to perform SSH related activities on your machine. Simply install it with yum, apt-get or Yast, if it is not already available. Once you have it use the following command to open multiple SSH tunnels to your services

Open tunnel and execute commands:

ssh 1.2.3.4 -lmyUser -L 3098:1.2.3.4:21 -L 3099:1.2.3.4:80 -L 3100:1.2.3.4:443

The command is explained below

1.2.3.4 – Your target IP

l – The user to login as

L – A local tunnel to a remote port

3098 – The local port to use when establishing this tunnel

21 – The remote port at 1.2.3.4 to which the tunnel will be established

Multiple tunnels can be opened by specifying multiple -L flags. For example http://localhost:3099 will now redirect to http://1.2.3.4:80/ That is a fancy way of saying all HTTP requests ( 80 is the default port ) for 1.2.3.4 can now be reached locally at port 3099.

Note that the above command will also log you into the remote system. If you want to open the tunnels alone, use the -N switch and the -f switch as shown below

Open tunnels only:

ssh -f myUser@1.2.3.4 -L 3099:1.2.3.4:25 -N

The -f switch asks SSH to work in the background and -N asks SSH not to execute any commands.

I logged into a gmail inbox today, surprised to find ‘Google Buzz’ asking me if I wanted in. I clearly said ‘No’. Not yet another social network. pfft. So after I said ‘No’ there it was sitting comfortably on the navigation bar and telling me people were following me and I could follow them.

There is a small link on the bottom of your Gmail page that says ‘turn off buzz‘. Click on that and you get disconnected from the social network. I wonder what google plans for Orkut.

After project owners were asked to move out of kenai.com, Oracle now admits that it did a poor job at communicating its plans for the project. An updated post from Oracle says that the future of kenai.com projects is secure. Those projects will be migrated to java.net and continued as is. Here is an extract from the post

We don’t believe it makes sense to continue investing in multiple hosted development sites that are basically doing the same thing. Our plan is to shut down kenai.com and focus our efforts on java.net as the hosted development community. We are in the process of migrating java.net to the kenai technology. This means that any project currently hosted on kenai.com will be able to continue as you are on java.net. We are still working out the technical details, but the goal is to make this migration as seamless as possible for the current kenai.com projects

Most developers seem to be moving out of kenai.com after being asked to leave. For those of you that do not want to move, there may still be hope

I ran into a weird little error while trying to tune the performance of a query in the oracle database. A table had an index on a VARCHAR2 column. After a fair number of inserts were made the population of this table grew to 1.5 million records. A query that did an join on the VARCHAR2 column was talking ages to get the results. Well, it was taking 900ms actually but the SLA for the API call is way below that.

An “explain plan for QUERY” statement was run followed by a select from plan_details. It turns out the index is never used and a full table scan was used to get the data. Hence the problem.

So all I have to do is find out why the index is not being picked up. It probably has to do with statistics, and a rebuild on the index might help I thought. Or may be I need to analyze the index. So I went about trying to do some of these things. There were no DBAs around so I had to experiment a little.

analyze INDEX YOUR_INDEX_NAME validate STRUCTURE;
 
Error: ORA-01418: specified INDEX does NOT exist
 
SQLState:  72000
ErrorCode: 1418

This was totally weird. I knew that the index did exist and the name was correct. To double check I ran this query

SELECT * FROM all_indexes WHERE INDEX_NAME='YOUR_INDEX_NAME'

And it did return my index correctly. After trying several other commands related to indexes they all returned the same thing. I tried to prefix the SCHEMA name and that did not help.

So it turns out, I do not have permissions to create indexes or do any operations related to them and that is what throws this error. The error is pretty misleading since there already exists an error code for insufficient prvileges for performing a given operation – Error: ORA-00990: missing or invalid privilege.

I passed the index related queries on to a DBA and the performance of my application queries were back on track. I hope this saves a developer the time I lost trying to find out why the index did not exist.

PS: Oracle 11 also has an index visibility option which can be verified with

SELECT INDEX_NAME,VISIBILITY FROM USER_INDEXES WHERE INDEX_NAME='YOUR_INDEX_NAME';

A few days ago the EU had approved the takeover of Sun microsystems by Oracle. This brings to an end the long awaited acquisition. There have been several blog entries about the fate of the open source projects that Sun currently supports. I have read some of the arguments put forward about why developers should move away from / use certain projects.

However I also find myself in a dilemma. I installed GlassFish and deployed a few apps on it recently. I found it interesting and was impressed with it within a few minutes of using it. MySql is also a great open source project. There are many tools built around the database and it competes well with PostGresql. Netbeans is also a great IDE and supports glassfish.

I am very tempted to continue using these projects for the development of new pet projects and to learn new specs. Right now I expect the support for these projects to continue for a few years, and I do not really expect to see great innovation of any sort. The best case scenario is that Oracle leverages the usefulness of these projects and decides to keep them alive while giving them healthy financial support. One is allowed to dream

So it boils down to the following question. “Will you as a developer / architect / <Insert technical position here> continue to use projects like GlassFish, Netbeans, MySql etc ?” Or are you looking to move away to other open source projects immediately ? Voice your thoughts through this poll

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

I have been a tad busy fixing some weird little bugs lately. They helped me appreciate the multitude of things that can go wrong in a live environment and served as a gentle reminder that you should always be on your heels.

Here they are

1. LDAP and the user

A web application product was configured to use an LDAP directory structure. The directory was segregated into roles / groups / OUs, the usual. One of the users had trouble logging into the product. This was weird because this person was a valid user and Outlook seemed to recognize him. Outlook uses the same LDAP tree. So I dug into it. The easiest way to check what is going on is to use a LDAP directory browser. I use the one provided by Novell for free. The LDAPs support is not great but it will do for basic lookups.

So as I dug into the tree, it turns out this user was mapped as a group. Yes a group. The product was configured so that only the “user accounts” LDAP directory was looked into for valid users. So since this user was, ummm a group, the application was unable to find him. My only thought was ‘wow ! how did this go under the radar for so long ?’. The mistake was understandable though since groups and users are under a similar looking structure. The LDAP admin must have had too many doughnuts at lunch and probably dozed through when configuring the user into the system

2. Authentication failure

This one involves the LDAP too. We built a web app recently that was supposed to check for users under a specific tree node. The logic was to bind to the LDAP and if that is successfull then it must mean that the user exists. Unfortunately the negative scenario to this logic was not tested all that well. The API did not throw an exception when the Bind failed. So this meant that I could login to the application without a password. This also meant I could use any user name I wanted to and login. Even one that does not exist. I was tempted to do some operations as MrBunnyRabbit76 but fixed the bug instead.

3. FTP and the CPU

Now comes this little gem. A FTP process had been scheduled in CRON to run every 10 minutes or so. Its job is to ensure that a local folder and a remote FTP folder are in sync with each other. The program used to do this is lftp with the R switch (for reverse). It so happens that the FTP account gets locked because it ran out of space. The account is unlocked the next day. However the lftp processes did not terminate for some reason when the account was locked.

I come back to check on a web application on the server and it gives a 503 error. hmm… weird I thought. Everything is fine, tomcat is up, apache is up and yet a 503. The problem was that the lftp processes that did not terminate caused the CPU to be overloaded. All other processes were begging for some CPU time. The lftp processes were sitting quietly and drinking up precious CPU power for almost 2 days. Once the processes were killed things went back to normal.

Weird things happen in PROD whether it is your fault or not. No matter how much you test, it always pays to be on your watch

I chanced upon a post a while back about how a switch statement should be replaced with the strategy pattern. If you have not had a chance to go through it, please do. The post is not very long.  I found myself disagreeing very strongly with the author of the post and I was surprised to find that people thought this was a good idea.

The gist of the post was that using the Strategy pattern was better than using switch statements to determine which logic to execute. Here is why I think the idea used there was bad

1. The introduction of the Strategy pattern, in the example, introduces three new classes. So for every case in a switch statement, we should go about replacing the corresponding code with a new class ? This could easily lead to an explosion in the number of classes.
2. Distributing logic across classes makes it harder for a developer to follow. When I traverse 2 / 3 links down a code path I might lose track of where I started.
3. Introducing a new case means I have to introduce one more class, remember to add that as a strategy and switch to that strategy correctly when the case is presented.
4. Introducing a new class to handle a ‘case’ might not make sense. Classes are supposed to cohesively carry out a function. Introducing a new class for every function that a class is supposed to perform, dilutes the purpose of the class.
5. The code that was supposed to perform the switch case is simply done else where and called a strategy.

Applying patterns where they do not belong, can be an anti pattern by itself. I can relate with what the author is trying to do. When I finished the ‘Head first design patterns’ book I was racing to implement a pattern for everything that was around me. The book stopped me right there and warned ‘HelloWorld does not need a design pattern’. And they are absolutely right. For a design pattern to succeed, you need

1. The right problem to apply the pattern.
2. Correct implementation of the pattern to solve the problem.
3. Knowledge and relevant documentation if necessary, to let the maintenance developer down the line know that this is how you used the pattern.

Patterns are meant to solve common design problems but trying to use them everywhere can lead to overly engineered code that is hard to read and maintain. Use patterns only when they are necessary and help you solve the problem.

After a long wait I got a google wave account. yay ! Took the wave for a spin over the last few days and there were some interesting things that I observed. I wrote my first java wave robot and it was pretty cool. But an explanation of how the robot works should be left to another post all together. I will share my general observations in this post.

Deleted welcome messages:

The first thing that was weird was that welcome messages are often deleted by wave users or by bots. This is nuts. The wave welcome messages also have a lot of noise amidst them with quotes like ‘Please do not delete this !’ in bold red with a big font size. Wave still does not have a feature to disable edits. It is coming soon but it is not yet active.

Lonely waves:

Almost every one is complaining about this. They have a wave account but the 8 invites that were promised have not yet arrived (for some). The reason being that google wave is expanding its user base very slowly to avoid a massive crash of the system. It is also difficult to find waves centered around a topic. I tried to find java discussion but my searches ended up with nothing If you are a java geek and have a google wave account simply add your wave account name in the comments section. At least those reading this post will know how to contact you.

Miscellaneous / Tit bits:

• Google wave was pretty slow. But I expected that since this is a DEV preview.

• A funky sidebar was present on each window. I did not like this. I am used to clicking somewhere within the side bar’s empty space to move up or down. Fortunately the mouse wheel works with the side bar.

• Some robots are creating havoc. Editing comments left by others and deleting wave replies. This leaves some people vexed.

• There are some cool robots out there. I tried out the ISBN wave robot that replaces the ISBN number with a picture of a book and a link to O’Reilly. I can think of many applications where such robots can come in handy.

• Editing / deleting wave replies (blips) involves 2 clicks. You click a drop down and then select an option from it. Usability of the blip can be enhanced by placing small icons near each blip, which you can click to perform that action. Just a thought.

• Removing yourself from a wave discussion is not yet supported. You can add contacts without asking the contact’s permission first.

• There are wave ‘groups’ out there that discuss specific topics. You can subscribe to these wave groups. But sometimes, like in a forum thread, the discussion takes a U turn and towards the end of the discussion the participants are talking about something entirely unrelated to the root topic.

• Searching for waves is done in a variety of ways. You can search by tags / contributors / authors / wave ID / gadgets etc etc. Each wave can be tagged to particular keywords like a blog post.

This is just a tip of the ice berg. The java robot was very exciting to create and it was more exciting to watch it in action. There are some good tutorials out there that tell you how to write a bot but I see a gap where the flow of events and the overall picture of the robot is not brought out. I will try to cover that in my next post. You can subscribe to the RSS feed if you would like to keep an eye on it.

Let me begin this post by saying that I am not writing this so that you can read this and become a haCkEr. I am writing this post so you can learn to identify a vulnerability and try to avoid an embarrassment.

Google is an amazing search engine. The problem is that it is too good at what it does sometimes Here are some ways that google can reveal vulnerabilities on your website by mistake.

You allowed google to index a critical file:

This happens more often than you think. WordPress for example houses important files under the wp-* folders and it is no one’s business except yours to look at these files. Other files like .htaccess htpasswd are critical to your site’s security (if you are using apache and ‘allow overrides’). Do not allow google to index them. You can prevent that by placing a robots.txt file on the root path of your website. More on that here.

The better option is to put in place a configuration that will not allow the sensitive file to be displayed in the first place. Not all robots will obey what you instruct using robots.txt. The FilesMatch directive on apache can help you protect your site.

You can double check that google can read your robots.txt using google analytics. You can check the files that google has indexed using the query‘site:yoursitename.com

Google indexed a service page that is being served on a non regular port:

Examples of this are login pages or services that do not require a password. Searching for such pages can be done using the “inurl” keyword in searches. Here is an example inurl:8080. There are ways to tweak that search string to reveal more information about services on other ports. When you complement inurl:something_unique_in_the_url with a search using quotes, like inurl:1234 intitle:”Administration blah”, it can yield some very interesting results. Pick your favorite admin tool and replace the port and title with the admin home page equivalent. The search works on many major application / web servers.

Remember that google indexes your page. Even if you correct the problem, the damage is done and is still being done. With cached pages, a service that does not ask for user name and passwords (yes there are important services that do not require a username/password) will be completely indexed. Yikes ! The data that your service exposes is cached and indexed for everyone to see. Not what we want.

To avoid this simply shutdown services you do not need. If you need a service but you want that service to be private, block the port with a firewall.

You can optionally tell google bot and other bots not to index the page in question. But that is not really a solution. Be proactive and secure the service. A cached page can end up earning you some DOS attacks.

Google cracks MD5:

I realized that google could be used to crack weak passwords from this post. If the encryption is done without salting, the password will result in the same hash every time. A weak password can be guessed easily using this technique.

The lesson here is to use a strong password that no one will guess. The other lesson is to ensure that the links on your site do not pass along sensitive information. Here is the google search in case it interests you

Cached directory pages:

Your web server is quite capable of displaying a directory listing. What this means is that besides displaying HTML, if I were to request for a directory name instead, your web server will reveal the contents of the directory to me. Why is this bad ? It helps find more vulnerable files that are housed inside those directories. You can ask apache not to serve directory content by configuring the same in httpd.conf. The line of configuration will look something like this



Options Indexes FollowSymLinks
# More stuff here

Remove the word Indexes.

The related search query in google is intitle:”index of /”. Tweaking it will provide better results.

Before you make any configuration changes, always make a backup. Read about the changes you are making and understand what you are doing before you do it. Try these tricks on your site and check if it is secure. Be creative. Think about other sensitive terms like jsessionid, username, passwd, password, id etc.