An article was written a while back about how neural networks can be used to classify source code. Yes the source code that you write to feed to compilers / interpreters.

The article explains at a high level what method could be used to perform this activity. In the end the author claims some level of success and wonders how other neural-network implementations / techniques would solve the same problem. This got me curious enough and I spent a weekend trying to crack this with Neuroph, the neural network library for java. I present to you my analysis and results below. For the impatient here is the code-classifier DEMO.

Why solve this problem with neural networks ?

So where do we begin ? How about the question ‘Why do we need neural networks to solve this problem ?’. Well actually it might not be the ‘best’ way to solve the problem. A lexer is a better solution to this problem since it can scan the source file and probably give you a much better accuracy on the clasification. But neural networks are also great for this problem since they can keep learning based on the input that is provided. Add one more programming language into the bag and the network can learn about this language without a programmer needing to type extra code into a lexer.

So how do we do it ?

One of the features that differentiates one source code type from another is the keywords used by the programming language in question. So lets target that. Our neural network will try to identify if source code belongs to the java language or the python language.

It takes as its input a list of all keywords available in both languages. Mind you that there may even be some overlap between the keywords. For example the keyword ‘if’ is used in both languages. But there may be keywords that are used solely in one language alone.

Neural network viewed in graph mode: (The network is too huge to fit into this picture )

A middle layer will contain a few nodes that will back propagate anything that it learns. The output layer (just one node) will decide if the language is java / python. The value will range from 0-1, where 0 represents closeness to python and 1 represents closeness to java.

How do we train it ?

The process of training is made easy by the neuroph API. First we need a few java / python files. A program was written that reads each file and splits it based on the whitespace delimiter. Based on a known set of keywords, the network is fed the number of times a keyword appears in a source file.

for(Map<Double, Double> map : keywordStrength)
{
    Collection<Double> values = map.values();
    Vector<Double> input = new Vector<Double>();
    input.addAll(values);
    SupervisedTrainingElement supervisedTrainingElement = new SupervisedTrainingElement(input,output);
    trainingSet.addElement(supervisedTrainingElement);
}

For example, recognized keywords may be ‘public’ represented by the input index 0 and ‘static’ represented by the input index 1.  A java file may contain the word pubic 10 times and static 5 times. The input to the network in this case would be 10 5. We can now use these numbers to tell the network ‘Hey if you see this pattern it is likely that the source code belongs to language X’. Neat !. Once the network is trained you can load it and use the same process to classify source code for any given input String.

Well… show us some stats:

Before I say anything else, I did not expect this approach to succeed more than 10-20% of the time. This is because the number of files that I trained this network with was around 20. Java + Python included. But I was surprised to see the network classifying the data correctly most of the time. There were very few cases where it failed.

The real test for this network is now in your hands. I made a java web app on top of this ‘code classifier’ application and built a simple UI using Jquery. You can type in any code / junk you want and hit the trained network. If it manages to classify your code correctly at least 60% of the time, I would consider that an enormous victory (for Neuroph ?)

So what are you waiting for, take her for a spin.

How do I game the results:

This is pretty simple. There are certain rules you must follow. Not following them will give you weird results

1. Make sure that your code does not contain comments. I do not strip comments from code before analyzing them for keywords. Including java like keywords in a python file comment will confuse the network.
2. Input only those files that will compile / run correctly on Java / Python. In theory you could feed in a Shakespeare novel to my network and it will still try to classify it as source code

So does the network work for you ? Feel free to leave a comment.

I also wrote a simpler tutorial for Neuroph that is less practical and more academic. Check it out if it interests you

Ever visit sites like topcoder.com and solve a problem or two ? No ? You should and do so regularly. It will help find tune your programming / design skills. This is because solving the problems requires going though the following thought process

1. Understanding the problem.
2. Thinking up a solution that covers all use cases.
3. Executing your thought process through code.
4. Testing.

Now as simple as that sounds, it is not so easy to get right. Especially when you are competing with thousands of fellow coders for time. And when time is a constrain you usually select the first solution that comes to your head, which might not necessarily be the best. So there is that risk of your code timing out even if it is following the right logic. I would love to delve into a problem right about now and throw in some examples and illustration, but you get the point.

When the thought process that we follow is not well defined and we lose focus, it often leads to buggy code. Don’t believe me ? Here are some bugs I caught in a code base on a recent project.

Bug 1

Bug 2

Bug 3

There were more bugs identified after development than what is listed here. But what I wanted to highlight was that the problem was introduced into the system due to lack of focus and clarity of thought. Well, more often than not anyway. Other times the bug sneaks into the system because the number of test cases to test a method were not sufficient.

Have a look at this google research blog post about broken binary search and merge sort algorithms for example. They pretty much have the same bug as the date calculation bug that I highlighted in a different post. We are all prone to creating bugs, but we can minimize the number of ones we introduce into the system.

So what have I learned from introducing bugs so far ?

1. As long as you keep your skills sharp and practice solving problems, the number of bugs you introduce will reduce. I try to visit sites like topcoder.com or codingbat.com whenever I find the time.
2. Learning from other’s mistakes and best practices can save you tons of time.
3. Try not to cram in your work within X hours. This tends to focus the effort on finishing the code on time instead of writing it neat and bug free.

As always its easier said than done. But hey, its never too late eh ? As for finding the bugs in the first place…

1. Use a tool like FindBugs after you are done coding. You will be surprised at how many problems turn up.
2. Try to maintain a code coverage of at least 80 %. Tools like Emma can help you track this.

I was testing an application today and came across a bug. A screen had some CRUD operations on a resource. The problem was, whatever I did, the system would perform the operation and tell me that the resource already existed. This left me scratching my head for quite a while. I add a resource and it says it already exists and then adds it. Delete and update also do the same.

So the debugging process started and I sat down with eclipse to get to the root of the problem. I verified that indeed the CRUD operations were reflecting in the persistence store. Then I came across these magical lines of code

public static final String S_SOMETHING_EXISTS = "something.already.exists";
public static final String S_SOMETHING_ADD_SUCCESS = "something.already.exists";
public static final String S_SOMETHING_DELETE_FAIL = "something.already.exists";
public static final String S_SOMETHING_DELETE_SUCCESS = "something.already.exists";

This left me smacking my head. It should have been

public static final String S_SOMETHING_EXISTS = "something.already.exists";
public static final String S_SOMETHING_ADD_SUCCESS = "something.add.success";
public static final String S_SOMETHING_DELETE_FAIL = "something.delete.failed";
public static final String S_SOMETHING_DELETE_SUCCESS = "something.delete.success";

The bug was introduced when I copied the first constant and pasted it three times. I forgot to change the keys . My point is, most of the bugs that I see introduced into the system are due to silly mistakes like the one above. This is where tools like FindBugs (which can be extended with your own rules) and CheckStyle and other bug finder / code review tools come in handy. A human code review is also a great tool. This little bug served as a reminder as to why we have such tools. To a developer this bug might seem small, but wait till an end user sees the message and says ‘What gives !’.