An article was written a while back about how neural networks can be used to classify source code. Yes the source code that you write to feed to compilers / interpreters.

The article explains at a high level what method could be used to perform this activity. In the end the author claims some level of success and wonders how other neural-network implementations / techniques would solve the same problem. This got me curious enough and I spent a weekend trying to crack this with Neuroph, the neural network library for java. I present to you my analysis and results below. For the impatient here is the code-classifier DEMO.

Why solve this problem with neural networks ?

So where do we begin ? How about the question ‘Why do we need neural networks to solve this problem ?’. Well actually it might not be the ‘best’ way to solve the problem. A lexer is a better solution to this problem since it can scan the source file and probably give you a much better accuracy on the clasification. But neural networks are also great for this problem since they can keep learning based on the input that is provided. Add one more programming language into the bag and the network can learn about this language without a programmer needing to type extra code into a lexer.

So how do we do it ?

One of the features that differentiates one source code type from another is the keywords used by the programming language in question. So lets target that. Our neural network will try to identify if source code belongs to the java language or the python language.

It takes as its input a list of all keywords available in both languages. Mind you that there may even be some overlap between the keywords. For example the keyword ‘if’ is used in both languages. But there may be keywords that are used solely in one language alone.

Neural network viewed in graph mode: (The network is too huge to fit into this picture )

A middle layer will contain a few nodes that will back propagate anything that it learns. The output layer (just one node) will decide if the language is java / python. The value will range from 0-1, where 0 represents closeness to python and 1 represents closeness to java.

How do we train it ?

The process of training is made easy by the neuroph API. First we need a few java / python files. A program was written that reads each file and splits it based on the whitespace delimiter. Based on a known set of keywords, the network is fed the number of times a keyword appears in a source file.

for(Map<Double, Double> map : keywordStrength)
{
    Collection<Double> values = map.values();
    Vector<Double> input = new Vector<Double>();
    input.addAll(values);
    SupervisedTrainingElement supervisedTrainingElement = new SupervisedTrainingElement(input,output);
    trainingSet.addElement(supervisedTrainingElement);
}

For example, recognized keywords may be ‘public’ represented by the input index 0 and ‘static’ represented by the input index 1.  A java file may contain the word pubic 10 times and static 5 times. The input to the network in this case would be 10 5. We can now use these numbers to tell the network ‘Hey if you see this pattern it is likely that the source code belongs to language X’. Neat !. Once the network is trained you can load it and use the same process to classify source code for any given input String.

Well… show us some stats:

Before I say anything else, I did not expect this approach to succeed more than 10-20% of the time. This is because the number of files that I trained this network with was around 20. Java + Python included. But I was surprised to see the network classifying the data correctly most of the time. There were very few cases where it failed.

The real test for this network is now in your hands. I made a java web app on top of this ‘code classifier’ application and built a simple UI using Jquery. You can type in any code / junk you want and hit the trained network. If it manages to classify your code correctly at least 60% of the time, I would consider that an enormous victory (for Neuroph ?)

So what are you waiting for, take her for a spin.

How do I game the results:

This is pretty simple. There are certain rules you must follow. Not following them will give you weird results

1. Make sure that your code does not contain comments. I do not strip comments from code before analyzing them for keywords. Including java like keywords in a python file comment will confuse the network.
2. Input only those files that will compile / run correctly on Java / Python. In theory you could feed in a Shakespeare novel to my network and it will still try to classify it as source code

So does the network work for you ? Feel free to leave a comment.

I also wrote a simpler tutorial for Neuroph that is less practical and more academic. Check it out if it interests you

The java neural network Neuroph was making news recently about its integration with Hadoop. Neural networks can solve some interesting problems once they are trained. This article aims to provide the baby steps necessary to writing your first java program that loads a trained neural network.

Before you even begin to read anything that follows, a basic understanding of neural network terminology and the concept behind the same is necessary. The following articles are great starting points to understanding neural networks

Neuroph and neural networks – Part 1

Neuroph and neural networks – Part 2

Neuroph and neural networks – Part 3

Intro to neural networks

Cars and Signals:

We will  simulate the scenario where cars wait at a signal and move only when the lights are green. This simple example should help get you started. Our aim is to define a neural network with the easyNeurons swing application; train it; import it into java and use it in an application.

There are 3 states for this signal

1. Red – Stay where you are

2. Yellow – Start your engines

3. Green – Go

The neural network will take 3 inputs and its architecture will be based on the multi layer perceptron setting. A hidden layer with 4 nodes will decide the output. The output itself will be based on one node, whose value will determine if the cars can move or not. This is what our network looks like so far

Create the network in neuroph: Network -> Multi layer Perceptron

Basic network with no training: View -> Graph

Now we need to train the network so that the output will be as expected when the signals change. The rules are pretty simple and are shown below.To train the network we create a training set

Neuroph training set:

Train the network to respond to the inputs:

After the training rules have been laid out, it is time to train this network. Simply press on the Train button and select the appropriate training set to use.

Set the parameters by which the network should learn:

The trained network:

Now that we have our network trained, lets try giving it an input. An input of 0 0 1 means the signal is green and the vehicles can go through. The output produced in this case is shown below. The output will vary based on the function used in the training set / error rate and other factors. But what is to be highlighted here is that the output is nearing 1 when the signal is green. Our network works as expected.

Signal is green and output is 1 (well almost ):

We can confirm how this decision was taken by the network by highlighting the weights. Additionally by representing the size of each node with respect to the activation contributed for that node, we can visualize how the input message propagates

Weight / Activation  highlighting:

So now that we have a neural network up and running, how do we actually use this inside java code ? Its pretty simple. Save the project as a .nnet file. Lets call this neural_traffic.nnet. To load the nnet file into your java project, simple use the classes provided by Neuroph like so

Loading a neural network into java:

public class TestTrafficNeural
{
    NeuralNetwork network = NeuralNetwork.load("neural_traffic.nnet");
 
    public static void main(String[] args)
    {
        new TestTrafficNeural().go();
    }
 
    private void go()
    {
        calculate(1,0,0);
        calculate(0,1,0);
        calculate(0,0,1);
    }
 
    private void calculate(double... input)
    {
        network.setInput(input);
        network.calculate();
        Vector output = network.getOutput();
        Double answer = output.get(0);
        System.out.println(answer);
    }
}

The code produces the following output

-1.6360230873976706E-6
-4.140786100885251E-6
0.9684448970000741

You can also define and train the network on the fly with code. But you would not want to do this for cases where large sets of inputs and nodes are involved. For simple problems the time taken to train a network is usually a few seconds. For large images ( Assuming you are trying to recognize images with Neuroph ) it can take a couple of hours.

This example is pretty easy to do and is certainly not a practical use case for real world problems. However I hope it gets you excited about using neural networks in your programs. With Neuroph doing this is pretty simple.