‘Acquire a certification in technology X‘ | ‘Earn a Domain Y certification‘

Do one of those statements list in your yearly goals ? It is almost certainly a bad idea. Let me explain why. I knew a company that required its employees to be acquire at least one certification on a technology per year. Its employees would work hard throughout the year and during appraisal time it is not uncommon to hear this converation

Foo:  ‘Dude did you take that certification you were supposed to ?’
Bar:  ‘Rats ! I didnt realize it was a part of our goals’
Foo:  ‘hmm… what do we do ? John has already passed it and that puts him ahead of us’

Foo and Bar will probably end up reading real exam questions (also referred to as brain dumps) and passing the exam with only a day’s worth of preparation. That another way of saying ‘They cheated’.  This hurts everyone.

• It hurts John.

He worked hard to pass this exam. The inclusion of Foo and Bar into the ‘pass pool’ dilutes his achievement.

• It hurts your employer.

The company probably reimbursed the certification and wasted its financial resources.

• It hurts the community.

When interviewers come across scores of folks that are certified for technology X, but their skills do not match what they were certified for, it reduces the value of that certification

Foo and Bar also do not realize how foolish they are going to look in their next job interview. If you are an employer, please consider removing this as a valid goal. If you are an employee who is stuck with this goal, do yourself a favor and either acquire the certification by working hard for it or default.

PS: I am not suggesting that all employees take the easy way out. I am merely pointing out a general observation that trends in our industry.

Java’s side that is not too well known would be its interface with external devices. Java can read bar codes interface with serial / parallel ports , and communicate with a web cam (JMF)

Speaking of webcams, the following snippet in the Java media framework (JMF) should list your webcam as a device

Vector info = CaptureDeviceManager.getDeviceList(null);

The problem is that this piece of code does not work all the time. Your webcam should be compatible with JMF to be listed by this call. The gstreamer API can come to your rescue should the JMF call fail. Here is how the gstreamer API describes itself

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

The API supports pipelines, which stream data from one node on the pipe to the next. A source node which plugs to a webcam can pass through a video-filter node and end up in a file node thereby recording the video from the webcam and saving it to a file. So where does java fit into this picture ?

The gstreamer-java project is a java interface to the gstreamer API. It allows one to define the piepelines in java and run them in the JVM. The excerpt to a simple video based pipeline is shown below

Video test with a simple pipeline: (No webcams)

public class VideoTest {
    public VideoTest() {
    }
    private static Pipeline pipe;
    public static void main(String[] args) {
        args = Gst.init("VideoTest", args);
        pipe = new Pipeline("VideoTest");
        final Element videosrc = ElementFactory.make("videotestsrc", "source");
        final Element videofilter = ElementFactory.make("capsfilter", "filter");
        videofilter.setCaps(Caps.fromString("video/x-raw-yuv, width=720, height=576"
                + ", bpp=32, depth=32, framerate=25/1"));
        SwingUtilities.invokeLater(new Runnable() {
 
            public void run() {
                VideoComponent videoComponent = new VideoComponent();
                Element videosink = videoComponent.getElement();
                pipe.addMany(videosrc, videofilter, videosink);
                Element.linkMany(videosrc, videofilter, videosink);
 
                // Now create a JFrame to display the video output
                JFrame frame = new JFrame("Swing Video Test");
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                frame.add(videoComponent, BorderLayout.CENTER);
                videoComponent.setPreferredSize(new Dimension(720, 576));
                frame.pack();
                frame.setVisible(true);
 
                // Start the pipeline processing
                pipe.setState(State.PLAYING);
            }
        });
    }
}

To convert this program to accept the feed from a webcam, all one has to do is replace the videotestsrc with the v4l2src that accepts a stream from a webcam or a tv. Voila, you now have a stream from a webcam to your java app. If you want to save the stream as an image to a file, you can always construct another pipeline to accept the jpegdec and filesink pipeline elements.

As for platform neutrality, the gstreamer API is itself supported in the distributions as noted on this page 

Note: There is a working example of content streaming through a webcam on this forum post. Unfortunately the source code in question (displayed below) did not work for the author of that post. It certainly worked when I tried it. Have fun trying it on your webcam

Webcam and gstreamer:

package gstreamervideo; 
import java.awt.BorderLayout; 
import java.awt.Dimension; 
import javax.swing.JFrame; 
import javax.swing.SwingUtilities; 
import org.gstreamer.Caps; 
import org.gstreamer.Element; 
import org.gstreamer.ElementFactory; 
import org.gstreamer.Gst; 
import org.gstreamer.Pipeline; 
import org.gstreamer.State; 
import org.gstreamer.swing.VideoComponent; 
public class Main { 
    private static Pipeline pipe; 
    public static void main(String[] args) { 
        args = Gst.init("SwingVideoTest", args); 
        pipe = new Pipeline("pipeline"); 
        // This is from VideoTest example and gives test image 
        // final Element videosrc = 
ElementFactory.make("videotestsrc", "source"); 
        // This gives black window with VideoComponent 
        final Element videosrc = ElementFactory.make("v4l2src", 
"source"); 
        final Element videofilter = ElementFactory.make("capsfilter", 
"flt"); 
        videofilter.setCaps(Caps.fromString("video/x-raw-yuv, 
width=640, height=480")); 
        SwingUtilities.invokeLater(new Runnable() { 
            public void run() { 
                VideoComponent videoComponent = new VideoComponent(); 
                // This gives only black window 
                Element videosink = videoComponent.getElement(); 
                // This gives 2nd window with stream from webcam 
                // Element videosink = 
ElementFactory.make("xvimagesink", "sink"); 
                pipe.addMany(videosrc, videofilter, videosink); 
                Element.linkMany(videosrc, videofilter, videosink); 
                JFrame frame = new JFrame("Swing Video Test"); 
                frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); 
                frame.add(videoComponent, BorderLayout.CENTER); 
                videoComponent.setPreferredSize(new Dimension(640, 
480)); 
                frame.pack(); 
                frame.setVisible(true); 
                // Start the pipeline processing 
                pipe.setState(State.PLAYING); 
            } 
        }); 
    }

Jconsole and visual VM help monitor java applications in any environment. Here are a few additional details that can help you monitor your java applications better

Jconsole and permissions:

Jconsole lists the PIDs of local java applications when you start it. But what if there are java applications that are running under a different user ? Jconsole will not display those java PIDs that do not belong to the user who started Jconsole. The JConsole FAQ lists a few troubleshooting tips to get around this problem. You can alternatively expose the Jconsole connection through a remote port using the -Dcom.sun.management.jmxremote.port property. Securing the remote JMX URL with a username and password should do the trick.

Viewing Jconsole / JMC Beans with VisualVm:

While it is great that you can profile your applications on VisualVm, some users miss some of the features provided by Jconsole such as viewing JMX beans etc. You can use the Tools -> Plugins option in VisualVm to install Jconsole / Jmx or related plugins. There is of course the option of writing a custom plugin to display more information. This article already explains how to go about writing a custom plugin.

OQL Plugin:

There are some cool plugins on the way for visual VM. One of them is support for OQL syntax. The language allows you to go through objects on the heap based on a selection criteria. You can use jhat and jmap to get a feel for what the language is like. The syntax is already supported on the existing heap dump analyzers.

Performance:

Switching on jconsole monitoring in production is a bad idea. The article linked on the last sentence made a performance comparison around 2006, so here is a more recent comparison running a java program which solves the FoxSequence problem (a problem at topcoder.com) 1000000 times. The overhead is highlighted by the orange area. The difference represents overhead in milli seconds.

The average overhead comes to around ~7% when calculated across several trials. The question to ask though is ’7% of what ?’. For applications that take on huge traffic, an overhead of 7% spread across millions of transactions is costly. The overhead % itself may vary for your application based on a variety of factors.

Take these factors into consideration the next time you profile / monitor java apps.

Java is sprinkled with classes that are immutable. Wrapper classes especially make for a good example. A co-worker recently asked me “Give me an example of immutability.”. “The String class” I answered, to which he scoffed. He had a good reason however. Consider this code…

public class GenericTest
{
    public static void main(String... args)
    {
        new GenericTest().go();
    }
 
    public void go()
    {
        String name = "David";
        name.replaceAll("D", "d");
        System.out.println(name);
    }
}

Those of you with a keen eye for detail would have quickly seen the problem with this code. If you didnt catch it, dont blame yourself. The String “name” is immutable and the replaceAll() method creates a new String with all the ‘D’s replaced with ‘d’. It appears as though the String has been mutated, but in reality nothing has changed. The problem has a simple fix. Simply change a line of code.

name = name.replaceAll("D", "d");

The bigger problem is with the way the API works. The replaceAll() method suggests that the class is mutable while it really is not. It gives the illusion of mutability which can confuse a lot of developers (me included). You are better off leaving the mutable operations to StringBuilder. When a String needs to be mutated, there is always the StringBuilder(String str) constructor that can help.

The problem is not isolated to Strings. Integer autoboxing and unboxing for example, take place when a ‘++’ operation is executed on an Integer wrapper reference. Each ‘++’ operation creates a new Integer while giving the illusion of mutability.

Take some lessons from the code above and try to keep immutable classes, well… immutable. There is nothing technically wrong per se with the code. But when it comes to usability, confusing method names / operations can cause lost time. Besides innumerable additional objects are created from these operations.

Domain specific languages (DSL) are great tools to communicate with non-programmers. Normally this group includes business users that would like to configure a system / rule using a fluent language (as in – a natural language). It also includes those like my 8 year old neighbor that knows absolutely nothing about programming. He would love to tell the computer how to perform a small series of operations, without delving into the specifics. Coincidentally, I have been reading up on methodologies to approach DSLs and was introduced to ANTLR.

Enter ANTLR:

What my neighbor needs is an English like grammar. This grammar needs to be parsed into something meaningful at runtime. Every time the grammar changes, the parser would need to change too. ANTLR, is a ‘parser generator’. Once a grammar is defined, ANTLR can code-gen a lexer and a parser for this grammar. The lexer identifies tokens in any input that adheres to the grammar and the parser makes sense of these tokens.

A GUI by the name ANTLRWorks helps you generate the grammar and its corresponding lexer+parser code. Lets take a look at an example that asks the computer if a number A is ‘bigger than’ or ‘smaller than’ number B.

Before you delve into the code, it might be a good idea to download the ANTLR library and the related ANTLRWorks language workbench.

The grammar:

Here is the grammar presented in full. Lets analyze it one piece at a time. (Some parts of the grammar are based on examples from ANTLR. You will find it easier to delve into documentation that way. )

grammar SimpleCalc;
 
@header {
  package com.example.antlr;
}
 
 
 
@members {
    private String answer = "";
 
    private void setAns(String num1, String num2)
    {
        Integer a = Integer.valueOf(num1);
        Integer b = Integer.valueOf(num2);
        if(a>b)
        {
            answer = "yes";
        }
        else
        {
            answer = "no";
        }
    }
}
 
 @lexer::header {
  package com.example.antlr;
}
 
/*------------------------------------------------------------------
 * PARSER RULES
 *------------------------------------------------------------------*/
 
is    returns [String expr]:    'IS' 
                (
                a=NUMBER 'BIGGERTHAN' b=NUMBER {setAns($a.text,$b.text); } 
                | 
                a=NUMBER 'SMALLERTHAN' b=NUMBER {setAns($b.text,$a.text); }
                )
                {$expr=answer;};
 
/*------------------------------------------------------------------
 * LEXER RULES
 *------------------------------------------------------------------*/
 
NUMBER    : (DIGIT)+ ;
 
WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+     { $channel = HIDDEN; } ;
 
fragment DIGIT    : '0'..'9' ;

Grammar definition:

grammar SimpleCalc;
 
@header {
  package com.example.antlr;
}

This defines the name of the grammar and the header under which the code needs to be generated. Lexer and parser code can be generated under different packages.

Custom java code:

@members {
    private String answer = "";
 
    private void setAns(String num1, String num2)
    {
        Integer a = Integer.valueOf(num1);
        Integer b = Integer.valueOf(num2);
        if(a>b)
        {
            answer = "yes";
        }
        else
        {
            answer = "no";
        }
    }
}

This is a piece of java code. It is embedded into the parser and can be called when a rule is executed. We maintain a private String by the name ‘answer’. When the rule that compares two numbers is invoked, the setAns() method can be called with the numbers in order to compare them.

Lexer rules:

/*------------------------------------------------------------------
 * LEXER RULES
 *------------------------------------------------------------------*/
 
NUMBER    : (DIGIT)+ ;
 
WHITESPACE : ( '\t' | ' ' | '\r' | '\n'| '\u000C' )+     { $channel = HIDDEN; } ;
 
fragment DIGIT    : '0'..'9' ;

In the lexer section, we identify the bits of the input grammar that the parser can make sense of. A DIGIT ranges from 0-9. When the parser encounters one or more DIGITs, it represents a NUMBER. We also tell ANTLR that any whitespace needs to ignored.

Parser rules:

/*------------------------------------------------------------------
 * PARSER RULES
 *------------------------------------------------------------------*/
 
is    returns [String expr]:    'IS' 
                (
                a=NUMBER 'BIGGERTHAN' b=NUMBER {setAns($a.text,$b.text); } 
                | 
                a=NUMBER 'SMALLERTHAN' b=NUMBER {setAns($b.text,$a.text); }
                )
                {$expr=answer;};

The parser section of the grammar, defines a ‘is’ operation that returns a String. The pattern to match is ‘IS NUMBER ‘BIGGERTHAN’ | ‘SMALLERTHAN’ NUMBER’. The numbers are stored in variables a and b. The code inside the {} brackets is embedded into the parser, just like the code inside @members. As such this code can call the methods in @members and is executed when an appropriate match is made.

The match ‘IS NUMBER BIGGERTHAN NUMBER’ results in the following call setAns($a.text,$b.text);. When the match is for the ‘SMALLERTHAN’ input, simply reverse the numbers in the method call.

Running the pieces:

Now that the grammar is in place, it can be debugged in ANTLRWorks or code-gened and ported to a java project. Debugging in ANTLRWorks reveals interesting internal details. Once can visualize how ANTLR goes about generating a parse tree for our grammar or what the syntax tree for the ‘is’ rule looks like.

ANTLRWorks:

Syntax tree:

The code-gen route allows one to execute the parser for a user defined input.

import org.antlr.runtime.ANTLRStringStream;
import org.antlr.runtime.CommonTokenStream;
 
public class CalcTest
{
    public static void main(String... args)
    {
        new CalcTest().go();
    }
 
    public void go()
    {
        try
        {
            SimpleCalcLexer lex = new SimpleCalcLexer(new ANTLRStringStream("IS 19 SMALLERTHAN 20"));
            CommonTokenStream tokens = new CommonTokenStream(lex);
            SimpleCalcParser parser = new SimpleCalcParser(tokens);
            String eval = parser.is();
            System.out.println(eval);
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}

Modifying bits of the program above would allow the input to be obtained from the console. So my neighbor types in ‘IS 20 BIGGERTHAN 19′ and is happy with the result. Now he wants me to make a language that will print multiplication tables. His homework is due this week

Google recently donated a code profiling product to eclipse. A related code analytics product named CodePro is worth a look. It integrates with eclipse nicely. Lets take it for a test drive

Code review:

Hello world

package com.test;
 
public class HelloWorld
{
 
    public static void main(String... args)
    {
        new HelloWorld().go(11);
    }
 
    public void go(int i)
    {
        System.out.println("Hello world");
        if(i==10)
        {
            throw new RuntimeException();
        }
        System.out.println(i);
    }
}

Codegened test cases:

    @Test
    public void testGo_1()
        throws Exception
    {
        HelloWorld fixture = new HelloWorld();
        int i = 1;
 
        fixture.go(i);
 
        // add additional test code here
    }
 
    @Test(expected = java.lang.RuntimeException.class)
    public void testGo_2()
        throws Exception
    {
        HelloWorld fixture = new HelloWorld();
        int i = 10;
 
        fixture.go(i);
 
        // add additional test code here
    }

There are commercial eclipse plugins that do the same job. With CodePro this comes as one of features that can be used.

This gets pretty interesting. There are several factors involved in writing readable code that are seldom measured. Number of lines of code per method is one of them. If you have ever had the misfortune of analyzing a method with 1200 lines of code (yes it happened to me ), you would love to have this metrics tool. It helps catch violations like this quickly. The metrics that I found most useful were

Jettison is a library that provides a streaming parser for the JSON data format. Reading that sentence again makes you want to compare it to Stax, the streaming parser for XML. Jettison provides implementation classes that inherit from the same interfaces that a caller would use for XML reading / writing. Diving into some code reveals how JSON is written using this API…

    public void go()
    {
        try
        {
            MappedNamespaceConvention con = new MappedNamespaceConvention();
            XMLStreamWriter writer = new MappedXMLStreamWriter(con, new PrintWriter(System.out));
            writer.writeStartDocument();
            writer.writeStartElement("alice");
            writer.writeCharacters("bob");
            writer.writeEndElement();
            writer.writeEndDocument();
            writer.flush();
            writer.close();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }

Output: {“alice”:”bob”}

Notice that the XMLStreamWriter is the one in the javax.xml.stream package. The underlying data format that is written using this interface is abstracted. You could use the XMLStreamWriter to write a XML document for example. A XMLStreamReader can be used in the same way to read a XML or JSON document.

This style is quite beneficial especially for those that write frameworks / APIs. Axis2 supports JSON using jettison. The amount of code changes necessary to support a new format would be minimal, if the code is already modularized into chunks and the interfaces used are the same. Of course we make the assumption that the data formats in question can be supported using a common set of callback methods / events.

While this style allows support for a new format with minimal code changes, the interfaces / method names / events can get in the way to create confusion. What business does one have writing JSON from an interface called XMLStreamWriter ? None really. The confusion that the interface and implementation can cause is also sizable. However after working with jettison for a while and reaping some of its benefits, at least for me, the benefits outweighed the minor drawbacks.

Every once in a while, there comes a need to convert an OutputStream into an Inputstream. The PipedInputStream and PipedOutPutstream classes allow one to achieve this. The idea behind the piped stream is that at one end of the pipe, a writer thread writes to a PipedOutputStream. A PipedInputStream thread concurrently reads whatever is written on the other side. Here is an example use case…

Assume that a thread streams random words and another thread needs to read these words and write them to System.out. Lets call the producer of the words, a DataSource, and the client the DataConsumer (which is the consumer of the producer). The DataConsumer has a problem. Some of the words from the DataSource are deemed inappropriate by the DataConsumer. The consumer would rather receive ‘***’ than the inappropriate word.

What would be great, is if we can pass the stream that is generated from the source to a middle-man which will filter out these words and pass them across to the consumer. This is what the DataSource and DataConsumer look like. The DataSource structures the data using XML, not that it is necessary.

DataSource and DataConsumer:

package com.pipe;
 
import java.io.OutputStream;
 
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamWriter;
 
public class DataSource implements Runnable
{
    private OutputStream outputStream = null;
    private String [] words = {"crack","hi","you","what","snort","me"}; 
    private XMLStreamWriter writer = null;
 
    public DataSource(OutputStream stream)
    {
        outputStream = stream;
    }
 
    @Override
    public void run()
    {
        try
        {
            XMLOutputFactory output = XMLOutputFactory.newInstance();
            writer = output.createXMLStreamWriter( outputStream );
            writer.writeStartElement("root");
            for (int counter = 0; counter < 100000; counter++)
            {
                int result = (int)(Math.random()*10) % 6;
                String word = words[result];
                writer.writeStartElement("word");
                writer.writeCharacters(word);
                writer.writeEndElement();
            }
            writer.writeEndElement();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
        finally
        {
            try
            {
                outputStream.flush();
                outputStream.close();
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
        }
    }
}

package com.pipe;
 
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
 
public class DataConsumer implements Runnable 
{
    private InputStream inputStream=null;
 
    public DataConsumer(InputStream inputStream)
    {
        this.inputStream = inputStream;
    }
 
    @Override
    public void run()
    {
        try
        {
            BufferedReader bufferedReader = new BufferedReader( new InputStreamReader( inputStream ));
            String temp=null;
            while((temp=bufferedReader.readLine())!=null)
            {
                System.out.println(temp);
            }
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }
}

Firstly, the producer and consumer cannot be connected without a pipe. The producer streams data to an OutputStream while the consumer consumes it through its InputStream. They can be connected to each other by passing one end of the pipe to each class.

int BUFFER = 2048;
 
PipedInputStream convertPipe = new PipedInputStream(BUFFER);
PipedOutputStream dataPipe = new PipedOutputStream(convertPipe);
 
DataSource dataSource = new DataSource(dataPipe);
DataConsumer dataConsumer = new DataConsumer(convertPipe);

Reading with a pipe:

When the PipedOutputStream is instantiated, it ‘connects’ with the PipedInputStream. Which is a fancy way of saying, data that is streamed to the PipedOutputStream will be passed on to the PipedInputStream. This is achieved by storing the data in an internal byte[] buffer, whose size can be chosen when instantiating the PipedInputStream.

It is also possible to insert another thread in the middle by attaching one more pipe. The WordWatcher thread, reads from one end of the pipe (PipedInputStream), filters out objectionable words, and writes data to the other end of the pipe(PipedOutputstream). It uses StaX to parse whatever was written (again, not that it matters)

WordWatcher:

package com.pipe;
 
import java.io.InputStream;
import java.io.OutputStream;
 
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamConstants;
import javax.xml.stream.XMLStreamReader;
 
public class WordWatcher implements Runnable
{
    private InputStream inputStream;
    private OutputStream outputStream;
 
    public WordWatcher(InputStream inputStream, OutputStream outputStream)
    {
        this.inputStream = inputStream;
        this.outputStream = outputStream;
    }
 
    @Override
    public void run()
    {
        try
        {
            XMLInputFactory factory = XMLInputFactory.newInstance();
            XMLStreamReader reader = factory.createXMLStreamReader(inputStream);
            while (reader.hasNext())
            {
                int next = reader.next();
                switch (next)
                {
                case XMLStreamConstants.CHARACTERS:
                {
                    String text = reader.getText();
                    if(text.equals("crack"))
                    {
                        text = "*****";
                    }
                    outputStream.write(text.getBytes());
                    break;
                }
                }
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
        finally
        {
            try
            {
                outputStream.flush();
                outputStream.close();
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
        }
    }
}

Multiple pipes:

The following code connects all the pieces. It creates 2 pipes and passes them along to three threads. The threads read and write to these pipes transparently without knowing what implementation class the InputStream and OutputStream belong to.

Connecting the dots:

package com.pipe;
 
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
 
public class Start
{
 
    public static void main(String... args)
    {
        new Start().go();
    }
 
    public void go()
    {
        try
        {
            long past = System.currentTimeMillis();
            int BUFFER = 2048;
 
            PipedInputStream convertPipe = new PipedInputStream(BUFFER);
            PipedOutputStream dataPipe = new PipedOutputStream(convertPipe);
 
            PipedInputStream convertedWordPipe = new PipedInputStream(BUFFER);
            PipedOutputStream outputPipe = new PipedOutputStream(convertedWordPipe);
 
            DataSource dataSource = new DataSource(dataPipe);
            WordWatcher pipe = new WordWatcher(convertPipe,outputPipe);
            DataConsumer dataConsumer = new DataConsumer(convertedWordPipe);
 
            Thread producerThread = new Thread(dataSource);
            Thread pipeThread = new Thread(pipe);
            Thread consumerThread = new Thread(dataConsumer);
 
            producerThread.start();
            pipeThread.start();
            consumerThread.start();
 
            consumerThread.join();
 
            long now = System.currentTimeMillis();
            System.out.println("Time taken: " + (now-past) + " ms");
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}

Gotchas:

Flush and close, lest you want a ‘Write end dead’:

When one end of the stream is done writing data, always remember to flush and close the output stream. Failure to do this will result in an IOException with the very helpful message “Write end dead”. The exception is supposed to imply that the thread that was at the end of the pipe writing the data, is now dead. Talk about cryptic messages !

Taking care of the pipeSize buffer:

In the programs above, try reducing the buffer size, say to 16 instead of 2048. Doing this will make the program slam the brakes. In the real world, there can be a latency between a read from one end and a write to the other end of the pipe. If the buffer is not large enough, the pipe will wait for a read before it can fill in the next chunk of data written by the OutputStream. That will inevitably slow down the program.

Any system that logs vast amounts of information, needs to think about performance. The activity of logging cannot be a synchronous blocking call that returns only when the message has been logged to a persistence store. Enterprise logging systems usually make use of a message bus to carry messages asynchronously to their target persistence store. Be it a database or a file.

Talking about logging brings us to System.out.println() (Lets call is SOP for short). It is a surprisingly commonly method to “log” messages. SOP is not meant to be used as a logging system, but unfortunately there is no dearth of projects that have these statements scattered around the code base. The adverse effects that this statement can bring on the performance of the system is often not recognized as well as it should be.

Why is SOP a bottleneck for performance ? This is why…

Code excerpt from PrintStream.java:

    private void write(String s)
    {
        try
        {
            synchronized (this)
            {
                ensureOpen();
                textOut.write(s);
                textOut.flushBuffer();
                charOut.flushBuffer();
                if (autoFlush && (s.indexOf('\n') >= 0))
                    out.flush();
            }
        }
        catch (InterruptedIOException x)
        {
            Thread.currentThread().interrupt();
        }
        catch (IOException x)
        {
            trouble = true;
        }
    }

All SOP calls on a String result in a corresponding call to the private write(String s) method in PrintStream.java. The synchronized block ensures that every thread has to wait for this call to end before it can proceed. Whats more, the calls to flushBuffer() on the BufferedWriter that is textOut result in the execution of more synchronized blocks. These blocks are expensive to execute.

Here is a chart that shows how performance degrades when a program logs 100,000 messages through various threads. The rate of degradation also depends on the number of characters that are passing through the stream.

System.out.println performance:

The degradation experienced in ‘Web application X / Y’ may vary, but it cannot be discounted. Avoid using SOPs to log messages in your app. Even one or two that are left over can harm performance under the right conditions. Let us also not forget about e.printStackTrace() and other forms of writing to console output that follow the same synchronized pattern.

While coming up with APIs or a framework, design forms a crucial part of the end result. While a solid design does indeed allow for flexible usage of an API, usability goes hand in hand with the design. What better way is there to create an API, than to learn from the mistakes and successes of other developers first ? Lets take a look at some examples in the java API where things could have been better…

Stack and Properties:

A java.util.Stack provides a standard Stack implementation in the java language. But did you know it extend a java.util.Vector ? Why is this wrong ? Because you can do this

package com.design;
 
import java.util.Stack;
import java.util.Vector;
 
public class StackTests
{
 
    public static void main(String... args)
    {
        new StackTests().go();
    }
 
    public void go()
    {
        Vector<String> vector = new Stack<String>();
        vector.add("first");
        Stack <String> stack = (Stack <String>) vector;
        String peek = stack.peek();
        System.out.println(peek);
 
        stack.push("next");
        stack.push("item");
        String remove = vector.remove(2);
        System.out.println("Removed: " + remove);
        peek = stack.peek();
        System.out.println(peek);
    }
}
 
Output:
 
first
Removed: item
next

There are so many things that are going on there that are wrong.

1. Vector is a slow DS. Its methods are synchronized and there is no way around this right now since this contract has been sealed. It cannot be changed without changing backward compatibility.
2. Vector has exposed its own methods to a stack. You should not be able to add() on to a stack.
3. Typecasting between Vector / Stack is plain weird.
4. Since the contract methods exposed by both classes operate on the same internal data structure, the outcome can often be confusing.

This is something the API folks got wrong. This is not the only class that should have used composition over inheritance. The java.util.Properties class is another example. It extends from HashTable and now there is no way around that.

Dummy variables:

This one is probably not well known. A HashSet() has a default constructor which takes a boolean variable called dummy. It is never used. So why does it exist ? Its there because a LinkedHashSet (which extends the HashSet) uses this constructor to tell its super class that the backing data structure should be a LinkedHashMap. The advantage of doing this is that LinkedHashSet will have very little code to house into its implementation.

From HashSet.java:

    /**
* Constructs a new, empty linked hash set.  (This package private
* constructor is only used by LinkedHashSet.) The backing
* HashMap instance is a LinkedHashMap with the specified initial
* capacity and the specified load factor.
*
* @param      initialCapacity   the initial capacity of the hash map
* @param      loadFactor        the load factor of the hash map
* @param      dummy             ignored (distinguishes this
*             constructor from other int, float constructor.)
* @throws     IllegalArgumentException if the initial capacity is less
*             than zero, or if the load factor is nonpositive
*/
HashSet(int initialCapacity, float loadFactor, boolean dummy) {
    map = new LinkedHashMap(initialCapacity, loadFactor);
}

It is not a great idea to write a separate constructor just because one sub class might need it. Secondly the javadoc explicitly mentions that only a LinkedHashSet uses this constructor. That message is a little confusing. Does that mean other subclasses are not meant to use this constructor ? Why does a subclass even need to know that ?

The side effect of this decision is not far reaching, as you may have already noted. Had the constructor been protected, the consequences would have been worse.

Usability:

The java community is pretty awesome. When a solution is too complicated, folks come out with a new framework / API to combat it. Take EJB for example. Versions 1 and 2 were pure horror. Version 3 is a little better. JEE6 has also learned a little from Spring.

This brings us to the java Date and its formatting APIs. Usability wise, many deprecated methods and confusing constructors make for a difficult time using the API. Which is why other solutions like Joda time and Apache commons date utils pop up. The ‘Why Joda Time?’ section helps explain what problem the API intends to solve where the JDK fell short.

In general, it will be hard to judge an API / framework without letting a few developers toy around with it first. The methods / constructors may well do what the contract defines them to do, but if it takes 100 steps against 10 to reach from point A to point B, guess which route a developer wants to take ?

In conclusion, these points are not about nitpicking the JDK / Java APIs in any manner. As long as developers learn from these mistakes, they can avoid them in their own code. The JDK is abundant with good usages of design patterns and some of its API are well thought out. Just avoid the bad design decisions whenever possible

This has been a sad week for java. One of the string of bad things happening to it, is the abandonement by Mac OS X. Among the interesting thoughts that have been thrown out there for a possible rescue plan, this one seemed to stand out.

In a nutshell, the link points to a post by Adam-Bien suggesting that users might pay for a JVM on the Mac, thereby opening up a business opportunity. That might be far fetched considering

1. Apple wants to be a control freak of sorts. Anything that is not objective-C is a target
2. Folks usually have a very hard time convincing themselves to pay for something that they used to get for free.
3. It would be pretty difficult to write a JVM targeted for the Mac from scratch, given that many native API calls are secrets.

But who knows. Why dont you be the judge and tell us what you think. Would you shell out some dough for a JVM on the Mac ? Take a small poll. The comments on Adam’s post seemed to swing both ways, so perhaps a poll will enumerate what people feel.

Every developer out there has to troubleshoot a problem in PROD every now and then. The series of steps goes something like this

1. User notices something fishy
2. Clicks on a button that reads ‘Do not click here’
3. Boom !
4. Developer tries to figure out what happened by fishing through log files.

Now step 4 can be relatively more simple than it needs to be. It is often complicated by bad practices or obstacles like the ones mentioned below

1. Developers log all application logs into a central server log
2. Access to logs are restricted by a ‘log viewing’ application.

Instead of fishing around for errors, would it not be cool if you could visualize the errors that occur in your application through some sort of timeline ? After investigating a couple of frameworks that would allow one to do this, I figured ‘Simile Timeline‘ would be a good fit.

Timeline is a framework that allows you to view a series of events through well… a timeline. You can write a page that uses the timeline JS to view events that occur over time. Events can be fed to this framework using a XML. So here is what you can do to visualize logs for your application and get to the error real quick

1. Write an application (batch ?) to stream though your log files and build an event XML for each log file
2. Use a small web application to load this event file and display it to the developer

Now, all the developer has to do is to scroll through the event timeline to figure out what went wrong and when it happened. Enough talk. Go ahead and check out how application errors in logs can be displayed through simile timeline.

You might also be interested in the event XML that helped display the timeline. Most production environments will retain logs for a while and then discard them. The presence of a XML describing all the errors that happened say a year back, can prove to be useful. That way when you scratch your head thinking ‘Hmm… did we not encounter this same problem 3 months back ?’ you can scroll back in time and quickly verify it. Archiving these XML files is easier since they will not exceed more than a few KB in size.

Timeline generation:

For those that want to play around with the framework itself, feel free to take a look at the HTML source of the example page. The project wiki is also helpful.

It was a normal day. The sun rose, birds were singing and everything seemed to be going fine until Kaboom ! A bug appeared in the live environment. Things turned downhill from there that day. But we are more interested in how things got fixed so lets focus on that.

An investigation of the code revealed that the bug had crept in because of a minor mistake. A variable was referring to the wrong column index on a table. ‘Hmmm… simple fix’ I thought. I told my manager it would not take more than 5 minutes to do. ‘Its a one line fix’. Mistake.

There were loads of other things to do after fixing this bug. This is what I ended up doing over the next 4 hours.

• Analyze if any other code flow would explode.
• Code the fix – This only took 5 minutes as expected.
• Move fix to DEV
• DEV server crashed. Great. Fix this and get DEV up and running.
• Test fix
• Export the old data before the fix
• Export the new data after the fix
• Compare new data with old data and verify the result for each column (Wrote a small program to do this).
• Move fix to PROD
• Close the bugzilla bugs and fill in a lot of bug details
• Inform PROD users about the fix

So ! There we are. All done. Loose ends tied. But at the end of the day, I had not set the right expectations when I went about fixing this problem. When you say things like ‘It will be done in 5 minutes’ and ‘It is a one line fix’, you need to remember that to an end user / manager that might sound something like ‘It will start working again after 5 minutes’. But that is not the case. Well… a manager with a technical background would be more understanding but that is not my point.

Sometimes there are things you cannot put down in writing when you provide an estimated time to fix stuff. I find it is always better to say ‘Sometime by Tuesday noon’ instead of ‘One line code change / 5 minutes’. It sets the right expectation and you don’t have people asking you ‘Its not done yet !?’. Explaining your work is sometimes more important than the work itself.

An article was written a while back about how neural networks can be used to classify source code. Yes the source code that you write to feed to compilers / interpreters.

The article explains at a high level what method could be used to perform this activity. In the end the author claims some level of success and wonders how other neural-network implementations / techniques would solve the same problem. This got me curious enough and I spent a weekend trying to crack this with Neuroph, the neural network library for java. I present to you my analysis and results below. For the impatient here is the code-classifier DEMO.

Why solve this problem with neural networks ?

So where do we begin ? How about the question ‘Why do we need neural networks to solve this problem ?’. Well actually it might not be the ‘best’ way to solve the problem. A lexer is a better solution to this problem since it can scan the source file and probably give you a much better accuracy on the clasification. But neural networks are also great for this problem since they can keep learning based on the input that is provided. Add one more programming language into the bag and the network can learn about this language without a programmer needing to type extra code into a lexer.

So how do we do it ?

One of the features that differentiates one source code type from another is the keywords used by the programming language in question. So lets target that. Our neural network will try to identify if source code belongs to the java language or the python language.

It takes as its input a list of all keywords available in both languages. Mind you that there may even be some overlap between the keywords. For example the keyword ‘if’ is used in both languages. But there may be keywords that are used solely in one language alone.

Neural network viewed in graph mode: (The network is too huge to fit into this picture )

A middle layer will contain a few nodes that will back propagate anything that it learns. The output layer (just one node) will decide if the language is java / python. The value will range from 0-1, where 0 represents closeness to python and 1 represents closeness to java.

How do we train it ?

The process of training is made easy by the neuroph API. First we need a few java / python files. A program was written that reads each file and splits it based on the whitespace delimiter. Based on a known set of keywords, the network is fed the number of times a keyword appears in a source file.

for(Map<Double, Double> map : keywordStrength)
{
    Collection<Double> values = map.values();
    Vector<Double> input = new Vector<Double>();
    input.addAll(values);
    SupervisedTrainingElement supervisedTrainingElement = new SupervisedTrainingElement(input,output);
    trainingSet.addElement(supervisedTrainingElement);
}

For example, recognized keywords may be ‘public’ represented by the input index 0 and ‘static’ represented by the input index 1.  A java file may contain the word pubic 10 times and static 5 times. The input to the network in this case would be 10 5. We can now use these numbers to tell the network ‘Hey if you see this pattern it is likely that the source code belongs to language X’. Neat !. Once the network is trained you can load it and use the same process to classify source code for any given input String.

Well… show us some stats:

Before I say anything else, I did not expect this approach to succeed more than 10-20% of the time. This is because the number of files that I trained this network with was around 20. Java + Python included. But I was surprised to see the network classifying the data correctly most of the time. There were very few cases where it failed.

The real test for this network is now in your hands. I made a java web app on top of this ‘code classifier’ application and built a simple UI using Jquery. You can type in any code / junk you want and hit the trained network. If it manages to classify your code correctly at least 60% of the time, I would consider that an enormous victory (for Neuroph ?)

So what are you waiting for, take her for a spin.

How do I game the results:

This is pretty simple. There are certain rules you must follow. Not following them will give you weird results

1. Make sure that your code does not contain comments. I do not strip comments from code before analyzing them for keywords. Including java like keywords in a python file comment will confuse the network.
2. Input only those files that will compile / run correctly on Java / Python. In theory you could feed in a Shakespeare novel to my network and it will still try to classify it as source code

So does the network work for you ? Feel free to leave a comment.

I also wrote a simpler tutorial for Neuroph that is less practical and more academic. Check it out if it interests you