Thursday, August 9, 2012

Microsoft Excel and BOM character

BOM Character? What is that? What does it have to do with Microsoft Excel?

Yes, I know a lot of questions do arise in the title of the blog itself.
This blog is to tell a story about how file encoding and its interpretations by Microsoft Excel can cause big issues. In one of the project, we had to create CSV files. CSV files are really simple to construct. The columns are separated with commas and the rows are separated with newline characters. Our development machines are mostly Linux. During development, we used Open Office Excel to open the document and check the validity of the data. All the data would show perfectly. It did not matter what language is used to create the file. During the whole development cycle we did not think we would spend 2 weeks on solving a mystery why the same file contains gibberish characters when opened with Microsoft Excel but Open Office Excel would shows everything nice and dandy. Our Quality Assurance guy documented the issue, I almost did not want to believe him but eventually I saw the truth in his tests. Now the mystery had to solved.

Even though we have built CSV files before this project, XML/XSL transformations (some proprietary stuff) took care of encoding. After doing lots of google search and forum reads, we found a good stackoverflow discussion that had valuable discussion about MS-Excel and CSV files. -  . There are a lot of ways suggested on this page, we tried many of them but only one finally worked. Adding Byte Order Marker character to the first row of the file. We did a small experiment by testing only file creation related code and found that file could be read successfully in Microsoft Excel.

Giveaways:

A] Two absolute great readings -

1) Joel Spolsky's blog on character encoding.
2) BOM Character Wikipedia Entry

B] Write a small and specialized class that only solves given problem, this reduces test time.

Next time you have to construct a CSV file which may/may not have foreign characters, use BOM :)

An easy way to find Quartz scheduler queries

We had a very interesting issue on production couple of weeks ago. This issue entailed the alert email to all the DBA's in the team. The issue was related to

ORA 03137 TTC protocol internal error

The error email contained the query but we had not written this query so we did not have much idea about it. Even though investigation of the issue is a whole new topic of blog, this blog is to tell you a small trick to find the query in Quartz scheduler code base. I had to find following query in the code base.


SELECT TRIGGER_NAME, TRIGGER_GROUP, NEXT_FIRE_TIME, PRIORITY FROM 
{0}TRIGGERS WHERE TRIGGER_STATE = ? AND NEXT_FIRE_TIME < ? AND 
(NEXT_FIRE_TIME >= ?) ORDER BY NEXT_FIRE_TIME ASC, PRIORITY DESC



Quartz scheduler constructs the queries on the fly so one can never find it easily in their code base. Even though one looks at their files called as StdJDBCConstants.java, one may not be able to read and pick through it. This file contains most of the queries for standard databases.

After some frantic search, I used the first part of the query to grep on the entire source code (actual source code and related documentation).

grep -lir "SELECT TRIGGER_NAME, TRIGGER_GROUP" *



Finally the search came up with the result. Now I could understand the context of the query and then could investigate the issue further on.. Grep saved the day!
docs/api/constant-values.html

Tuesday, July 24, 2012

Introduction to GSON





I know I am probably adding one more introduction to thousands of articles about GSON. I learned about GSON with my own experiment. I wrote this document for my colleagues so that they will have easy time learning one feature at a time. I am posting this to web so that everybody else can also benefit from it. If you like it, please let me know. If you don't like it, feel free to suggest improvements. Please note that this is not production ready code.


What is Gson?



Gson is a Java library that can be used to convert Java Objects into their JSON representation. It can also be used to convert a JSON string to an equivalent Java object. Gson is an open-source project hosted at http://code.google.com/p/google-gson.

Gson can work with arbitrary Java objects including pre-existing objects that you do not have source-code of.



Understanding Gson





 Simple object representation in JSON and from JSON



Program







import java.io.BufferedReader;

import java.io.File;

import java.io.FileReader;

import java.io.FileWriter;

import java.io.IOException;

import com.google.gson.Gson;

/**

 *

 * This simple class does two things, it serializes a java object Person into JSON representation and then deserializes the JSON string from file into * Person object.

 * @author Shilpa Deshpande

 */

public class GsonTest {

    private static final String FILE_LOCATION = "./file_" + System.currentTimeMillis() + ".json";

    public static void main(String[] args) {

        Person person = new Person("Jane", "Doe", 35);

        Gson gson = new Gson();

        String json = gson.toJson(person);

        // Print JSON representation

        System.out.println("JSON Representation: \n"+json);

        // Write converted json data to a file named "file_somenumber.json"

        File file = new File(FILE_LOCATION);

        try {

            if (!file.exists()) {

                file.createNewFile();

            }

            FileWriter writer = new FileWriter(file);

            writer.write(json);

            writer.close();

          }catch (IOException e) {

              e.printStackTrace();

         }

        // Read file and then convert it into object

        try {

            BufferedReader br = new BufferedReader(new FileReader(FILE_LOCATION));

            // convert the json string back to object

            Person jperson = gson.fromJson(br, Person.class);

            // Print object representation.

            System.out.println("Object Representation: \n"+jperson);

        } catch (IOException e) {

            e.printStackTrace();

        } finally {

              if (file.exists()) {

              file.deleteOnExit();

         }

  }

    }

public static class Person {

        private String firstName;

        private String lastName;

        private int age;

        /**

         * We need to have no-arg constructor

         */

        public Person() {

            super();

        }

        public Person(String firstName, String lastName, int age) {

            this.firstName = firstName;

            this.lastName = lastName;

            this.age = age;

        }

        /**

         * @see java.lang.Object#toString()

         */

        @Override

        public String toString() {

            StringBuilder builder = new StringBuilder();

            builder.append("Person [First Name=" + firstName + ", Last Name=" + lastName + ", Age=" + age + "]\n");

            return builder.toString();

        }

    }

}





Output:




The output will be
JSON Representation:
{"firstName":"Jane","lastName":"Doe","age":35}
Object Representation:
Person [First Name=Jane, Last Name=Doe, Age=35]


Observations:


One should have a no-arg constructor for the class which is being serialized.
One may or may not have getters and setters for the class.


Adding a list of objects to the Person.



Program






public static class Place {

        private String cityName;

        private State stateName; // Use enum

        public Place() {

            super();

        }

        public Place(String cityName, State stateName) {

            this.cityName = cityName;

            this.stateName = stateName;

        }

        /**

         * @see java.lang.Object#toString()

         */

        @Override

        public String toString() {

            return "Place [City=" + cityName + ", State=" + stateName + "]\n";

        }

    }



    public static enum State {

        AK,         AL,         AR,         AZ,         CA,         CO,         CT,         DC,         DE,         FL,         GA,         HI,         IA,         ID,         IL,         IN,         KS,         KY,         LA,         MA,         MD,         ME,         MI,         MN,         MO,         MS,         MT,         NC,         ND,         NE,         NH,         NJ,         NM,         NV,         NY,         OH,         OK,         OR,         PA,         RI,         SC,         SD,         TN,         TX,         UT,         VA,         VT,         WA,         WI,         WV,         WY

    }


The test code will change in following manner


public static void main(String[] args) {

        List placeList = new ArrayList();

        Place place = new Place("Salt Lake City", State.UT);

        placeList.add(place);

        place = new Place("Lincoln", State.NE);

        placeList.add(place);

        Person person = new Person("Jane", "Doe", 35);

        person.setPlaceList(placeList);

     .............

}




Output



The output will be
JSON Representation:
{"firstName":"Jane","lastName":"Doe","age":35,"placeList":[{"cityName":"Salt Lake City","stateName":"UT"},{"cityName":"Lincoln","stateName":"NE"}]}
Object Representation:
Person [First Name=Jane, Last Name=Doe, Age=35]
Place [City=Salt Lake City, State=UT]
Place [City=Lincoln, State=NE]

Observations




  1. One can have embedded objects, as long as every object has no-arg constructor, deserialization works fine.
  2. One can use enum as a member of class.



Use of interfaces



When member of the object is a interface, it is important to let Gson know what impl is going to be used. Following changes in program are required to work with interfaces


Program




public static void main(String[] args) {



        JobType job = new JobTypeImpl("Application Developer","Create software application");

        List placeList = new ArrayList();

        Place place = new Place("Salt Lake City", State.UT);

        placeList.add(place);

        place = new Place("Lincoln", State.NE);

        placeList.add(place);

        Person person = new Person("Jane", "Doe", 35);

        person.setPlaceList(placeList);

        person.setJobType(job);

        # Create an implementation of InstanceCreator

        class JobTypeInstanceCreator implements InstanceCreator {

            @Override

            public JobType createInstance(Type arg0) {

                return new JobTypeImpl();

            }

          }



        # Register JobType adapter with GsonBuilder and then create instance of Gson.



        GsonBuilder builder = new GsonBuilder().registerTypeAdapter(JobType.class, new           JobTypeInstanceCreator());

        Gson gson = builder.create();

       public static interface JobType {

       public String getDescrption();

       public void setDescription(String description);

       public String getTitle();

        public void setTitle(String title);

    }

    public static class JobTypeImpl implements JobType {

        private String title;

        private String description;

        public JobTypeImpl() {

            super();

        }

        public JobTypeImpl(String title, String description) {

            super();

            this.title = title;

            this.description = description;

        }

        @Override

        public String getDescrption() {

            return description;

        }

        @Override

        public void setDescription(String description) {

            this.description = description;

        }

        @Override

        public String getTitle() {

            return title;

        }

        @Override

        public void setTitle(String title) {

            this.title = title;

        }

        /**

         * @see java.lang.Object#toString()

         */

        @Override

        public String toString() {

            return "JobTypeImpl [Title=" + title + ", Description=" + description + "]\n";

        }

    }





Output



JSON Representation:
{"firstName":"Jane","lastName":"Doe","age":35,"placeList":[{"cityName":"Salt Lake City","stateName":"UT"},{"cityName":"Lincoln","stateName":"NE"}],"job":{"title":"Application Developer","description":"Create software application"}}
Object Representation:
Person [First Name=Jane, Last Name=Doe, Age=35]
Place [City=Salt Lake City, State=UT]
Place [City=Lincoln, State=NE]
JobTypeImpl [Title=Application Developer, Description=Create software application]

Observations


One needs to use Implementation of InstanceCreator to specify exact implementation of the interface.


Use of third party classes


Let's assume Person class has another instance of org.joda.time.DateTime type. We do not have source code of this class, so we cannot modify it. Following piece of code shows how serialization and deserialization can work.
Program



        class DateTimeSerializer implements JsonSerializer {

            public JsonElement serialize(DateTime src, Type typeOfSrc, JsonSerializationContext context) {

                return new JsonPrimitive(src.toString());

            }

        }

        /*

         * Gson calls toJson() when it runs into a DateTime object during serialization. Writing a Deserializer Here is an example of how to write a custom deserializer for JodaTime DateTime class.

         */

        class DateTimeDeserializer implements JsonDeserializer {

              public DateTime deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {

                return new DateTime(json.getAsJsonPrimitive().getAsString());

            }

        }

        JobType job = new JobTypeImpl("Application Developer", "Create software application");

        DateTime date = new DateTime();

        List placeList = new ArrayList();

        Place place = new Place("Salt Lake City", State.UT);

        placeList.add(place);

        place = new Place("Lincoln", State.NE);

        placeList.add(place);

        Person person = new Person("Jane", "Doe", 35);

        person.setPlaceList(placeList);

        person.setJobType(job);

        person.setDate(date);



        GsonBuilder builder = new GsonBuilder().registerTypeAdapter(

                                              JobType.class, new JobTypeInstanceCreator());

        builder.registerTypeAdapter(DateTime.class, new DateTimeSerializer());

        builder.registerTypeAdapter(DateTime.class, new DateTimeDeserializer());

        Gson gson = builder.create();





Output



JSON Representation:
{"firstName":"Jane","lastName":"Doe","age":35,"placeList":[{"cityName":"Salt Lake City","stateName":"UT"},{"cityName":"Lincoln","stateName":"NE"}],"job":{"title":"Application Developer","description":"Create software application"},"date":"2012-03-30T13:01:18.284-05:00"}
Object Representation:
Person [First Name=Jane, Last Name=Doe, Age=35]
Place [City=Salt Lake City, State=UT]
Place [City=Lincoln, State=NE]
JobTypeImpl [Title=Application Developer, Description=Create software application]
2012-03-30T13:01:18.284-05:00

Observations



 One may need to use custom serializer, deserializer for third party classes.
Refer to the attached file for the complete program.

Ref:


Gson User Guide


How to send an email with attachment from command line in unix?


How to send the email with attachment from command line?


This is useful when one is working remotely and does not have access to any mailing programs other than command line.
Install following programs on Ubuntu as followed
sudo apt-get install mailx sendmail
sudo apt-get install sharutils

We can use following command to actually send the email. 

 [user@machine~]$ uuencode FileWithExtention FileWithExtention | mailx -s 'Subject' user_name@domain

This page provides more information.

Monday, July 23, 2012

String Concatenation Performance

Title:

Find string concatenation performance using different ways of string concatenation.

Experiment details

Following are the details of the environment.
  •  Operating System: Linux version 2.6.28-19-generic
  •  Processor
    •  model name   : Intel(R) Xeon(R) CPU            5140  @ 2.33GHz
    •  stepping         : 6
    •  cpu MHz        : 2327.593
    •  cache size       : 4096 KB
  •  JVM
    •  java version : "1.6.0_13"
  •  Memory:
    •  MemTotal:  3094916 kB

Method

  1.  Create an array of 5000/ 10000 String objects.
  2.  Calculate the start time before staring the concatenation operation.
  3.  Write a loop to concatenate all these objects using various ways (.concat method of String class, using StringBuffer, using StringBuilder, using + operator)
  4.  Calculate the end time after concatenation operation.
  5.  A Java program that was used to calculate the metrics is at the end of this document.

Terminology

Iterations - Number of times the process of concatenation is performed during the execution of the program.
Operations - Number of concatenation operations done (5000/10000).

Observations

Following charts provide data for two different set of experiment.
  1.  Multiple iterations - 10, 100, 1000 iterations
  2.  Multiple operations  - 5000/ 10000 operations

Here are the graphs that show the performance of each of the concatenation operation.




  1. It is observed that wrong usage of StringBuilder requires largest time to perform operation.
  2. Using + sign to concatenate performs second worst.
  3. Using String.concat() method performs relatively better than using + sign.
  4. Contradictory to traditional belief the usage of StringBuffer does not cause any issues. It performs as well as using StringBuilder.
  5. It is important to understand that StringBuilder should be used in one shot. One call to StringBuilder.append() method is the right way to concatenate strings.

Conclusion

  1.  StringBuilder and StringBuffer perform in the same way when used in the right fashion.
  2.  Wrong usage of StringBuilder can cause severe performance penalties.

Tip

  1. StringBuilder should be used when only one thread is going to perform concatenation operation. 
  2. StringBuffer should be used when multiple threads are going to perform concatenation operation.

Further Experiments

* Calculate memory usage while performing the same experiments.

Ref:


* StringBuffer

Program


Here is the program that I used to get above metrics. One needs to provide different arguments to this program while running to have different iterations. One can update the value of variable limit to change the number of operations. In this program, currently the limit is set to 5000.

import java.io.BufferedReader;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.util.ArrayList;


/**
 * 


 * @author Shilpa Deshpande
 * 


 */
public class StringConcatenationPerformanceMeasurer {

    private final int iteration;
    private final String fileName;

    class Measurement{
        private long concat, plus, wrongBuilder, rightBuilder, stringBuffer;
    
        
        /**
         * @return The concat
         */
        public long getConcat() {
        
            return concat;
        }
    
        
        /**
         * @param concat The concat to set
         */
        public void setConcat(long concat) {
        
            this.concat = concat;
        }
    
        
        /**
         * @return The plus
         */
        public long getPlus() {
        
            return plus;
        }
    
        
        /**
         * @param plus The plus to set
         */
        public void setPlus(long plus) {
        
            this.plus = plus;
        }
    
        
        /**
         * @return The wrongBuilder
         */
        public long getWrongBuilder() {
        
            return wrongBuilder;
        }
    
        
        /**
         * @param wrongBuilder The wrongBuilder to set
         */
        public void setWrongBuilder(long wrongBuilder) {
        
            this.wrongBuilder = wrongBuilder;
        }
    
        
        /**
         * @return The rightBuilder
         */
        public long getRightBuilder() {
        
            return rightBuilder;
        }
    
        
        /**
         * @param rightBuilder The rightBuilder to set
         */
        public void setRightBuilder(long rightBuilder) {
        
            this.rightBuilder = rightBuilder;
        }
    
        
        /**
         * @return The stringBuffer
         */
        public long getStringBuffer() {
        
            return stringBuffer;
        }
    
        
        /**
         * @param stringBuffer The stringBuffer to set
         */
        public void setStringBuffer(long stringBuffer) {
        
            this.stringBuffer = stringBuffer;
        }
    }

    /**
     * @param fileName 
     * @param iteration 
     * 
     */
    public StringConcatenationPerformanceMeasurer(int iteration, String fileName) {
        this.iteration = iteration;
        this.fileName = fileName;

    }

    /**
     * @param args
     */
    public static void main(String[] args) {
        
        int iteration = 10;
        String fileName = "StringConcatenationMeasurement.xls";
        try {
            System.out.println("Please enter number of iterations.");
            BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
            String readLine = in.readLine();
            iteration = Integer.valueOf(readLine);          
        }
        catch (IOException e) {
            e.printStackTrace();
        }
        StringConcatenationPerformanceMeasurer instance = new StringConcatenationPerformanceMeasurer(
                iteration, fileName); 
        try {
            instance.checkStringConcatenationPerformance();
            
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * @throws Exception 
     * 
     */
    public void checkStringConcatenationPerformance() throws Exception {
            
        ArrayList measurementList = new ArrayList();
        for (int k = 0; k < iteration; k++) {
            int limit = 5000;
            String[] array = new String[limit];
            for (int i = 0; i < limit; i++) {
                array[i] = "e" + i + " ";
            }
            // Native String Concatenation using concat method
            long t0 = System.currentTimeMillis();
            String sbS = "";
            for (int i = 0; i < limit; i++) {
                sbS = sbS.concat(array[i]);
            }
            long t1 = System.currentTimeMillis();
            //System.out.println("Final String using concat " + sbS);
            sbS = null;
            
            
            // Wrong String concatenation using StringBuilder
            StringBuilder sb = new StringBuilder();
            sbS = "";
            for (int i = 0; i < limit; i++) {
                sbS = (sb.append(sbS).append(array[i])).toString();
                sb = new StringBuilder();
            }
            long t2 = System.currentTimeMillis();
            //System.out.println("Final String - wrong usage of StringBuilder " + sbS);
            sbS = null;
            
            
            // Right String concatenation using StringBuilder
            sb = new StringBuilder();
            for (int i = 0; i < limit; i++) {
                sb.append(array[i]);
            }
            sbS = sb.toString();
            //System.out.println("Final String - right usage of StringBuilder " + sbS);
            long t3 = System.currentTimeMillis();
            sbS = null;
            
            
            // String concatenation using +
            sbS = "";
            for (int i = 0; i < limit; i++) {
                sbS = sbS + array[i];
            }
            long t4 = System.currentTimeMillis();
            //System.out.println("Final String using + " + sbS);
            sbS = null;
            
            
            // String concatenation using StringBuffer
            StringBuffer buffer = new StringBuffer();
            for (int i = 0; i < limit; i++) {
                buffer.append(array[i]);
            }
            sbS = buffer.toString();
            //System.out.println("Final String using StringBuffer " + sbS);
            long t5 = System.currentTimeMillis();
            sbS = null;
            
            
            System.out.println("=================================");
            System.out.println("Native String Concatenation using concat method (Time in Milliseconds) " + (t1 - t0));
            System.out.println("Wrong String concatenation using StringBuilder (Time in Milliseconds) " + (t2 - t1));
            System.out.println("Right String concatenation using StringBuilder (Time in Milliseconds) " + (t3 - t2));
            System.out.println("String concatenation using + (Time in Milliseconds) " + (t4 - t3));
            System.out.println("String concatenation using StringBuffer (Time in Milliseconds) " + (t5 - t4));
            System.out.println("=================================");
            
            Measurement mm = new Measurement();
            mm.setConcat((t1-t0));
            mm.setWrongBuilder((t2-t1));
            mm.setRightBuilder((t3-t2));
            mm.setPlus((t4-t3));
            mm.setStringBuffer((t5-t4));
            measurementList.add(mm);
            
        }      
    }    
}


.

Wednesday, March 30, 2011

HttpFox

A few months ago, I worked on a project where the request originating on browser was to be transferred to a whole different server farm. The project involved Apache webserver, GWT, Tomcat servers, of course some configuration in each of these layers. The Firefox Extention that really helped me was - HttpFox. This extension allows us to see the originating request. In the case of GWT, one can see the request type, its parameters. We can observe the response from the server. Sometimes just the response code is more helpful than any logs on the server side. One can see the URL that we are hitting. In our case, it was crucial to construct right URL so that we know we reach to the right farm of servers.


I used the same extension two months ago when I had to reverse engineer a working URL. I needed to find how the URL is generated and what does it translate to. This extension was really really helpful. I had finished debugging in 15 minutes from the time I started. If you are using Firefox 3.6 and cannot download this extension using firefox, try downloading it manually. It worked for me.

Next time you have to develop/reverse engineer anything that deals with URL's, fire up this one. Good luck!

Tuesday, June 8, 2010

GWT Canvas

In my recent adventure about using GWT canvas to draw some images at work lead me to one discovery. Lets assume that we are trying to draw a polygon, so we will use following code

canvas = new GWTCanvas(500,500);
canvas.setFillStyle(color);
canvas.beginPath();
canvas.moveTo(100, 500);
canvas.lineTo(150, 400);
canvas.moveTo(150, 400);
canvas.lineTo(450, 400);
canvas.moveTo(450, 400);
canvas.lineTo(500, 500);
canvas.moveTo(500, 500);
canvas.lineTo(100, 500);
canvas.closePath();
canvas.stroke();
canvas.fill();


This will draw the polygon but will not fill it with the Color that we have set as fill style. If one comments out the lines corresponding to moveTo() method, the fill() method works and fills the polygon with the color that we have specified.

Wanted to share.
Cheers!!