Notes from Life Science Software Development

caBIO iPhone Application 1.1

2009-08-10T15:41:00.003-04:00

The 1.1 update for the caBIO iPhone App is now live on the App Store. This release fixes a bug with the cached "brca1" search which is loaded when the app first starts. In addition, the interface has been improved with result caching, a refresh button, and new toolbar icons.

caBIO iPhone Application 1.0

2009-06-29T22:03:00.006-04:00

I've been working on an iPhone App for the caBIO data service and just submitted it to the App Store for review. This site will serve as the "support site" which Apple requires each app to have.

Update (7/18/2009): The application is now live on the App Store. Search for "cabio" to find and install it.

Struts 2: Putting i18n messages in the page context

2009-02-13T12:15:00.002-05:00

Struts 2 has very sparse documentation when compared with Struts 1, so this took me longer than necessary to figure out, and I thought I'd document it here. Let's say we have a property called my.property in a properties file called messages.properties. We want to display a substring of it on a JSP. The first thing to do is modify struts.properties to load the properties:

struts.custom.i18n.resources=messages

This will load messages.properties into Struts. There's a similar setting available in struts.xml, but beware of a bug which renders it useless in versions prior to 2.0.4. The project I'm working on happened to be using Struts 2.0.1, so I had to add the struts.properties file for this one setting. You can just drop this file into the classpath (i.e. WEB-INF/classes).

Now if we just want to display this property in a JSP, it's actually quite easy. First make sure to load the Struts 2.x Tag Library:

<%@taglib prefix="s" uri="/struts-tags"%>

And then just display the property:

<s:text name="my.property"/>

But now lets say we want to do some mangling on the value which would be difficult or impossible with OGNL alone. We need to get the value into the page context:

<s:set name="myprop" scope="page" value="%{getText('my.property')}"/>
<%
    String url = (String)pageContext.getAttribute("url");
    // do stuff to myprop
%>
<%=myprop%>

Now, I realize that modern JSP theory frowns on introducing any Java code directly into the page, but there are cases where this is necessary (for example if you don't have access to anything else!). Figuring out how to get a value from the property bundle and set it into the page context is not really documented in any Struts documentation and involves some tricky syntax.

Auto-complete: the server side

2009-01-30T11:14:00.006-05:00

Last time we looked at developing a auto-complete implementation, but we stopped short of implementing the server side. We do have a servlet that returns a list of strings, but those strings are hard-coded. In this post, we'll look at creating a more useful servlet which returns relevant suggestions.
First, we note that the jquery.suggest plugin always sends a parameter to the server called q which is the text currently typed into the search box. We should implement our servlet in such a way that it returns ~10 results relevant to the query string q. What is "relevant" is obviously very application dependent. It's important that the search return as quickly as possible, because the result list must be updated upon each keystroke.
The servlet can be very simple, depending on your needs. If your text box is for a disease name, just have the script search the list of diseases for something starting with q and you're done! But suppose you want to auto-complete against a full domain model. We may have several relevant objects in our database, like Gene, Disease, Drug, Organ, and so forth. How do we build our phrase list, and more importantly, how do we order it so that the most useful results are presented first?
Let's create the database table of keywords first.

    CREATE TABLE KEYWORD ( 
        ID NUMBER NOT NULL,
        SCORE NUMBER NOT NULL,
        VALUE VARCHAR2(512) NOT NULL
    );
    ALTER TABLE KEYWORD ADD CONSTRAINT PK_KEYWORD PRIMARY KEY (ID);

We can very easily dump words and phrases into here from any class, but what about the score? When the user types "b" we want to return the 10 best results starting with "b", but we may have 10000 of them in the KEYWORD table, so how do we score them to pick the top 10?
Generally, it is most useful to sort results by popularity. If you are Google and have an abundance of users, then you can simply keep track of past search results and put popular searches first. But in the life sciences, it's far more likely that you're creating a smaller application, with a limited user base. In that case, one way to judge the popularity of a phrase is to count how many times it is referenced by other objects. For example, suppose you have a Gene object with an association to an ExpressionArrayReporter. For each Gene, count how many reporters it is referenced by. In this way, you get a rough estimate of how popular that Gene is.
Of course, when you estimate popularity like this, the average scores will vary wildly across classes. To normalize the data, we can order the keywords in each class and then take the order as our real score. This guarantees that each class is equally represented.
After you populate the KEYWORD table you will need to query it from the servlet. I will assume you're using Hibernate, and I'll skip over the ORM which can be easily generated. Presumably you now have a Keyword class which maps to your KEYWORD table in the database. Let's write a method to fetch the word list for a given query string. With auto-complete, it's usually most natural to only match at the beginning of words. The following Hibernate code does just that.

    public List getKeywords(String s) throws ApplicationException {
        List results = null;
        Session session = null;
        String lwrs = s.toLowerCase();
        try {
            session = sessionFactory.openSession();
            Criteria c = session.createCriteria(Keyword.class);
            c = c.add(Restrictions.or(
                new EscapingLikeExpression("value",escape(lwrs)+"%"),
                new EscapingLikeExpression("value","% "+escape(lwrs)+"%")));
            c = c.addOrder(Order.desc("score"));
            c.setMaxResults(10);
            results = c.list();
        }
        finally {
            session.close();
        }
        return results;
    }

    private String escape(String s) {
        return s.replace("\\", "\\\\").replace("_", "\\_").replace("%", "\\%");
    }
    
    private class EscapingLikeExpression extends LikeExpression {
        EscapingLikeExpression(String propertyName, String value) {
            super(propertyName, value, '\\', false);
        }
    }

Note that escaping special characters is not that straight-forward with Hibernate's Criteria objects. In any case, you can now call getKeywords in your servlet and print out all the Keyword values returned.

public class AutoCompletionService extends HttpServlet {

    public void doGet(HttpServletRequest request, HttpServletResponse response) 
            throws ServletException, IOException {

        String q = request.getParameter("q");
        PrintWriter pw = new PrintWriter(response.getOutputStream());
        
        response.setContentType("text/plain");
        
        try {
            for(Keyword k : getKeywords(q)) {
                pw.println(k.getValue());
            }
        }
        catch (ApplicationException e) {
            log.error("AutoCompletion error",e);
        }
        
        pw.close();
    }
}

You should have a working auto-complete, minus error handling. You can tweak the population algorithm for the KEYWORD table to heart's delight. You might expect different words to come up when you type in a particular string. Tuning that is more of an art than a science.

Auto-complete: the client side

2009-01-26T09:39:00.008-05:00

Adding an "auto-complete" or "suggestion" feature for your search is an easy way to greatly enrich your users' experience. This is especially true in applications related to the life sciences, where difficult, lengthy terms are commonplace. Try typing "mechl" into Google, and it immediately suggests "mechlorethamine." The implementation of such a feature is naturally divided along the client/server chasm. In this post I will describe the very simple client-side implementation. Next time, I'll describe the server-side, which may become complicated if you have a very large database without user search statistics.
In the case of a web application, the client will be a simple Javascript-driven text box. Popular Ajax frameworks like jQuery have auto-completion implementations out of the box. For example, jQuery has a number of suggestion plugins. The best for my needs ended up being jquery.suggest, an extremely simple plugin, that nonetheless features all the important basics such as client-side caching. Implementing the client-side was a matter of importing a few files, and enabling the listener on my input box:

<link rel="stylesheet" type="text/css" href="css/jquery.suggest.css"/>

<script src="js/jquery-1.2.3.min.js" type="text/javascript"></script>
<script src="js/jquery.bgiframe.js" type="text/javascript"></script>
<script src="js/jquery.dimensions.min.js" type="text/javascript"></script>
<script src="js/jquery.suggest.js" type="text/javascript"></script>

<input name="searchString" id="searchString" value="">
<script type="text/javascript">
    jQuery(function() {
        jQuery("#searchString").suggest("suggest",{minchars:1});
    });
</script>

That's it! I passed in minchars:1 because the default is 2, and I wanted the user to get immediate feedback when they were typing. To have this work well, I also had to make a slight tweak to jquery.suggest.js to line 100:

} else if (($input.val().length != prevLength)||(prevLength === 0)) {

There are other customizations you can do in the CSS, and by passing additional options to the suggest() function. At this point your server-side should just be a dummy servlet located at /suggest, which returns a newline-delimited list of strings. When you type in the input box, the list returned by your script should appear below. In the next post, I will describe how to make that script return relevant results for what the user is typing.

Highlight occurrences with Javascript

2008-09-30T17:43:00.003-04:00

Just a simple function today which takes a list of words (strings) and highlights all occurrences of each word in some text. This is useful for presenting search results from an Ajax query for example, where the highlighting is not done on the server side.

function highlight(label, wordList) {
    var h = label;
    for(var i=0; i<wordList.length; i++) {
        h = h.replace(new RegExp("("+wordList[i]+")","gi"), "<b>$1</b>");
    }
    return h;
}

Named parameters in Javascript

2008-09-27T16:42:00.017-04:00

Many Javascript programmers are aware of the benefits of using hashes for parameters to constructors and other complicated functions. Using a hash effectively gives you named parameters:

  function MyWidget(opt_opts) {
    this.background = opt_opts.background  || "white";
    this.border = opt_opts.name || "1px solid black";
  }

Named parameters are often used for optional parameters that have good defaults. If the programmer is happy with the defaults, they can instantiate a class very easily, passing only the parameters they want to change.

  var widget = new MyWidget({background:"red"});

Establishing the defaults can be very easy with the || trick, as above. If no parameter is given, the || operator yields its second operand, the default value for the parameter. However, one has to be careful with boolean parameters. Here's some code I noticed recently in the production version of a popular Javascript library:

  function MyWidget(opt_opts) {
    this.editable = opt_opts.editable || true;
  }

See the problem? The editable variable will always be set to true, no matter what the user passes in! It's easy to get carried away with the || trick, and start thinking of it as a "default value operator" when in fact it has boolean semantics. In fixing this, you end up writing something like:

  function MyWidget(opt_opts) {
    this.editable = (opt_opts.editable!==undefined) ? opt_opts.editable : true;
  }

Not quite as pretty, is it? One alternative method is to define the defaults up front, with a hash literal, and then copy in the user options as "overrides":

  function MyWidget(opt_opts) {
    this.options = {
      background: "white",
      border: "1px solid black"
    };
    for (var s in opt_opts) {
      this.options[s]=opt_opts[s];
    }
  }

Of course, you could always take the fool-proof, Java-inspired approach taken by markertracker.js. Each option is meticulously defined with a default "constant", and then the user option is only copied if it's defined:

  this.iconScale_ = MarkerTracker.DEFAULT_ICON_SCALE_;
  if (opts.iconScale != undefined ) {
    this.iconScale_ = opts.iconScale;
  }

It's not pithy, but I suppose it works. Personally, I'll stick with the first approach. It reminds me of my Perl days.

Unit testing eager fetching

2008-09-25T16:37:00.014-04:00

When unit testing Hibernate queries, it's sometimes useful to assert that a particular association was eager loaded. For example, suppose you are querying for Genes and want to eager fetch the Gene's aliasCollection and taxon. The first is (obviously) a One-to-Many association, and second is a Many-to-One. Your criteria might look something like this (leaving out restrictions for simplicity):

    DetachedCriteria criteria = 
        DetachedCriteria.forClass(Gene.class);
    criteria.setFetchMode("aliasCollection", FetchMode.JOIN);
    criteria.setFetchMode("taxon", FetchMode.JOIN);

Or in HQL:

    select gene from gov.nih.nci.cabio.domain.Gene gene 
    left join fetch gene.geneAliasCollection as aliases
    left join fetch gene.taxon as taxon

Well, in a JUnit test you would definitely want to ensure the eager fetches are actually being done. If the associations are not eager loaded, no error will occur. Hibernate will just silently do additional queries, slowing down your system inexplicably. But how can we test for such a thing? We need to look inside the Hibernate proxy to see if the object has been loaded, or if it's just an empty shell. This is fairly straightforward for Collections (One-to-Many or Many-to-Many associations):

    // get the underlying target object
    Object target = ((Advised)obj).getTargetSource().getTarget();
    Gene gene = (Gene)target;
    // get the collection and cast to PersistentSet
    PersistentSet ps = (PersistentSet)gene.getGeneAliasCollection();
    // assert that the set was eagerly loaded
    assertTrue(ps.wasInitialized());

Things are slightly more convoluted in the case of a Many-to-One or One-to-One association. If the object is eager loaded, it is instantiated as its native class. But if not, then a proxy is created which can do additional queries later. We can detect these proxies by looking at the class name of the associated object. Proxies have class names like gov.nih.nci.cabio.domain.Taxon$$EnhancerByCGLIB$$e4aca6e7. Here I've elected to just check for the CGLIB token:

    // get the underlying target object
    Object target = ((Advised)obj).getTargetSource().getTarget();
    Gene gene = (Gene)target;
    // get the associated object
    Taxon taxon = gene.getTaxon();
    // ensure the object was eager loaded
    assertFalse(target.getClass().getName().contains("CGLIB"));

Now, that's a lot of code to write for each test case. To make it easier, I've created a TestCase subclass with extra assert methods. It is available in the caBIO code base.

Now you can simply write:

    assertPreloaded(gene,"geneAliasCollection");
    assertPreloaded(gene,"taxon");

Application scope variables with Struts

2008-09-24T16:34:00.008-04:00

Often times you want to load some static data just once, when an application starts up, and put it in the application scope for use in JSPs. With Struts, it is not immediately obvious how to do this. Just check out the myriad of answers given in this thread. The discussion there eventually settles on a sub-optimal solution involving a servlet that checks if the variable is set in the application context, and setting it otherwise.

There is a better solution (conceptually and in terms of performance) involving Struts Plugins. Let's say we have a class GlobalQueries, which is initialized with data by its constructor. We can implement a simple Plugin to create a singleton of our class, and put it in the application context.

public class AppLoadPlugin implements org.apache.struts.action.PlugIn {
    public void destroy() {}
    public void init(ActionServlet actionServlet, ModuleConfig moduleConfig)
            throws ServletException {
        actionServlet.getServletContext().setAttribute(
            "globalQueries", new GlobalQueries());
    }
}

Then register it in the struts-config.xml:

    <plug-in className="your.package.AppLoadPlugin"/>

We can use globalQueries directly in any JSP, using JSTL expression language, or Struts taglibs. For example, we might write something like this to populate a select list (see previous post):

    <html:options name="globalQueries" property="stateValues"/>

Or in a JSTL forEach statement:

    <c:forEach var="state" items="${globalQueries.stateValues}">

So that's a very easy way to expose global data to your application.

Select list patterns with Struts

2008-09-23T14:50:00.005-04:00

The Struts User Guide is confusing when it comes to populating select lists. Here are some useful patterns I ended up with, after a lot of trial-and-error.

1) Application-level bean with a List of Strings which is used for both values and labels.
Here the globalQueries.getStateValues() method returns a List with values like "AL", "AK", "AS", etc.

<html:select property="state">
<html:option value="">Select...</html:option>
<html:options name="globalQueries" property="stateValues"/>
</html:select>

2) Application-level bean with a List of beans.
Here the globalQueries.getStates() method returns a List, where State defines getAbbreviation() and getName(). If the values are immutable, State could even be an enumeration and getStates() would simply return State.values().

<html:select property="state">
<html:option value="">Select...</html:option>
<html:optionsCollection name="globalQueries" property="states" value="abbreviation" label="name"/>
</html:select>

3) Action Form with List of beans.
Same as 2, but the getter is on the ActionForm.

<html:select property="state">
<html:option value="">Select...</html:option>
<html:optionsCollection property="states" value="abbreviation" label="name"/>
</html:select>

Sine scientia, ars nihil est

2008-09-22T16:33:00.004-04:00

There are many times when I find myself rediscovering techniques that I researched and employed only months before. This notebook is an attempt to keep track of various tricks of the trade which I learn, use, and ultimately forget. It may be code snippets, commands, workarounds or bookmarks. They may deal with Java, Python, mobile devices, bioinformatics, Google Maps, Javascript, or just about anything else related to programming. Despite the title of this blog, I expect to also be posting about code from personal projects which may not relate to life sciences.