Large page rendering using appfuse stack

I recently created an application that started with the appfuse stack. Everything performed admirably, until I created a page with numerous form elements (~30K)that was large (~38M when saved to disk). The page would either cause an OutOfMemory (OOM) error or would take 3+ minutes for the first byte download. Here's what I did to bring full page load times to < 15s.

Analysis

One could argue that poor UI design could be to blame, but in this case, I couldn't design my way out of the performance problem (e.g. pagination, more data filters, etc). It's always nice to try that first. The client knew this was a massively data intensive screen and was able to put up with up to 30 seconds for this page to load. Data access (which I had already optimized) was taking 12 seconds, so why was rendering taking 3 minutes by itself?

I hooked the application into a profiling tool (YourKit) to determine the culprits. I found that I had both memory (excessive GC) and cpu utilization problems.

Memory Optimizations

Sitemesh

Sitemesh is a very useful package. It allows you to decorate any page with others. So you can make a tiny webpage and decorate it with a header, footer, navigation bar, etc and not have to include any of that on the individual page. It makes consistent look and feel a non-issue. The only problem is that Sitemesh pulls the page to be decorated into memory so that it can be parsed and adorned properly. This normally isn't a problem, but with a large page this turns into an OOM error pretty quickly.

Unfortunately, I couldn't figure out a non-intrusive way to modify Sitemesh such that it didn't buffer the page. I ended up adding the decorating template to the jsp, and the URI to the excludes element in the decorators.xml configuration file. In short, I disabled Sitemesh for the big page. I hear that Tiles (similar conceptually) doesn't do this kind of parsing/caching, which means it might not suffer from the same problem.

Acegi

Let's say the user types in a URI that requires a login. The application presents the login screen then magically returns to the URI that the user originally typed. In Acegi, the original URI is stored via the SecurityContextHolderAwareRequestFilter. More specifically, the SavedRequestAwareWrapper class (the default) is used to store important data from an http GET or POST operation (in memory). Normally this isn't a problem. But when you have ~30K form fields, it gets a bit memory intensive (lot's of unnecessary object creation and collection).

Luckily the Acegi framework provides a more simple wrapper which does not cache the data. Too bad the simple wrapper doesn't have the proper constructor . So I had to write my own extension class which included the proper constructor.

public class 
  ValidSecurityContextHolderAwareRequestWrapper
    extends SecurityContextHolderAwareRequestWrapper {

  public ValidSecurityContextHolderAwareRequestWrapper(
    HttpServletRequest arg0,
    PortResolver arg1) {
    super(arg0);
  }
}

and modify the security.xml file

<bean id="securityContextHolderAwareRequestFilter" 
class="SecurityContextHolderAwareRequestFilter">
  <property name="wrapperClass" 
value="ValidSecurityContextHolderAwareRequestWrapper"/>
</bean>

(packages omitted for brevity)

Displaytag

The display tag library lets you make nice looking, robust tables, really easily. The problem with it, is much the same as the Acegi request filter; in order to construct links necessary to sort a table, displaytag pulls all parameters sent to a page into a Map in memory. Again, this is normally fine, but when you have 30K parms, the parsing turns into a burden. I attempted to use the excludedParams="*" attribute but that didn't stop the parsing; it added an iterative remove for each parm! The tag should probably be rewritten to not add excludedParams instead of the current remove after parse implementation.

On my page, I do not have sorting enabled on the tables, it didn't make business sense. Since excludedParams didn't fit my needs, I made my own extension.

public class NonParamForwardingDisplayTag 
  extends ELTableTag {
  /**
   * Performance Enhancement.. 
   * don't parse the request parms
   */
  protected void initHref(RequestHelper arg0) {
  }
}

Nice and simple. I then copied and renamed the displaytag-el.tld (changing the tag-class) and included the tag declaration in the web/common/taglibs.jsp file ..

<div class="geshifilter"><pre class="php geshifilter-php" style="font-family:monospace;"><span class="sy0">@</span> taglib 
  uri<span class="sy0">=</span><span class="st0">&quot;/WEB-INF/tld/noparmforward-displaytag-el.tld&quot;</span> 
  prefix<span class="sy0">=</span><span class="st0">&quot;noparm-display-el&quot;</span></pre></div>

After that, I modified the jsp use the new tag instead of the normal tag.

Commons-lang

Appfuse makes use of the commons-lang package for the implementation of equals(), compareTo() and hashCode() types of functions. When using this package over 30K elements, the amount of object creation becomes prohibitive. I'm not sure what the tipping point is though. So instead of

public int compareTo(Object object) {
  LotsOfMe that = (LotsOfMe) object;
  return new CompareToBuilder()
    .append(this.startDate, that.startDate)
    .append(this.endDate, that.endDate)
    .append(!this.mutable, !that.mutable)
    .toComparison();
}

you get

public int compareTo(Object object) {
  LotsOfMe  that = (LotsOfMe ) object;
  int ret = 0;
  ret = this.startDate.compareTo(that.startDate);
  if (ret == 0) { //start dates are equal
    ret = this.endDate.compareTo(that.endDate);
  }
  if (ret == 0) { //start and end dates are equal
    if (this.mutable == that.mutable) {
      ret = 0;
    }
    //mutable needs to come before non-mutable
    if (this.mutable == false) {
      ret = +1;
    } else {
      ret = -1;
    }
  }
  return ret;
}

Obviously the commons-lang is more succinct, easier to change and nicer on the eye. The latter doesn't require any object creation.

Spring Validator

Appfuse (spring MVC front-end) uses Validators to process form submissions and check them for errors. In the public void validate(Object arg0, Errors errors) method, I was making use of the errors.pushNestedPath() and errors.popNestedPath() methods. I had a pretty complicated object graph to validate so the placeholders were nice; I didn't have to remember that I was 17 levels deep when I wanted to throw an error. The problem with this approach was two-fold:

1. I has creating a StringBuffer for each pushNestedPath() call (to make the path segment).
2. pushNestedPath() creates its own StringBuffers internally.

So the result was massive object creation (and subsequent collection) for an error condition that is very unlikely. I changed the Validator such that one StringBuffer is created when a real error (as opposed to potential) is created.

CPU Optimizations

Displaytag TableDecorator

I needed to implement a decorator to handle a complicated footer that needed to show on each table. The issue stemmed from the fact that, for each column, displaytag asked the decorator if it had a getter for that particular property being rendered. This seemingly innocuous call was pretty expensive. The tabletag implementation acknowledges this and tries to minimize the expense by caching calls for the same column (optimized for lots of rows). However, my page had 70 tables with 118 columns a piece and very few rows; the goggles [they] do nothing. Since I was not ever going to decorate a column on these fancy footer tables, I overrode the method call and solved the problem:

public boolean hasGetterFor(String arg0) {
  return false;
}

Client-side javascript

I noticed that after I received a full response from the server, my browser would sit at 100% CPU utilization (on the client) and then sometimes die. We were using Internet Explorer exclusively (no idea if this happens in other clients). I tracked this down to some javascript. In Appfuse, the global.js file defines:

window.onload = function(){
  highlightFormElements();
  ...
}
function highlightFormElements() {
  // add input box highlighting
  addFocusHandlers(
    document.getElementsByTagName("input"));
  addFocusHandlers(
    document.getElementsByTagName("textarea"));
}
function addFocusHandlers(elements) {
  for (i=0; i < elements.length; i++) {
    ...
  }
}

So everytime a page loads, every single form element is iterated over and nice little stylings are applied to each input and textarea. This stuff looks great, but killed the client's pc when there are a boatload of elements. In addition to making stuff look great, this script had the unintended consequence of disabling the page's ability to override the onblur() javascript function (which was sometimes a problem for me).

I added an override so that each page could disable this functionality if it wished...

var overrideBlur = false;
function addFocusHandlers(elements) {
  if (!overrideBlur) {
    for (i=0; i < elements.length; i++) {
      ...
    }
  }
}

then on the jsp declare

<script type="text/javascript">
  var overrideBlur = true;
</script>

Conclusion

Java is great in a best-of-breed sense; you pile a bunch of packages on top of each other and get an elegant, easy to maintain application. The only time there is a potential problem is in the overlap. In Appfuse's case, how the request is parsed is a point of overlap; the servlet engine, acegi, spring mvc and displaytag all do it and each in their own way.

After profiling an application built on the Appfuse stack, I discovered that the two biggest problems with it with regard to large pages (with lots of form elements) were:

1. Parsing/caching of pages and POSTed parameters.
2. Iteration of form elements via javascript.

Before tuning the application, a large page request took over 3 minutes was 7% data access, 93% data rendering! After tuning, the same request took <15s and was 85% data access, 15% rendering.