When trying to improve performance, caching is usually an easy win. Its also important to apply user quotas to potentially costly resources. (For example, enforcing a maximum number of friends may significantly decrease the load to compute a page.) Since web applications are read dominated, they can often be supported by a database servers in a master/slave hierarchy. Only database writes are sent directly to the master. These updates are duplicated on the slaves in a timely manner. All read operations use a slave server. One slave is often assigned to backup and computationally expensive tasks. Instead of master/slave servers one can use knowledge about the application and partition the database across functional requirements. Instead of a series of identical slaves, the data is divided based on value, required access time or some other intelligent metric.

Schwartz gives a nice discussion of sharding and why it often isn’t he right solution. Under a section titled more risky solutions that should not be used, he lists multiple write masters, multilevel replication, ring replication, and entity attribute value databases.

The bottom of NetApp’s architecture diagram contains one or more RAIDs. Above that sits a NetApp aggregate. The aggregate manages the storage resources. An administrator creates volumes within the aggregate to provide resources to developers and applications. One volume can, for example, be configured to provide SAN storage to an Exchange box while another volume provides NSF mounted NAS storage to a research group. Each volume enforces policies such as WORM storage, data encryption data, provide frequent snapshots, etc. Via a Gateway, NetApp manages non-NetApps storage resources.

A virtual volume provides data for testing (and may have other uses). It can provide a zero-storage copy of production data to developers.

NetApps boxes contain many interface options (GigE, fiber, etc.). In-box redundant controllers share non-volatile memory so if one controller fails the second continues without the loss of any data.

The storage managed by an aggregate can be increased by adding more drives. Volumes may be resized on the fly. The minimum volume size is 20MB.

• Creates a browser object
• Sends it to a URL
• Manipulates UI elements
• Screen scrapes to verify correct results

Its easy, although the screen scraping can be (not surprisingly) a little fragile.

I have used Selenium 2 to construct end-to-end tests for the Geo Portal. The portal is a search tool for an institution’s repository of geographic information. The repository holds many thousand layers so the portal must provide appropriate, well-ranked search results.

When using Selenium 2, you must first create a web browser object and send it to the portal page we want to test:

WebDriver webDriver = new FirefoxDriver();
webDriver.get("http://geoportal-demo.atech.tufts.edu");


With the browser object in place, the first two steps are complete. The only things left to do are manipulate UI elements and screen scrape.

For every search the portal executes it, like Google, displays the number of results where each result is a geographic data layer. These total number of results are displayed in a SPAN element. Since my tests often rely on the number of layers found, I wrote a function to screen scrape and return the search result count. The code simply gets the relevant DOM element by its ID, obtains the text and strips of the “(” and “)” characters that surround the number.

public static int getSearchResultsCount(WebDriver webDriver)
{
WebElement spanElement = webDriver.findElement(By.id("resultsNumber"));
String text = spanElement.getText();
String temp = text.replace("(", "");
temp = temp.replace(")", "");
int layerCount = Integer.parseInt(temp);
return layerCount;
}


The portal supports geo-coding. That is, there is a text field for the user to enter an address or a place name. If a location is entered, the application uses Google to obtain geodetic coordinates for the place and uses the type of place to determine the zoom level. Then, the map is updated and a search for geographic data centered on the new map is performed and the results are displayed.

An easy way to grossly test this functionality is to zoom the map all the way out and note how many layers the search returned. Then, enter a location to zoom in on a region and note how many layers the search returned. Since we zoomed in to a small region, many layers should have been considered not relevant and filtered out of the result set. Therefore, the number of search results should have decreased if the geo-coding worked. Here’s the code that performs the geo-coding test:

private static void geoCodingTest(WebDriver webDriver)
{
logger.info(" in geoCodingTest");
zoomToWorld(webDriver);
pause();
int startResultsCount = getSearchResultsCount(webDriver);
setLocation(webDriver, "Boston, MA");
pause();
int endResultsCount = getSearchResultsCount(webDriver);
assertTrue("geoCoding test failed", startResultsCount > endResultsCount);
}

The function zoomToWorld simply clicks the UI button to zoom the map all the way out. The function pause sleeps for a second. This is important because the operation like zooming the map to view the entire world is not instantaneous. Map tiles and search results must be obtained via AJAX and displayed. Finally, JUnit’s assert it used to check the observed results.

Entering text into a UI element is very simple:

private static void setField(WebDriver webDriver, String fieldId, String value)
{
WebElement field = webDriver.findElement(By.id(fieldId));
field.clear();
field.sendKeys(value);
}


Note that the main test function is trivial. This means you can quickly develop many tests. Then, whenever the application changes one can simply run the tests to see if anything might have unexpectedly broken. This is especially if your modifying code you did not write or wrote a while ago. Instead of haphazardly testing an application you’re not intimately familiar with and hoping for the best you can run a real test suite and have confidence the application works.

The portal is a nice single-page web app that helps users find GIS data layers. It uses Solr to quickly generate search results. The original design was straightforward. The page has a map, a keyword field, and several other controls for search parameters. When the user initiates a search the JavaScript code sends an AJAX call back to the web server. The server computes a Solr search query based on the passed parameters. Then, it sends the query to the Solr server, gets the search results and returns them as the return value to the AJAX call. The Solr query is fairly complicated, it takes a couple hundred lines of Java code on the server to generate.

This original design works very well but it can be significantly improved because Solr supports JSONP. Instead of computing the Solr search string on the web server, the client can generate it. Then, the client can send off the request directly to the Solr server. This has several obvious advantages. It will be a little faster since the web server hop is eliminated. The load on the web server is reduced. Much more importantly, we have ported a chunck of server-side Java code to JavaScript runningnon the client. I care about this because some institutions might want to stand-up an instance of the portal but they might not be Java shops. By eliminating server side code, I can make the portal server-side technology agnostic. If the portal only uses HTML and JSONP, it can be easily deployed by php shops, Java shops, etc.

Its was easy to change the portal to use JSONP. A couple hundred lines of Java code had to be ported to JavaScript. Since the AJAX call was already returning a JSON object, it was trivial to change the AJAX call to JSONP. The only concern with adding client code is that it typically runs in the UI thread. Having the client compute the Solr search string instead of the server is an additional client side computational load. However, it is of order O(1) and only involves some concatenation to generate a string that is under two kilobytes in length. It is a negligible load.

I put the JavaScript code to generate Solr search strings in its own JavaScript file. Now, anyone can integrate the Tufts GIS data repository into their own portal. They simply include our JavaScript file on their page and call is functions to perform searches. They don’t need detailed knowledge of our Solr instance and its schema. Instead, the JavaScript library provides a high level API.

The JSONP based approach makes the portal independent of server-side technology. It also provides a high level API that our portal and other institutions can use. Both are big wins.

This graph shows page views for Spark for the entire semester.  Usage is highest mid-week (Tuesday through Thursday) and typically peaks on Wednesday.  Over the semester, Spark had nearly 106000 page views.  Peak usage was on Oct 20 with almost 3400 page loads.  On this peak day, the peak hour saw 330 page loads.  Heaviest usage on Oct 20 was between 5pm and 8pm.

Here’s a graph of page views broken down by time of day:

Usage peaks in the morning and again after 4pm.  This latter peak suggests significant use by students.

The following table shows page views over the past three fall semesters:

It is difficult to predict the future, but we should plan on Spark receiving at least 150000 page views in the Fall 2011 semester.

The book talk about all the different things that should be measured. This includes low level operations (e.g., disk read/writes), mid-level operations (Apache or database operations) and high/application level operations.  All must be measured so you fully understand the system.  With this data you can identify potential bottlenecks and understand trends.

The book uses data from Flickr avoid exclusively dealing with abstract issues.  For example, Flickr disk drive usage increases at the rate of 1% per month.

Many web sites are limited by the performance of their database servers.  Depending on the site, the issue could be CPU, network or disk I/O.  The book describes the open-source tools needed to determine which issue is the limiting factor on any given site.

Understanding usage trends is important not just for customer satisfaction but also to avoid buying hardware early.  Buying capacity before it is really needed just means you’ll pay more to get something less capable.

The book has a chapter on deployment.  It reviews what tools to use (e.g., PXE, iClassify and Puppet) to centrally control a large number of servers.

Most surprisingly, the book describes approaches to test load on production servers with production data (httpref, JoeDog’s Siege).  The author doesn’t like relying on artificially generated data and test hardware to understand system performance.

New browsers (Firefox 3.5) support “Web Workers”. Each new worker is given a new thread that is initialized by code loaded from the specified JavaScript file (e.g., new Worker(“foo.js”)). Worker threads can not update the UI; they don’t have access to the DOM and a bunch of other global variables. However they can send messages (with serializable objects) back and forth to code running in the shared UI thread. In the UI thread you could use a timer to periodically check for new messages and use their contents to modify the DOM.

This workshop will focus on tools and technologies to develop better web applications. It will combine presentations, attendee demonstrations and breakout/development sessions. This workshop is for technical staff that build, support or maintain web-centric solutions. While many general concepts will be discussed, most specific solutions will be oriented towards Java based development.  Attendees will also be invited to deliver presentations.  This presentations may review successful technologies and solutions or seek suggestions to address persistent problems. The list of technologies and techniques the organizers will present include:

• Rapid application development with Spring Roo
• Injecting test data with DBUnit
• Automating end to end testing via Selenium 2.0
• Measuring test coverage using Emma
• Writing client-side code with JQuery
• Improving web site performance with Firebug and PageSpeed
• Providing excellent search results with Solr/Lucene

After each topic has been presented, students can complete a hands-on exercise.  This exercise will emphasize developing code with new technology.  Alternatively, students may join a discussion group with an organizer to explore how the technology might impact their institution’s specific needs.