Rodney Waldhoff: Java

On Programming Idioms

Fri, 14 Nov 2003 17:36:09 GMT

Two things I'm always keen to learn when picking up a new programming language are:

How does one organize large projects? In other words, how does one partition responsibilities and types across namespaces, modules and files?
What are the common idioms in the language?

I've been doing some string processing work with Ruby recently, and it's got me thinking about examples of the latter.

For example, in Java, the String class doesn't have a direct, boolean-valued method that will tell you whether or not a String contains another String, i.e., there's nothing like:

if(someString.contains(anotherString)) { ... }

Instead, most Java developers will write:

if(someString.indexOf(anotherString) != -1) { ... }

where String.indexOf(String) returns the index of the first occurrence of the argument String, or -1 if the given String isn't found. Most Java developers will immediately recognize that as the "String.contains" idiom, and won't miss a beat.

This idiom is so strong in the Java community that it's almost counter-productive to write a custom utility method:

public class StringUtils {
  public static boolean contains(String a, String b) {
    return (a.indexOf(b) != -1);
  }
}

since many developers who see

if(StringUtils.contains(someString,anotherString)) { ... }

are likely to wonder whether the StringUtils.contains method really does what it is implied--Is this equivalent to the String.indexOf idiom? Is that someString.contains(anotherString) or vice versa? How are null's handled? etc. Unless the developer is already comfortable and familiar with the StringUtils class being used, this code is probably less readable to an experienced developer than the "indexOf != -1" formulation.

(This is not to say that "String.indexOf(x) != -1" is actually preferable to "String.contains(x)", but rather that in the absence of String.contains, the idiom is more widely recognized than a custom utility method. Why Sun can't at some point introduce a String.contains method, say in JDK 1.5, isn't entirely clear to me.)

Now, in the Ruby scripting I've been doing recently, I keep needing to determine whether a String begins with a given prefix. In Java, that's the String.startsWith method, of course. The Ruby String class does not have a startsWith method, but one of the neat things about Ruby is that it's possible to literally add such a method the the String class, as follows:

class String
  def startsWith str
    return self[0...str.length] == str
  end
end

after which everything behaves exactly like the built-in String class contained that definition. For example, one can then write:

if someString.startsWith(anotherString) ...

or even

if "a literal string".startsWith(anotherString) ...

etc.

Of course, the implementation I used for String.startsWith (self[0...str.length] == str) is just one of several possible implementations. Regular expressions provide one way of implementing such a check. The Java-like indexOf function provides another (e.g., self.indexOf(str) == 0).

Since there is no built-in String.startsWith (or for that matter String.contains) in Ruby, I wonder if there is some common idiom that experienced Ruby developers find more readable than adding a custom method to String? If not for String.startsWith, how about String.contains?

Apache Jakarta Commons Primitives 1.0 Released

Thu, 06 Nov 2003 17:08:37 GMT

I've blogged a bit about the Primitives Component in the past, and I'm happy to note that commons-primitives now has an official 1.0 release (binaries) (source).

Currently Primitives provides Java-primitive based versions of various Collection interfaces, which is substantially smaller and faster than working with the Object-wrapper equivalents. For instance, an ArrayShortList requires 1/10th the space of an ArrayList of Shorts, and an ArrayLongList requires 2/5ths the space of an ArrayList of Longs. There's a substantial time savings as well, in that one isn't constantly "boxing" and "unboxing" the primitives and their Object equivalents when moving them in and out of the Collection. There are readability and other advantages as well.

(And, no, as I understand it, neither the auto-boxing nor the generics features of JDK 1.5 will obviate the need for this library. You could use auto-boxing to emulate the syntax of list.add(int), but the internal representation will still be Object based.)

We use the primitive collections fairly extensively within the Axion database project for things like indices and lists of row identifiers or data file offsets, indeed the code was originally developed there.

The Jakarta Commons Problem in a Nutshell

Thu, 30 Oct 2003 15:26:23 GMT

Exhibit A, Exhibit B

Wiki + Blog = PIM

Wed, 22 Oct 2003 15:40:46 GMT

I'm one of those people who's so obsessed with organization that he's poorly organized. I'm constantly trying out new ways to manage the notes I scribble on various physical and electronic media, so much so that the frequency of change leads to greater disorganization.

I've tried a number of "information management" tools, including index cards, sticky notes, notebooks, a Franklin planner, Microsoft Outlook, and various outlining, mind-mapping and PIM programs, and found them all wanting in one way or another (maybe it's me). Paper based systems were too hard to search and too static in their organization. Lacking a PDA, electronic systems weren't portable enough. No matter what the system, I'd always spend more time trying to set up organizational categories than I ever saved by using them. In fact I had pretty much broken the habit, falling back to a collection of loosely organized text files that I could easily scp to and from whatever workstation I was currently using.

About a year ago, I got a Sharp Zaurus, and since that addresses one of my recurring PIM complaints--having something that's always readily available, my hopes for finding the perfect PIM system were renewed. I've played around with a number of the PIM systems available. Using the bundled Address Book/Calendar/To Do List/Email Client was too jumbled. The "outliner" IQNotes was promising, but a single hierarchy is inflexible and using it to create dated journal entries requires too much manual effort. In the end, I came back to a loosely organized collection of text files, this time stored on the Zaurus.

Back in my first post, I mentioned that one of the reasons I was interested in blogging is that I had become interested in journaling in general. These two concepts--the search for the perfect organizational system and interest in journaling--interrelate.

As I'm sure I'm not the first to suggest, it seems that the ideal PIM system presents at least two perspectives on the information to be managed:

1) A time based view, answering such questions as "what have I been working with recently?" or "what else was I working on around that time?"

2) A "concept network" view, answering such questions as "what's related to this topic?" or "what else do I know about this subject?"

Clearly, I presume, blog-like software is an excellent starting point for the former, and wiki-like software is an excellent starting point for the former. Fortunately, a number of projects have begun to explore combinations of the two, either by implementing a blog-as-wiki (like SnipSnap does) or by adding blogging features to a wiki engine (like JspWiki and no doubt others do).

Having a fair amount of Java experience (indeed, I've built a little wiki or two in Java), my first thought was to get a Java based wiki up and running on my Zaurus. JspWiki looks nice, and like most Java-based wikis is Servlet/JSP based, so I endeavored to get a WAR-friendly Java Servlet engine up and running on my Zaurus. Having tried both Tomcat and Jetty under both the Jeode evm and Personal Java cvm, I came close, but fundamentally failed to do this. (I note that my SL-5500 had just barely enough disk space or memory to do this as well.)

I briefly considered setting up a stand-alone (i.e., non-Servlet/JSP) Java wiki engine on the Zaurus, but without a moderate amount of development that seems like an awkward environment for templating and modification, and non-servlet Java web development feels like dead-end development. (Its only a matter of time before a reasonably micro servlet engine is available.) That's when I stumbled across PhlIp's Mini Ruby Wiki, which implements a featureful, stand-alone wiki server in around 1000 lines of Ruby code.

(For the record, one could readily build a similarly featured wiki in Java, and probably a number of languages, in roughly the same number of lines, it might not be as much fun.)

It still took me a while to get a Ruby interpreter up and running on the Zaurus (after trying a few different builds, I finally got the one from Vincent Fiack working), but the resulting binaries are smaller in terms of disk and memory footprint than the JVMs I was playing with. Perhaps more importantly, as a purely interpreted language, the engine itself is easier to tweak. And this gives me a reasonably good excuse to hack out some Ruby code. I've already added/modified a few minor features, adding some simple blog-like features to the calendar features that already existed. If I come up with something I'm not to embarrassed to share, I'll post it here or submit it to PhlIp's RubyForge project.

My only problem now is resisting the temptation to continually tweak the wiki engine code.

Kudos to Jakarta Commons FileUpload

Mon, 15 Sep 2003 15:37:40 GMT

Recently Ted and Jason blogged on an old complaint: open source projects aren't well documented.

In a curious coincidence, at the same time I had an very positive experience with open source documentation. Being a Jakarta Commons committer, and having done my share of multipart/form-data parsing in the past, I was vaguely familiar with Commons FileUpload but hadn't had much cause to look into it until recently. Not only does it feature an easy-to-use API that works like a charm, the Using FileUpload page tells one everything he needs to know to get up and running in minutes.

Granted FileUpload addresses a pretty limited domain, and not all the documentation is as complete, but I can't remember the last time I picked up a Java library and started using it without even cracking open the JavaDocs.

Commons Logging was my fault

Fri, 15 Aug 2003 18:59:08 GMT

I'll come right out and admit it: commons-logging, at least in its initial form, was my fault, though probably not mine alone.

Back in 2001 I did a fair bit of work refactoring and debugging commons-httpclient. As part of that effort, I replaced a custom setDebug()/System.out.println based logging system with log4j. I did this for several reasons but all of them come down to the fact that a fully-fledged logging system is often quite useful. In fact, that debugging effort probably wouldn't have happened without it.

This change turned out to be controversial, and a runtime dependency upon log4j was in fact vetoed. After an enormous amount of discussion, not all of it pretty, a compromise was finally agreed to. This compromise basically said (a) use a thin wrapper around either log4j or the System.out logging initially implemented and (b) allow the user to specify whether to use log4j or System.out.println based upon a method call (inversion-of-control style I guess) or an external property. I did the first implementation of this. It was later suggested, probably reasonably so, that this "log wrapping" package be extracted from httpclient for use in similar circumstances. In a more robust or complicated form (depending upon your perspective) this eventually became commons-logging.

For all its warts, this brief history of Commons Logging shows the "bazaar-style" of open source development working as it should: the operative word here being "compromise".

Personally, I don't believe it to be especially important that Jakarta Commons committers be able to express their creativity by the selection of logging frameworks. I'd have been happy simply using log4j, and I think Ceki's work with the "light" log4j-ME would have been sufficient to address most of the technical concerns raised. Yet others disagree, and I'll respect that position. Certainly respect for that position has encouraged advocates of alternative logging implementations to participate more fully in Jakarta Commons.

That said, I think Hani and Glen miss the point entirely. The purpose of Commons Logging is not to isolate your code from changes in the underlying logging framework. (That's certainly easy enough to do on your own, and not really worth doing in the first place given the ease of switching from one logging framework to another.) The purpose of Commons Logging is not to somehow be more useful than actual logging frameworks by being more general. The purpose of Commons Logging is not to somehow take the logging world by storm. In fact, there are very limited circumstances in which Commons Logging is useful. If you're building a stand-alone application, don't use commons-logging. If you're building an application server, don't use commons-logging. If you're building a moderately large framework, don't use commons-logging. If however, like the Jakarta Commons project, you're building a tiny little component that you intend for other developers to embed in their applications and frameworks, and you believe that logging information might be useful to those clients, and you can't be sure what logging framework they're going to want to use, then commons-logging might be useful to you.

On a slightly unrelated note, I think that Hani's post fails to adequately explain how the problems caused by trying to integrate systems that use incompatible versions of the same component are unique to Commons Logging. If you want a discussion of the technical issues with Commons Logging, which are quite genuine, there are much better analyses available.

Clover Plug-in for Eclipse (and NetBeans)

Fri, 08 Aug 2003 12:45:59 GMT

Matt Quail writes "We have just released (beta) an Eclipse plugin and a NetBeans plugin for Clover, allowing you to instrument your code and view coverage results all from with the those Java IDEs."

Haven't tried it yet, but the screenshots look like exactly what I was hoping for. Sweet.

Wanted: GUI Wrapper for Text-Based Java Applications

Wed, 30 Jul 2003 00:10:38 GMT

I've got a simple console-based (i.e., text interface) Java application that I'd like to expose via Java Web Start or as an Applet. I can't, however, because System.in and System.out aren't attached to anything useful, so my application runs invisibly and can't collect any input.

What I need is a simple wrapper application that creates a basic text i/o frame that looks like stdin/stdout to my application's main method. (Or, failing that, to a custom main(String[] args, InputStream stdin, PrintStream stdout, PrintStream stderr) method.) More elaborate interfaces, such as color coded output, distinct stdout and stderr frames, spool to file, etc., are easily imagined.

I believe such an application would have fairly broad utility. A web search finds quite a few elaborate telnet/ssh oriented terminal emulators, which one might be able to pare down to what I'm looking for, but it seems like someone would have already tackled this problem. Lazy web to the rescue?

A is for Axion ... Z is for Zaurus

Thu, 24 Jul 2003 18:11:39 GMT

I've got Axion up and running on my Zaurus.

If you'd like to do the same, here's how:

Obtain a build of the Axion HEAD (1.0M3-dev). See the instructions on building Axion for details.
Obtain binaries of Axion's runtime dependencies, namely Jakarta Commons Collections (currently a nightly build is required) and Jakarta Commons Logging (you'll want release 1.0.3, earlier versions don't work on the Jeode VM).

If you want to use the LIKE operator, you'll also need Jakarta Regexp.

If you want to use the BASE64ENCODE or BASE64DECODE functions, you'll also need Jakarta Commons Codec.
Ensure that you have the java.util Collections and java.sql JDBC packages available on your platform. Since the Insignia Jeode VM installed on the Zaurus by default is JDK 1.1 based, I had to copy over the JDK Collections manually. I had thought the Collections package was available as a standalone JAR to drop in to JDK 1.1 environments but as I was unable to find it. I just copied over the rt.jar from a JDK 1.3 installation, but I think the java.util package would suffice.
Obtain a copy of axiondb.properties. You can pull this out of the Axion JAR or from CVS. Normally Axion loads this file out of the JAR automatically, but the Jeode VM seems to always return null for getClassLoader so we had to add a mechanism for specifying the configuration as an external file.

Now simply copy those files over to the Zaurus, add the JARs to your classpath, and you are ready to go. For example to run the Axion console I use:

evm 
 -Dorg.axiondb.engine.BaseDatabase.properties=axiondb.properties 
 -classpath axion-1.0-M3-dev.jar:
            commons-collections.jar:
            commons-logging.jar:
            commons-codec.jar:
            regexp.jar:
            rt.jar 
 org.axiondb.tools.Console $1 $2

Similar steps should get Axion up and running on other micro or J2ME platforms or for that matter, other JDK 1.1 environments. If the console is any indication, Axion seems to run rather well on the Zaurus (even running off a compact flash card rather than the RAM disk).

If folks were interested, it wouldn't be difficult to create an IPK installer for Axion, although my interest here, and others as well I imagine, is in using Axion within other apps on the Zaurus, rather than playing with Axion via the console.

If you're curious, I did have to make a few minor changes to Axion to get it run on the Jeode/JDK 1.1 VM. Here's a brief list of what I had to change:

The Jeode VM always seems to return null for getClassLoader (rather than throwing a security exception), so I had to add checks for null, and provide an alternative mechanism for loading the properties file.
We had used File.createNewFile in several places, which is a JDK 1.2 method. These calls turned out to be unnecessary anyway, so I simply removed them.
Although it's easy to add java.util.Comparable and java.util.Comparator to the classpath, none of the core objects (Number, String, etc.) actually implement Comparable in JDK 1.1, so I had to add custom Comparators replacing ComparableComparator for those DataTypes.

For reasons I don't understand, in several places where we had an hierarchy like this:

interface Foo {
  void someMethod();
}
 
abstract class AbstractBar implements Foo {
  void anotherMethod() {
    doSomething();
  }
}
 
class Bar extends AbstractBar {
  void someMethod() {
    doSomethingElse();
  }
}

I had to declare the interface methods in the abstract class:

abstract class AbstractBar implements Foo {
  abstract void someMethod();
 
  void anotherMethod() {
    doSomething();
  }
}

to make various AbstractMethodErrors go away.

These changes have already been checked into the HEAD version of Axion, and should be part of the Milestone 3 release.

There are few Axion-based apps I've been thinking of tinkering with on my Zaurus, some for personal use, others for my day job. That may shake out a few issues I haven't yet encountered, but so far I've got feature I've tried working without too much trouble.

LGPL and Java: More confused than ever

Fri, 18 Jul 2003 17:35:52 GMT

After reading the various comment threads, and seeing Andy and David's exchange in a bit more context, I'm more confused than ever.

As quoted by Andy, Roy wrote:

What the FSF needs to say is that inclusion of the external interface names (methods, filenames, imports, etc. defined by an LGPL jar file, so that a non-LGPL jar can make calls to the LGPL jar's implementation, does not cause the including work to be derived from the LGPL work even though java uses late-binding by name (requiring that names be copied into the derived executable), and thus does not (in and of itself) cause the package as a whole to be restricted to distribution as (L)GPL or as open source per section 6 of the LGPL.

and Andy asked:

Is this statement true with regards to the use of LGPL Java libraries by non-LGPL Java libraries?

To which David replied:

If I understand the statement correctly, yes -- that's exactly what section 6 is for.

Seen in context, my reading is that LGPL does not infect Java code that simply references, invokes methods of, or extends an LGPL'ed class. Brad Kuhn's comments mentioned in the slashdot "update" seem to be reiterating that position. I see this as saying all of my examples are not infectious. Joe's comments on that post focus on the word "distribute" in what may be a clarifying way (i.e., the in-memory, late-bound version of code that combines LGPL and non-LGPL code is not distributed, it is assembled at runtime, and therefore not in violation).

I hesitate to ask but the question seems inevitable: what does it mean to apply aspects to LGPL'ed code? If I hook in code to a cut-point in some LGPL'ed code, is my cut-point code infected? Does it matter if my AOP-system is implemented with byte-code manipulation, reflection or composition? Does it matter if this happens at runtime or at compile time? It seems that it would, and that feels like a fairly arbitrary distinction in some ways.

IANAL and this makes my head spin. I'm going back to writing code. Someone let me know when there's a clear answer here. (Unfortunately I think it may take a court to decide that. Let's hope it never comes to that.)

The [L]GPL, Java and Asymmetry

Wed, 16 Jul 2003 17:35:04 GMT

Andy has finally tracked down some answers on applying the LGPL to Java. Dave and Sam had some additional comments.

A few points:

In his comment thread Dave suggests that "Class.forName() might also be a workaround", echoing questions on the LGPL in Java I previously mentioned. The answers to those questions still aren't entirely clear to me, although the position of simply steering clear of LGPL/GPL code for Java is seems increasingly rational.
Sam writes "An example of something that does solve the issue: JDBC" but I'm not sure I follow. Does this mean Class.forName? In other words, I can use a LGPL'd database code as long as I never directly invoke the LGPL code?
Andy writes "Thus we need a license for Java that guarantees contributors will donate back to the library that does not infect the outside code."

Do we? Why? I think there is sufficient incentive to release most derivative works anyway, and if someone doesn't, who cares? The open source project still gets more users, more support, more field testing, mo' better, than it otherwise would. If you make a few bucks selling proprietary extensions to my open source code, more power to you. (You run the threat, of course, of the community re-implementing your proprietary extensions in an open source fashion, thus destroying your business model. This disincentive to selling proprietary derivatives augments the incentives for releasing those derivatives to the community for further extension, testing, maintenance and support.)

A big issue I have with the [L]GPL is the asymmetry of the license. If Company X releases code under the [L]GPL, I'm less troubled by the need to similarly open my derivative works (although the people who sign my paycheck quite reasonably have a different perspective on that--there is some work that is best kept proprietary, if only because it pays the bills) than by the fact that Company X, typically acting as the "umbrella" copyright holder (to both their original work and my contributions) now has more rights to my contributions than I do. Under the viral GPL, I cannot use the larger work under any terms but those provided by the GPL, and in isolation my contributions are probably pretty much useless. Yet Company X (as the holder of the collective copyright) is free to take my (donated) work and theirs and release (or sell) it under whatever terms they choose. I wonder if Stallman had foreseen this consequence when constructing the GPL.

The "umbrella" copyright holder on a BSD- or ASF-type license has the same rights of course, but the license grants me the ability to do pretty much whatever I want with the larger work, short of claiming it all to be my own. There is very little that the copyright holder can do that I, as a user let alone contributor, cannot also do. It seems to me that the license with the least restrictive terms is the one that is most "free".

my pet bug, or another example of how sun doesn't get community development

Tue, 15 Jul 2003 20:02:07 GMT

I've submitted a few insignificant bugs to Sun's Bug Parade over the years, most recently "Collections.ReverseOrder.equals method is lacking", which I submitted back in January of 2003 and was finally accepted yesterday (14 July).

This isn't a critical bug, of course: It's easy to work around (that is, replace) and frankly it's been an annoyance but not a real problem in my development efforts. Moreover, it seems like Sun has been pretty busy over the last few months working on other things. Nevertheless, this whole experience has been frustrating.

First, in my limited and anecdotal experience the time to respond to bug reports is getting much worse. It took six months for the Java team to acknowledge this simple, and if I may say reasonably well documented (including a "patch" and unit test), bug.

Second, this is a good if trivial example of how Sun's community efforts fail in execution. Here's a trivially simple (it's clear from inspection alone) and readily demonstrated (a complete unit test is provided) defect. A trivially simple (one line, two if you add hashCode()) patch is provided, and yet it took six months to get the issue acknowledged. Now I'll wait an indeterminate amount of time to see the problem fixed. All that and this problem could have been analyzed, patched and regression tested in less time than it took me to write this post. Even the slowest moving of open source projects would have had this problem patched, if not released, in a matter of weeks. If Sun can't find a way to move more quickly (and stop chasing the most superficial features of C#) Java seems destined to be eclipsed by community developed languages.

I'll uncharacteristically paraphrase Jerry Seinfeld here: "Sun knows how to take community input, they just don't know what to do with it. And that's really the most important part of community: the doing." (text, audio)

PS: Also note that the second bug I linked to above, "Repackage sun.net.www.content.* Classes (java.net.ContentHandler)", was already addressed by the time I submitted it, I personally added a comment to this effect nearly two years ago, and yet the bug sits marked "In progress" (and with 3 votes, none of them mine).

Axion 1.0 Milestone 2 Released

Fri, 11 Jul 2003 22:24:45 GMT

It seems Friday is a good day for releases:

Axion has released their third binary distribution, just one year after their first release, and only three months since their previous one.

This release adds several new features, including the DML and BTree performance tuning I previously described, as well as support for OUTER JOINs, LIKE clauses and JDBC 3.

Wanted: Modular/Extensible Parser Generator

Mon, 07 Jul 2003 20:02:31 GMT

Over at the Axion project, we're using a JavaCC grammar to implement the SQL parser.

One of things we've found Axion to be good for is unit testing database applications. In other words, one can use an in-memory, in-process Axion database as a "mock" replacement for a regular production database. In this case, it's quite useful to have Axion closely mimic the syntax of other RDBM systems. For example, Axion supports LIMIT and OFFSET clauses like PostgeSQL and MySQL as well as a ROWNUM pseudo-column like Oracle. Similarly, Axion supports the ISO SQL 99 syntax for outer joins (FROM a LEFT OUTER JOIN b ON a.id = b.id) , but it would be nice to support Oracle's custom syntax (FROM a, b WHERE a.id = b.id(+)) as well.

Supporting the idiosyncrasies of several of the popular database engines in a single grammar file seems cumbersome at best. and probably impossible. It tends to bloat our keyword namespace. Eventually, it must lead to conflicts. (For example, if I'm trying to unit test code that eventually interacts with an Oracle database, then the "mock" database shouldn't accept LIMIT and OFFSET clauses, and shouldn't consider those to be keywords either.)

Axion's design is modular enough to allow for pluggable parser implementations. Indeed, anything that implements:

interface Parser {
  AxionCommand parse(String sql) throws AxionException;
}

can be dropped right in. Hence it is straightforward to define, for example, MySqlSyntax.jj, OracleSyntax.jj, SqlServerSyntax.jj, etc. to support a specific SQL dialect. The trouble is, each of those files is going to be 90% or more the same. What I'd like is a clean mechanism for either:

declaring SpecializedGrammar to extend from GeneralizedGrammer, perhaps with abstract productions and the like, or
declaring SpecializedGrammar to be the composition of SubGrammarA and SubGrammarB

or both. I'm especially interested in being able to combine grammars at runtime. Anyone have any suggestions? Can you point to an example?

(I'll be honest, I'm not much of a *CC expert. This might be straightforward in JJTree or ANTLR, I just haven't dug into it much. I'm pretty sure it's not straightforward in plain old JavaCC.)

Re: Test Driven Development versus Component Reuse

Tue, 01 Jul 2003 22:01:22 GMT

Over on the Software Craftsmen blog, Mike Hogan asks what is meant by "the simplest thing that could possibly work". For what it's worth, Beck actually addresses this point directly in Extreme Programming Explained [ISBN:0201616416]:

Here is what I mean by simplest--four constraints, in priority order:

The system (code and tests together) must communicate everything you want to communicate.

The system must contain no duplicate code. (1 and 2 together constitute the Once and Only Once Rule).

The system should have the fewest possible classes.

The system should have the fewest possible methods.

[...]
If you view the design as a communication medium, then you will have objects or methods for every important concept. You will choose the names of the classes and methods to work together.

I have my own issues with that definition that I hope to pick up in a later post. (For starters, define "everything you want to communicate".)

Mike goes on to question whether there are times when TSTTCPW conflicts with design for reuse.

I think this misses the test first aspect. Consider approaching the Grep example test first. A small set of tests that might lead to the first Grep implementation is:

static final String TEXT = "This is\na simple test";
Grep grep = null;
BufferedReader reader = null;
 
void setUp() {
  grep = new Grep();
  reader = new BufferedReader(new StringReader(TEXT));
}
 
void testDoesContain() {
  assertTrue(grep.contains(reader,"is");
}
 
void testDoesNotContain() {
  assertFalse(grep.contains(reader,"is not");
}
 
void testPatternsSplitAcrossMultipleLinesAreNotFound() {
  assertFalse(grep.contains(reader,"is*a");
}

Mike asserts that the Grep implementation would be more useful if it could interoperate with multiple regular expression frameworks, and provides an example "Inversion of Control" approach for doing so. Want to make that Grep implementation work with multiple regular expression frameworks? Great. First, write a test that fails:

void testGrepWithJakartaRegexp() {
   RegexpProvider rep = new RegexpRegexpProvider();
   ReusableGrep grep = new ReusableGrep(rep);
   assertTrue(grep.contains(reader,"is");
   assertFalse(grep.contains(reader,"is not");
}
 
void testGrepWithJakartaOro() {
   RegexpProvider rep = new OroRegexpProvider();
   ReusableGrep grep = new ReusableGrep(rep);
   assertTrue(grep.contains(reader,"is");
   assertFalse(grep.contains(reader,"is not");
}

(Alternatively, (1) define a "mock" instance of RegexpProvider for the purpose of this test rather than using specific implementations and (2) define an abstract getRegexpProvider method in your test class, an implement these tests as concrete extensions of that abstract test case, but I digress.)

Now we can justify the creation of the RegexpProvider interface, and ReusableGrep still meets the "simplest" criteria. (Ignoring that ORO and Regexp likely support slightly different syntaxes.)

I think Mike's first instinct--simple is "the smallest amount of code" that will "get the test case[s] running and refuses to concern itself with any potential future requirement" is the right one. Have additional requirements you'd like to assert? Then express them as tests and this simple rule allows you to support them. When approaching development test first I think a lot these questions about "what is simple" simply fade away, as does much premature generalization. (And I say that without taking a position on whether ReusableGrep represents premature generalization or not. I recognize that it's meant as a trivial example.)

PS: I can't resist the tempation to plug a commons-functor approach to this Grep implementation. How about something like:

Lines.from(reader).contains(RegexpMatch.of("my expression"))

Like Clover? Check out JCoverage

Mon, 30 Jun 2003 14:59:57 GMT

Like Clover, JCoverage is a code coverage analyzer for Java. "Instrument" your code with either of these tools, run your unit test suite (really, execute the code in any way), and one can generate a report on what was executed (lines, methods, branches, etc.), and more importantly, what wasn't.

Unlike Clover:

JCoverage is GPL'ed
JCoverage instruments the byte code (via BCEL) rather than the source, which seems substantially faster, at least under casual observation.
JCoverage is clever enough to not instrument select lines--lines that invoke log4j for example--which means that logging calls don't pollute your coverage metrics, whether or not you run the test suite with logging on.
JCoverage can generate a complete, parsable coverage report in XML from which you can render custom reports or derived statistics.
JCoverage includes custom Ant tags for instrumentation, reporting and even asserting levels of coverage at a fine grained level.
JCoverage makes it easy to merge coverage databases across several runs--for unit and functional tests, for example, or to create a single report for independently built components. (Of course, given the XML report, it may be tedious but shouldn't be difficult to do this sort of merge on "manually" either.)

With JCoverage, I think I'll need to reconsider my position that coverage analysis may be too slow to execute with every continuous integration build.

Re: Why Java is not Open Source

Thu, 26 Jun 2003 22:47:03 GMT

The CTO for Sun's Desktop Division Hans Muller writes:

I think that one of the primary reasons that Java is not an open source project is that given the size of the developer community, forks are unacceptable. In other words the millions of developers who build software on top of Java value its stability more than they value the right to get under the hood and fix it.

Hrrrm. That seems like a moderately controversial statement to me, for several reasons:

The community's experience with Python (source), Ruby (source), Perl (source) (and others) might be actively proving otherwise.
The metaphor about cowboys and power plants, like many physical metaphors for software development, simply doesn't work. A version control system alone would alleviate this problem, as would a "gatekeeper" as Muller himself describes in the proceeding paragraphs.
Why not just define a specification and a TCK, hang on to the Java trademark and make folks pay to describe their JRE as such. It's precisely the same strategy used for the J2EE brand, isn't it? Oh, now I get it.
Simply adding a patch mechanism to the Bug Parade and actually accepting them once in a while would seem to be a dramatic step forward, giving Sun many of the benefits of open source (and the development community a few) with little or no risk.

I didn't and still don't expect Sun to open source Java, although not for any of the reasons Muller describes. Yet not having followed the java.net phenomenon very closely, I had took it to be a sign that Sun is getting more clueful about the role of the developer community, and of the open source developer community in particular, in the success of Java, especially relative to the Java Community Process. Perhaps I was wrong.

Experimenting with Jester

Wed, 25 Jun 2003 19:29:35 GMT

In a previous post I alluded to the use of mutation testing to evaluate the completeness of a test suite, rather than relying upon pure test coverage metrics. In a comment to that post Adewale Oshineye suggested that I check out Ivan Moore's Jester, a Java/JUnit based mutation testing tool. I'd seen Jester before, but I'd never used it nor looked at it in much detail. The anecdote about what Jester uncovered in Bob Martin's and Robert Koss's test driven bowling score calculator example was certainly interesting, so this morning I bit the bullet and downloaded a copy.

Getting Jester up and running was a minor hassle (and it seems like much of that hassle could be alleviated), but I've written my share of open source projects with quirky configuration and installation, so I'll leave that alone. Once my files were placed in the proper position and after tweaking the python scripts to get them to run on whatever version of python I've got on my RedHat box, Jester ran slowly (waiting on javac) but well. It ran much faster once I figured out how to tell Jester to stop mutating my comments (set shouldRemoveComments=true in jester.cfg). The result was a basic HTML report like this one.

(On an unrelated note, is there something more or less equivalent to python -version?)

As an experiment, I took a small component (196 non-blank, non-comment lines of code spread over 34 methods) I knew to be well tested (100% coverage of statements and conditional expressions) and ran it through Jester. It found 21 mutations total, 2 of which didn't lead to a test failure. Those were (with the modified code in bold):

if(TRUE || MESSAGE_LOG.isDebugEnabled()) {
   MESSAGE_LOG.debug("Broadcasting " + msg);
}

and

List list = new ArrayList(_listeners.size() + 12);
list.addAll(_listeners);
list.add(...);

In the first case, believe it or not, I actually had unit tests that confirm that MESSAGE_LOG generates log events when set to the DEBUG priority, and no log events when set to higher than DEBUG priority. (I wouldn't normally do that, except this is the single log message in that component, and I really wanted 100% coverage. Besides, it wasn't that hard, I just added a mock Appender to that Category, and checked to see if a message was added or not.) Of course, both of these tests still pass, even without the isDebugEnabled call, since the debug method won't generate a log event when using a higher priority. Adding a test that fails as a result of this mutation isn't particularly useful, but it isn't difficult either--I just pass in a mock instance of the msg object and check whether or not the toString method is invoked. Not invoking toString is indeed the reason for this if(isDebugEnabled()) block, so maybe that's not such an odd test to have after all.

The second case is the kind of thing Ivan Moore describes as a "false positive" in his writeup on Jester. This code initializes that ArrayList to the precise size it knows will need. Allocating it a little bit bigger or a little bit smaller doesn't alter the functional behavior of the method (although it will be slightly less efficient). Maybe that indicates a premature optimization on my part, but it seems like a pretty small one. In any event I don't see any way to confirm that that List was allocated to precisely the right size without breaking encapsulation profoundly, so I think I'll let that one go.

I had hoped to run Jester on some larger, more complicated but less well tested code (3,287 nc,nb lines, roughly 77% coverage) to get a feel for how it works in a more useful scenario, but I've been unable to get it to complete a run on the this larger component. I may poke around with something in-between, but 4,000 lines is on the smallish side for the kinds of modules I'd want to run this on. I may have better luck mutating a single class at a time.

In short, I think Jester meets Sam Ruby's criteria for a successful open source project--it's a good idea with a bad implementation. I have some thoughts on how to improve that implementation (mainly obvious ones--e.g., use a ClassLoader and an actual parser, or consider mutating the byte code rather than the source) that maybe I'll cover in a later post. All in all, Jester is an interesting project, and like a lot of things, I wish it worked a bit better.

Testing Testing

Tue, 24 Jun 2003 15:09:46 GMT

My fear, and perhaps it's not a well founded one, is that the pursuit of pure test coverage--the percentage of lines, statements, methods, etc. tested--will provide a false sense of completeness. If "percent executed by test code" is your sole metric, it's easy to write superficial tests that execute statements, but actually "test" very little.

For example, a couple of weeks ago I added the following test to one of our suites:

public void testStartStop() {
  AppMain app = new AppMain();
  assertFalse(app.isStarted());
  app.start();
  assertTrue(app.isStarted());
  app.stop();
  assertFalse(app.isStarted());
}

This single test--directly invoking just four distinct methods and comprising only three assertions--executed an additional 1,500 lines or so, roughly 10% of the code for this module. You can be sure that the bulk of the functionality provided by those 1,500 lines is not actually tested here. Indeed, the following (test first) implementation would suffice:

class AppMain {
  void start() {
    started = true;
  }
  void stop() {
    started = false;
  }
  boolean isStarted() {
    return started;
  }
  boolean started = false;
}

Is this test useless? At this point I'll argue it isn't. Superficial testing of these 1,500 lines is better than no testing at all. This test at least confirms that we've got the classpath right (including any resources loaded on application startup) and that there aren't any uncaught exceptions being thrown on startup. (When better tests are in place, this superficial test may indeed become useless.) But the resulting test coverage metric is certainly misleading.

One solution, probably the right one, is to simply develop the code test first. If every change to production code was made to address a failing test (or to refactor under a unchanging test suite) then one certainly wouldn't find himself in this situation. But this solution doesn't address my (our) current situation, in which we have 100,000 lines of "legacy" code and perhaps 60% of that code was developed "test last" if test at all. The lack of tests has become a drag on our ability to refactor or simply work with that code. If it was sufficient to simply tell folks to develop test first, I wouldn't be as concerned with measuring test coverage in the first place.

Mutation testing--in which we randomly change some piece of production code and check to see if our test suite detects the change--is another approach I've seen proposed before, but I'm having trouble buying into that. If we start with poorly factored code, it seems likely that a random change is going to break something, probably profoundly (like leading to an exception being thrown). Even superficial testing will detect those sort of problems.

Alternative but indirect metrics may be a better approach. Tracking the number of distinct tests or facts being asserted might give a rough idea of how robust the test suite is. To do this right I think I'd need some numbers from well tested portions of the code base that correlate size or complexity metrics with the number of tests or assertions in the test suite. These might be interesting numbers to collect.

Re: Liskov's Substitution Principle and JUnit Testing

Tue, 24 Jun 2003 13:58:20 GMT

The author of Manageability has discovered the technique of inheriting test cases. This testing strategy is extremely useful when testing multiple implementations of some interface, and is actually fairly common. In fact, there's not one but two (if not more) general purpose libraries supporting this approach. I distinctly remember seeing this strategy written up in pattern form. You can find similiar write-ups here and here, but neither of those are the one I'm thinking of.

A Frog Boiling Approach to Increasing Test Coverage

Mon, 23 Jun 2003 17:28:51 GMT

Some thinking out loud about concrete goals for increasing test coverage.

The production (i.e., non test) Java code base at my day job consists of roughly 103,000 non-comment, non-blank lines of code, split over 132 "modules". Our automated unit test suite exercises roughly 29% of those lines.

(That 29% figure sounds a little bit worse than it feels to me. Some areas of the code base are well tested, several are even at 100% coverage. Others have few if any tests, but as a result of remaining essentially untouched since before a formal unit testing initiative was launched 30 months or so ago. Yet many modules are woefully undertested, and probably not coincidentally several of those have substantial bloat--the number of lines in those modules is way out of proportion with the functionality they provide.)

Whether that 29% figure is indicative or not, it's clearly much lower than desirable. (Personally I've been striving for and generally achieving 100% coverage for new development.) I've been thinking a bit about laying out concrete goals for increasing this coverage.

We've talked a bit about simply targeting some figure, say 80% coverage, and perhaps some intermediate goals (for example, 40%, then 60%, then 80%).

While the "frog boiling" approach--slowly raising the temperature on test coverage--appeals to me, something about the arbitrary "percent coverage" goals doesn't seem right. I think I'd prefer goals that call for complete (100%) coverage of something, perhaps with different values of "something". I'm not entirely sure why.

Specifically, I'm thinking of the following stages:

All modules have tests. Conveniently, this can be easily and quickly tested at build time. We can make the absence of tests a build failure. It is possible to programmatically evaluate the remaining goals, but not as quickly. We'd have to execute the test coverage check on every continuous integration build, something that may take too much time.
All packages have tests. This should be relatively easy to achieve once the first goal is reached.
All classes have tests.
All methods have tests.

I'm not sure where to go from that point. Complete coverage of conditionals (every boolean expression is evaluated at least once to true and at least once to false) may be a good next step, but isn't as pithy as the other goals. It may be that once we've reached 100% method coverage, 100% line/statement coverage is within easy reach. I suspect that once we've reached that fourth goal, the next step will be pretty clear.

I wonder if any reader has some experience with similar strategies for increasing test coverage through a series of concrete goals. What goals did you select, and why? Did a given step turn out to be too large or too small?

the fixmeister role, or ''you, help me''

Tue, 17 Jun 2003 20:12:10 GMT

In some previous posts, I discussed some of the challenges we've experienced maintaining a continuous integration discipline at my day job. In my last post on this topic, I alluded to "additional measures" which we adopted. Time to describe what I was talking about.

Although I hadn't seen it at the time of the events described below, I recently ran across a comment on Ward's Wiki that accurately captures the problem we encountered and its solution:

There was an interesting psych study along these lines. There is one "victim" and several "bystanders". Each of the bystanders individually would be perfectly capable of helping the victim, but since they NeverVolunteer, none of them do. This happens in real life situations. If you are the victim and want help, what you need to do is to pick one person from the bystanders and say to them specifically "you help me". If you just do "somebody help me" it won't work. -- AndyPierce [from Never Volunteer]

Toward the end of April, in the midst of one of the longer low points in our development cycle, I sent the following (slightly edited) email to my peers on the technology management team. At the time it felt a bit like a failure.

Subject: build failures, and what to do about them
Sent: Wed 4/23/2003 10:29 AM
There was a time, not so long ago, when we would regularly see a dozen or more good integration builds per day. In fact the vast majority of builds were clean ones. (See [internal link to an historical report on continuous integration builds].) Recently (meaning the past few months) we're lucky to see a dozen good builds in a week.
As we've seen rather directly the past couple of weeks, our inability to regularly integrate changes across the entire code base slows our development process, delays production releases and may threaten our ability to deliver products according to schedule. Moreover, integration problems compound themselves. When we go several hours (let alone several days) without a clean build, it's no longer just one problem we need to fix, but several problems that are hidden behind the first one, masked because problems in some dependent library stops the build before it discovers the later problems.
These integration problems are real problems. A "broken build" means that there is some code that either doesn't work at all, doesn't work in relation to other libraries, or doesn't work outside a particular developer's (or developers') sandbox. These problems may not impact every developer at all times, but it is quite likely they impact some developer. It is guaranteed that they will impact the entire technology team and for that matter the entire enterprise when it comes time to release a product (and as I understand it, we have a few of those queued up in the near future). By and large, the way in which developers avoid this problem is to simply not update their local repositories. That's a false sense of progress if there ever was one.
Some developers have complained about this problem. Some managers have as well. We've made increasingly obnoxious efforts ([internal link to our nag servlet, as described in a previous entry]) to increase the visibility of this problem, but it's not getting better. And due to the compounding behavior described above, when it's not getting better, it's getting worse.
I propose we do the following: For each series of build failures (that is, for every continuous sequence of failures, no matter how many underlying causes they might have) we name one arbitrarily selected developer to be "on point" for achieving a clean build. Achieving a clean build becomes this developer's top priority (if they need to weasel out of it in order to something that is "more important", they do that by finding someone else willing to trade slots with them). In exchange for this responsibility, we give this developer the authority to grab whomever then need to do whatever they need to accomplish this goal. If Alice, being on point, needs Jose's help to diagnose a problem, or needs Jose to make changes to some library in order to bring said library into sync, then this becomes Jose's top priority as well (see the "weaseling out of it" strategy above). Similarly, if Alice, being on point, needs Jose to take over some task Alice would otherwise be working on, she is empowered to do that as well.
Why?
1) Having a code base that is not integrated is costing us a substantial amount of developer time, which means it's costing the company a substantial amount of money and threatening release schedules.
2) Things that are everyone's responsibility become no one's responsibility. Actually, this isn't really it. More accurately, when we fail to send a message that things that are everyone's responsibility are important, those things become the responsibility of a handful of truly responsible people. There are developers who take build failures seriously, and make a well above average effort to address them. These efforts are undermined by those who don't take build failures seriously (or insufficiently seriously), and eventually, the patience of those good Samaritans wears thin. I know mine did several weeks ago and I've notice a degradation in the integration quality since that time. It was my hope that others would step up to take greater responsibility for integration builds, some did, but seemingly not enough.
[It may seem that I'm overestimating my contribution there, but I literally, personally, addressed perhaps 30% of the build failures for several months. I looked at each and every build failure, and when it seemed no one was making progress in addressing it, I'd fix it myself. Frankly, I did this because (a) when our CI process was first initiated, there were a number of detractors that claimed this simply wouldn't work, and (b) it seems like this simply the right approach to a CI process--I'd expect everyone to do more or less the same thing.]
3) Making fixing the build a top priority sends the message that we take continuous integration seriously, as a management team, as a development team, and as an enterprise.
4) Selecting an arbitrary developer to be on point offers a number of advantages:
a) It increases the visibility of build issues, the kinds of problems that cause them, and an effective strategy for resolving them, across the whole team. Folks who may have never before thought about how to diagnose an arbitrary problem reported by the CI server are forced to do so. Folks who may have never reflected on the kinds of changes that are problematic, or how to make changes in an incremental and backwards compatible way are exposed to those sorts of issues and those sorts of solutions.
b) It applies stronger peer pressure. For quite some time I would personally "nag" folks to fix build issues, even sit down with them to explain the nature or the problem and the needed fix. After a while, this nagging decreases in effectiveness, since "Rod's always complaining about the build". When several different folks in a week are nagging you to fix problems you've caused, the pressure not to cause such problems (or to fix them on you own initiative) is increased. When several different folks find themselves nagging the same person over and over again, the pressure on that person to improve their personal process is increased.
c) It causes greater knowledge sharing. When put on point to fix a problem in some otherwise unknown library, a developer is forced to learn something about it, probably by pairing with someone more knowledgeable about it. When a developer finds that they are consistently being "bothered" to update a library because they are the only ones that understand it, the pressure to make it more understandable or to share some of that understanding is increased.
d) Indirectly, it sends the message that fixing the build really is everyone's responsibility. The arbitrary developer "on point" is simply putting a face on the team's concerns.
I'd very much like to start putting people "on point" for fixing the build today, probably by calling an all developer meeting to explain this protocol. I'd also very much like to not have this protocol undermined by some direct or indirect message of "well, that's important, but this is more important" (where "this" is some more urgent, but not necessarily more important task). We have more than enough developers to manage both the urgent and important tasks on our plate, and I'll argue that an approach that doesn't make integration a top if not the top priority is going to cost us in the long run. If anyone has objections or an alternative strategy, I'd love to hear about it.

This strategy was well received by the management team, and was implemented almost immediately. This strategy was less well received by some members of the development team, but there was at least grudging acceptance from everyone.

The team has come to call this role the "fixmeister" (a name which I'm personally not very fond of). The first fixmeister had the excellent suggestion that each subsequent fixmeister be selected by the current one, which has led to an interesting variety of selection algorithms, some whimsical, some instructional, some a bit malicious.

In the eight weeks or so since this process was initiated, we've been around the team slightly more than once. It's been effective in achieving the primary goal--increasing the ratio of build successes to build failures. I'd like to think it's been effective in achieving some of the harder-to-measure goals as well. (It's certainly increased awareness of the CI process among the less process oriented members of the team, and perhaps taken away some of the mystery of the process.) Maybe time will tell on the other points.

The Pleasures of Profiling

Mon, 05 May 2003 21:52:21 GMT

Prompted by Nick's post about Axion's insert performance and some internal product delivery schedules, I've been doing some profiling and tuning of Axion's performance (with some good results). I'm reminded of how much I truly enjoy this activity.

I can tell you don't believe me, so I took some notes and wrote up a brief account.

Unit tests with log statements are a code smell

Fri, 02 May 2003 18:15:02 GMT

While this is not an earth shattering realization, I've come hold the opinion that log statements (log4j, logkit, java.util.logging, commons-logging, what have you) within unit tests are a code smell, perhaps universally.

While I'll sometimes add a few System.out.println calls to a unit test while I'm trying to diagnose a particular failure, configuring a full-blown logging setup within a unit test always seemed like more time and trouble than it was worth. From time to time I'll encounter a heavily logged TestCase in some code base I'm working with. The more I work with such TestCases, the more I find this to be an indication that something is not right.

Here's why:

I find it hard to imagine a test first/test driven development approach that leads to log statements within unit tests (but I can imagine "test last" approaches that will). The presence of logging strongly suggests that the tested code was not developed in a test driven fashion.
The role of logging frameworks and that of automated unit testing frameworks are at odds. Logging calls provide persistent, if only intermittently used, diagnostic and informational messages, typically intended for manual inspection ^*. Automated unit tests are meant to be self-interpreting, success or failure should be obvious without manual inspection. The use of diagnostic or informational log messages within unit tests suggests your tests aren't sufficiently self-interpreting.
Anecdotally, the objects being tested by these cases are brittle in the face of change. This may stem from poor factoring: there are too many subtle and perhaps unplanned interactions between methods, or methods aren't well focused enough to allow for orthogonal changes.
Anecdotally, test failures remain difficult to diagnose and fix despite the log messages. This also stems from poor factoring: the individual test cases and assertions are too coarse grained to help identify the root cause of a test failure.

* If you find yourself wanting programmatic inspection of log messages, I'll suggest you're looking for messaging, not logging.

build cycles, development cycles, and the nag server

Wed, 30 Apr 2003 01:32:15 GMT

One thing I've noticed about our continuous integration process and its meta-process is that both are cyclical. Here's three examples:

..FF...FFFFFFFFFF...., or when it rains, it pours

One of the first things you notice when applying a continuous integration process to parallel development is that integration problems compound themselves. (This isn't a side effect of continuous integration, CI simply makes this point clear.)

On most days we'll have (indeed, we expect) the occasional build failure--someone neglected to check in a file here, or forgot that component was using this method whose signature has changed, etc. On good days, the team is on its collective toes and responds quickly to the failure notice. The problems are resolved in a build or two and we're right back on track. On bad days, an integration problem will persist for several builds, and as yet undiscovered and unreported problems will begin to queue up behind the first one as folks continue to make commits. The longer the queue of integration problems get, the longer it gets. As a result, bad days have a nasty habit of turning into bad weeks.

Anecdotally, the tipping point seems to be around three consecutive build failures. When we hit the third consecutive failure, the odds of hitting a fourth, a fifth, or even a tenth failure seems to increase dramatically. I think there are at least two reasons for this: (1) Unless the developers notice and fix the problem on their own--independent of the CI build failure notifications--failures will almost always come in pairs. By the time the nag email is delivered, let alone diagnosed and fixed, a second build will already be in progress. If the problem is going to be fixed "immediately", then it will be fixed for the third build, but generally not before it. (2) Depending upon the specific component that encounters problems, a failed build takes 20 to 40 minutes. Three consecutive failures means its been an hour, maybe two, since our last complete build. On a busy day we can have quite a few commits in an hour, so the number of unintegrated changes grows quickly. All of these changes need to be addressed before we reach a successful build.

two steps forward, one step back

We've found that our CI process doesn't progress monotonically. We're constantly trying to strike a balance between the functionality provided by the CI builds, the time it takes to complete a build cycle, and the likelihood of extraneous build failures. Often we find that we've stretched a bit too far and need to pull back.

Often the time it takes to complete a build is a driving factor. For a time we generated and published JavaDoc documentation and various source code and test coverage metrics following every successful build, but we found this added too much time to the build cycle. (As discussed above, the longer changes wait to be integrated, the greater our exposure to risk.) Instead, we rely upon cron-driven or manual processes to generate these artifacts.

At times, and to my great frustration, we've had to remove aspects of the build that were simply too brittle for continuous use. Our Latka-based functional test suite, which tests a number of our web applications, was largely removed from the CI builds largely because of test rot and the instabilities introduced by being too dependent upon external services that change outside of the build process (database servers are one, although not the only example here). The CI process still deploys our web based applications, but leaves most of the functional testing to manual invocation.

the nag server

From time to time we find that the discipline of continuous integration begins to slip: successful builds become less frequent and broken builds are fixed more slowly. The team gets used to seeing frequent build failures, begins ignoring nag messages, becomes complacent, and tends to look for local workarounds rather than addressing the global integration issues. (This is an instance of the fix broken windows pattern.) Despite what Fowler will tell you, in my experience there are some developers who are more than happy to give up the benefits of a continuous integration process, or who fail to recognize what those benefits are in the first place. (Perhaps not coincidentally, many of these developers haven't been working here as long as the CI process has.) In part these cycles are an extension of the cycles above--cruft and insufficiently considered work-arounds build up until the code base is fundamentally brittle. In part these cycles are related to the evolution of the code base--major, cross-component refactorings and shotgun maintenance sometimes lead to periods of build instability. We've found that whatever the cause sometimes the team needs a little kick to get back on track.

Morgan recently suggested one pleasingly simple kick of this sort. While discussing a low point in our CI cycle, Morgan half-jokingly suggested that what we needed was a "nag server"--a giant, prominently located monitor that displays the status of the current build. We jumped on it literally immediately, grabbed the biggest monitor we could find and an underused server; and set them up in the hallway in front of the primary development cube farm with a web browser continuously refreshing a simplier, bigger and bolder version of our build status page. Morgan later added an automated count of the number of consecutive build failures or successes (replacing a flip chart we manually updated for a while).

The nag server worked rather well for a while--it improved the build success:failure ratio and increased the visibility of the continuous integration process both within the development team and among the "customer" team. We often find developers gathered around the nag server discussing the current integration problem or checking to see if their changes made it into the current build. Unfortunately, the nag server eventually became less effective as a motivational tool. Additional measures seemed necessary. More on that in a later post.