One thing I've noticed about our continuous integration process and its meta-process is that both are cyclical. Here's three examples:

..FF...FFFFFFFFFF...., or when it rains, it pours

One of the first things you notice when applying a continuous integration process to parallel development is that integration problems compound themselves. (This isn't a side effect of continuous integration, CI simply makes this point clear.)

On most days we'll have (indeed, we expect) the occasional build failure--someone neglected to check in a file here, or forgot that component was using this method whose signature has changed, etc. On good days, the team is on its collective toes and responds quickly to the failure notice. The problems are resolved in a build or two and we're right back on track. On bad days, an integration problem will persist for several builds, and as yet undiscovered and unreported problems will begin to queue up behind the first one as folks continue to make commits. The longer the queue of integration problems get, the longer it gets. As a result, bad days have a nasty habit of turning into bad weeks.

Anecdotally, the tipping point seems to be around three consecutive build failures. When we hit the third consecutive failure, the odds of hitting a fourth, a fifth, or even a tenth failure seems to increase dramatically. I think there are at least two reasons for this: (1) Unless the developers notice and fix the problem on their own--independent of the CI build failure notifications--failures will almost always come in pairs. By the time the nag email is delivered, let alone diagnosed and fixed, a second build will already be in progress. If the problem is going to be fixed "immediately", then it will be fixed for the third build, but generally not before it. (2) Depending upon the specific component that encounters problems, a failed build takes 20 to 40 minutes. Three consecutive failures means its been an hour, maybe two, since our last complete build. On a busy day we can have quite a few commits in an hour, so the number of unintegrated changes grows quickly. All of these changes need to be addressed before we reach a successful build.

two steps forward, one step back

We've found that our CI process doesn't progress monotonically. We're constantly trying to strike a balance between the functionality provided by the CI builds, the time it takes to complete a build cycle, and the likelihood of extraneous build failures. Often we find that we've stretched a bit too far and need to pull back.

Often the time it takes to complete a build is a driving factor. For a time we generated and published JavaDoc documentation and various source code and test coverage metrics following every successful build, but we found this added too much time to the build cycle. (As discussed above, the longer changes wait to be integrated, the greater our exposure to risk.) Instead, we rely upon cron-driven or manual processes to generate these artifacts.

At times, and to my great frustration, we've had to remove aspects of the build that were simply too brittle for continuous use. Our Latka-based functional test suite, which tests a number of our web applications, was largely removed from the CI builds largely because of test rot and the instabilities introduced by being too dependent upon external services that change outside of the build process (database servers are one, although not the only example here). The CI process still deploys our web based applications, but leaves most of the functional testing to manual invocation.

the nag server

From time to time we find that the discipline of continuous integration begins to slip: successful builds become less frequent and broken builds are fixed more slowly. The team gets used to seeing frequent build failures, begins ignoring nag messages, becomes complacent, and tends to look for local workarounds rather than addressing the global integration issues. (This is an instance of the fix broken windows pattern.) Despite what Fowler will tell you, in my experience there are some developers who are more than happy to give up the benefits of a continuous integration process, or who fail to recognize what those benefits are in the first place. (Perhaps not coincidentally, many of these developers haven't been working here as long as the CI process has.) In part these cycles are an extension of the cycles above--cruft and insufficiently considered work-arounds build up until the code base is fundamentally brittle. In part these cycles are related to the evolution of the code base--major, cross-component refactorings and shotgun maintenance sometimes lead to periods of build instability. We've found that whatever the cause sometimes the team needs a little kick to get back on track.

Morgan recently suggested one pleasingly simple kick of this sort. While discussing a low point in our CI cycle, Morgan half-jokingly suggested that what we needed was a "nag server"--a giant, prominently located monitor that displays the status of the current build. We jumped on it literally immediately, grabbed the biggest monitor we could find and an underused server; and set them up in the hallway in front of the primary development cube farm with a web browser continuously refreshing a simplier, bigger and bolder version of our build status page. Morgan later added an automated count of the number of consecutive build failures or successes (replacing a flip chart we manually updated for a while).

The nag server worked rather well for a while--it improved the build success:failure ratio and increased the visibility of the continuous integration process both within the development team and among the "customer" team. We often find developers gathered around the nag server discussing the current integration problem or checking to see if their changes made it into the current build. Unfortunately, the nag server eventually became less effective as a motivational tool. Additional measures seemed necessary. More on that in a later post.

I've been tinkering with a long post on some experiences with continuous integration, but since I'm having trouble finding the time to put it all together, I'll try splitting it up into multiple posts (as seems to be all the rage these days). At the very least this should help me get something posted.

Part 1, a little background.

At the shop where I work we've been running an increasingly continuous integration service for a little over two years. Our first fully automated, detect-a-change, build-and-smoke-test occurred on 15 October 2001. Prior to that we had been running a complete build-and-smoke test at least nightly for several months.

Through fits and starts, this CI process has grown to be pretty comprehensive. It performs a complete build-and-unit-test, and in limited circumstances, deploy-and-functional-test across nearly 100 modules (roughly 200,000 non-blank, non-comment lines of code) supporting various internal and external, server, web-based and desktop applications being developed full time by more than 20 developers. The service is based upon a modified version of CruiseControl (1.2.1) driving a common Ant build script, JUnit unit tests, Latka and jfcUnit functional tests. Build results are reported through email and on our intranet. Clean builds are tagged, and the generated artifacts are placed in a repository that serves as the foundation for both production deployments and sandbox development. (Curiously, we've found CVS to be one of the weakest links in that tool chain, but that's a topic for another day.)

It has its problems, but all in all, it's an admirable, perhaps even enviable, setup. In later posts I hope to discuss some of those problems and some observations we've made along the way.

I work for a company that at one time tried to position itself as a "dot com" and as a result I've seen a lot of friends and colleagues come and go over the years. This inevitably results in a "farewell" email of one form or another. For reasons both morbid and practical I archived many of these emails. I named this mail folder "I quit", although it might more accurately be named "I've been laid off". I find the variety of length, content, and tone fascinating. Some are short but sincere: "It has been a privilege (and a blast) working with you, but I am moving on..." Others are business-like and more than a little bitter: "As of [date] I will be resigning from my position as [title] of [department]." Others strive to be poetic: "...and so castles made of sand, fall in the sea, eventually." Others wax philosophical: "I imagine it was probably a dreich, sodden morning as two gentlemen nervously paced the stony, moss-covered steps..."

Going through my mail yesterday afternoon, I had a sensation while flipping through the April 18th edition of InfoWorld magazine similar to the one I have when looking through my "I quit" archive. It seems that InfoWorld is changing formats a bit and has bid farewell to a number of its columnists. I'm not a regular reader of InfoWorld, and probably couldn't identify any of its columnists by name, but the number of farewell columns is still a little sad, a little chilling, and strangely, morbidly fascinating nevertheless. You can see an index of InfoWorld columns on their site, but it looks like that probably changes from month to month.

Recently I realized that I've almost subconsciously been following a design heuristic that even I found surprising--I avoid concrete parent classes. Since coming to this realization, I've been struggling (and failing) to come up with a good case for an instantiable parent class. Is there a case for classes that are neither abstract nor final? Could concrete parent classes be a code smell?

full story...

For example, yesterday’s added entries include "Radio Tidbit: Waypath It! Macro" and "The Hundred Year Language (update on "Bruce Eckel has an interesting blog")", even though neither of them were published on 15 April 2003, and neither of them are published on my java channel.

My theory is this: javablogs is correctly tracking the java feed to determine when my blog has changed, but is pulling from the general feed to obtain new entries. That's why entries that were several days old and unrelated to Java suddenly got pulled in to the javablogs aggregator (because a java related post, "I/O Iterators for Java" finally came along).

My test is this. This entry is only categorized under my main feed. It will sit here quietly in the general feed and not the java feed, and will not appear on the javablogs site until I post a new entry under my Java category, when it will pull in both that new and this old entry.

I’ve noticed other folks posts sometimes bunch up in this way, I wonder if others have the same problem.

I suspect the underlying cause for this is some minor bug in the javablogs record update process. I had originally registered my general feed with javablogs, and later changed that record to only point to my java category.

This allows one to treat any InputStream as a ByteIterator (and hence as an Iterator over Bytes) without resorting to a byte array buffer. This means that Iterator-based APIs like the functor algorithms can be applied to I/O streams. Similarly, this allows one to treat any ByteCollection (or Collection of Bytes) as a source for InputStreams. This means that stream-based APIs can be applied to ByteCollections and Collections of Bytes.

This brings Java one small step closer to the conceptual uniformity and resulting combinatorial expressiveness enjoyed by languages like Lisp. This is reason enough in my mind to extend the primitive collections to Char/char and to Reader-adapters as well.

(Also see this related post.)

The full story is here.

This is the address that was summarized in Bruce Eckel's weblog, as I previously mentioned.

I'd pull quotes, but it's really an good read all the way through if you're at all interested in language design or evolution. (And nearly any program design really is language design.)

[via lemonodor]

/**
 * First Para.
 *
 * Second Para.
 *    * bullet one
 *       * _sub_ bullet
 *    * bullet two
 */

/**
 * First Para.
 * <p />
 * Second Para.
 * <ul>
 *   <li>
 *    bullet one
 *    <ul>
 *     <li><i>sub</i> bullet</li>
 *    </ul>
 *   </li>
 *   <li>bullet two</li>
 * </ul>
 */

Neat, huh? Wish I'd thought of that.

I haven't tried it yet, but it seems like a really useful tool for those who find that HTML formatting within JavaDoc comments interrupts the flow of coding.

I've had to do a hard reboot a few times (most my fault, not the Zaurus's) and hence reinstall any apps that aren't included in Sharp's ROM. As a result I've whittled down my list of applications a number of times. Here (in no particular order) are the apps that make the cut:

embeddedkonsole

What's the point of having a linux box in your pocket if you can't get a console window? The Sharp supplied CD has a workable terminal application, but embeddedkonsole adds some neat features, notably support for multiple, tab navigated consoles.

MinIDE

A solid text editor, built with Java. To be honest I haven't tried many alternative editors on the Zaurus, but this sure beats the basic Text Editor or vi through the console. Calling it an IDE isn't too much of a stretch, you can invoke a compiler (any compiler) directly from the editor.

Opie-Reader

Opie-Reader is an "e-text" reader, which renders various compressed text formats. Using JPluck or something like it, you can copy web sites into a local .pdb archive and copy them over to your Zaurus for viewing with Opie-Reader.

Tiny Scheme

A Zaurus port of Tiny Scheme. A Scheme interpreter, interactive terminal and help files on your Zaurus. I've been using this to work through the exercises in Structure and Interpretation of Computer Programs, and my "computer lab" is significantly smaller than the book itself.

IQNotes

A great outline/tree editor, with an XML file format. A pretty good example of palmtop HCI design, in my opinion.

Java Compiler

Grab the classes.zip archive from a JDK 1.1 release and you can run your favorite Java tools (javac, jar, javap, etc.) via the Jeode EVM. (I've heard rumors of folks getting Kopi or Jikes running on the Zaurus as well, but I've never tried it personally. I'm unable to find any definitive links, but if you google you can find some hints.)

XSLT Support

There may be better ways to do this, but one easy way to get XSLT transform support on the Zaurus is to grab James Clark's XP xml parser and XT xslt processor. I've got stylesheets that transform various IQNotes generated documents into HTML reports (which I can then view right on the Zaurus with Opera). If anyone has had success getting a more modern XSLT system up and running on the Zaurus, I'd love to hear about it. I expect one could grab some C tools to do the job.

SSH

ssh or scp to or from your Zaurus. Very handy way to transfer files, or to have a big (remote) console and keyboard for editing files that are on the Zaurus.

See the Zaurus Software Index for a comprehensive directory of Zaurus software.

On a related note, I recently bought some compact flash cards and a desktop compact flash reader/writer with the intention of trying out Open Zaurus, but I haven't had a chance to install it yet.

• Jakarta-Commons Collections: Complete migration of the primitives package to the new purely primitive interfaces with Collection adapters. Move toward a 3.0 release.
• Jakarta-Commons Functor: Repackage the various org.apache.commons.functor.core.* subpackages to org.apache.commons.functor.*. I had initially expected a tighter dependency between core. and core, in practice there are no dependencies.
• Jakarta-Commons Functor: Normalize the names of the methods used to obtain the static instances of the stateless functors. Probably something as uninspired as getInstance.
• Axion: Update Axion to use the new primitive collections interfaces instead of the old.
• Axion: Add Axion to Gump, now that Axion builds with JDK 1.4.
• Jakarta-Commons Pool: Introduce functor-based interfaces and implementations for specialization through composition.
• Jakarta-Commons DBCP/DB-Commons: Extract JDBC-proxy types from DBCP, use it and the functored pool implementations to clean up DBCP. In particular, use composition to support the folks who want features like "abandoned" object detection and the like.
• Jakarta-Commons DBCP: Use Axion to implement more complete/robust unit tests for DBCP.
• Jakarta-Commons JUX: The trivially simple JUX (JUnit eXtensions) package may be helpful for some of the above, as might the "tunneled exception" stuff I previously mentioned.

I've got some other tasks in mind of course, but these are the ones that have been floating around for a while. We'll see if this helps act as a motivator, it should at least be useful for my own reference.

Morgan posted a similiar list a while back (a blog entry that was lost it seems). I'm not sure he found it helpful, but I like the idea of having some modern equivalent of .plan files.

1. Import and use a type from org.gnu.foo:

   import org.gnu.foo.Bar;
   class ClientCode {
     public static void main(String[] args) {
       Bar bar = new Bar();
       bar.someMethod(args[0]);
     }
     /* ... */
   }

   or

   import org.gnu.foo.Bar;
   class MoreClientCode {
     void clientMethod(Bar bar) {
       bar.someMethod("xyzzy");
     }
     /* ... */
   }

2. Extend from a type in org.gnu.foo:

   import org.gnu.foo.Bar;
   class ClientCode extends Bar {
     /* ... */
   }

3. Specialize a type from org.gnu.foo, but via composition rather than inheritance:

   import org.gnu.foo.Bar;
   class ProxyBar {
     ProxyBar(Bar bar) {
       this.bar = bar;
     }
    
     void someMethod(String str) {
       return bar.someMethod(str.toUpperCase());
     }
    
     /* ... */
    
     private Bar bar;
   }

4. Invoke a method from org.gnu.foo using "hard-coded" reflection:

   class ClientCode {
     public static void main(String[] args) throws Exception {
       Class klass = Class.forName("org.gnu.foo.Bar");
       Method method = klass.getMethod("someMethod", new Class[] { String.class });
       Object obj = klass.newInstance();
       method.invoke(obj,new String[] { args[0] });
     }
   }

5. Invoke a method from org.gnu.foo using runtime-determined reflection:

   class ClientCode {
     public static void main(String[] args) throws Exception {
       Class klass = Class.forName(args[0]);
       Method method = klass.getMethod(args[1], new Class[] { String.class });
       Object obj = klass.newInstance();
       method.invoke(obj,new String[] { args[2] });
     }
   }

   invoked via

   java ClientCode org.gnu.foo.Bar someMethod xyzzy

6. Invoke a method from org.gnu.foo as an external process:

   class ClientCode {
     public static void main(String[] args) throws Exception {
       Process p = Runtime.getRuntime().exec("java org.gnu.foo.Bar xzyyz");
       /* ... */
     }
   }

How do these answers change if org.gnu.foo is licensed under the "regular" GPL?

Of course, Java's not the first or only language with a refactoring tool, indeed there's a long list of refactoring browsers available on Fowler's site. And Eclipse supports much more than Java. Nevertheless, now that mature refactoring tools are available for Java, the speed and degree to which they've caught on (perhaps even dominate?) is pleasantly surprising.

It's been a long time since I used templates (in C++), and I haven't looked at the public review draft of JSR 14 in any detail, but it seems like something along those lines might work.

Someone calling himself JavaBear also had some kind comments regarding this post.

Here's some snippets from one (of several) entries that I found interesting (dated "3-26-03", but there aren't any anchors in log, so you'll have to just search for it). This one happens to be mostly paraphrasing Paul Prescod.

"Paul Prescod gave the keynote [at a Python conference], and talked about what programming languages will look like in 100 years, to control flying cars and the like. He called COBOL and Java neanderthal languages that have no descendents on the evolutionary tree. How close is Language X to the evolutionary trunk? And what should we bet on? He noted that the Python community has a direct influence on the evolution of the language, and that it gets closer to the trunk at each successive iteration of Python.

[...]

"Cruft breed cruft. The more of the language you can write in itself, the better its chance for long-term survival.

[...]

"Specifications change while a program is being written. This is not only inevitable, but desireable. Some of the best programs are essays, in that the programmer didn't know what they were trying to write when they started.

[...]

"We could design a language now that could appeal to people in a hundred years, assuming unlimited resources. We could design the core language today. Optimistically there might only be a few languages in a hundred years. If the 100-year language were available today, would we want to use them? He says yes. Why not try writing this language now and see what it's like to program in?

"When driving, you don't match the hood of the car with the lines on the road, but point it at some distant landmark and correct it as you go. He says we should do the same things with a language."

There's lots of interesting stuff in there, I suspect I'll be a regular visitor.

Update [April 9, 2003]: It seems that this keynote was actually delivered by Paul Graham, and he has written it up as "The Hundred Year Language". Also see my followup post.

for(String str : myCollection) {
  System.out.println(str);
}

for(Iterator iter = c.iterator(); iter.hasNext(); ) {
  System.out.println((String)(iter.next()));
}

In addition, the document describes some changes to Java Iterators to support this.

This in and of itself is neat. Some URL hacking on that link leads to a list of files documenting some other interesting features, including:

• Autoboxing Support - automated conversion from primitives to primitive wrappers
• A Typesafe enum Facility - public enum Suit { clubs, diamonds, hearts, spades }
• Importing Static Members - where import java.util.Math.* will expose Math.abs(x) as simply abs(x).

It's strange that Java's "Community" Process doesn't expose this sort of detail more publicly. I'm unable to find a link to these documents on either of jcp.org or java.sun.com.

Googling on "JDK 1.5" yields some hits, including an entry from Henri's blog and several others, a brief article from java.sun.com, even an article on slashdot, all dated at least a month ago, so maybe this is old news to many of you.

(JSR 14 adding C++ template style generics to Java is another major change planned for JDK 1.5 it seems, but that one's been pretty widely known. I wonder if generics together with autoboxing will allow me to replace the org.apache.commons.collections.primitives package?)

full story...

(I had intended this to be a simple blog post, but it turned out to be too long for that. At least this gives me a chance to try out radio's "story" feature.)