Rodney Waldhoff: Tech

An SQL Annoyance

Mon, 08 Dec 2003 13:27:27 GMT

SQL isn't consistent under row/column transposition. For example:

select 3 + NULL

yields NULL. Yet,

create table TEMP ( NUM number );
insert into TEMP values ( 3 );
insert into TEMP values ( NULL );
select sum(NUM) from TEMP

yields 3 (since NULL valued rows are ignored by aggregate functions).

This inconsistency is all the more annoying since both:

select sum(NUM) from TEMP where NUM is not NULL

and

select sum(coalesce(NUM,0)) from TEMP

would yield the same result under an "aggregation of NULL is NULL" rule. Yet under the "aggregation function ignores NULL" rule, creating a single, efficient, cross-database query the yields NULL if there's a NULL row and the SUM otherwise is awkward at best.

Ruby Userland

Fri, 21 Nov 2003 18:58:26 GMT

As I previously mentioned, I've been toying with custom clients to the XmlStorageSystem XML-RPC protocol used by Radio Userland and several open source blog servers, with the ultimate goal of hosting a custom blog on the radio.weblogs.com host.

Over the past couple of evenings I've hacked out xmlStorageSystem.rb, a reasonably functional Ruby-based client to the XmlStorageSystem system protocol. It works like this:

To create a new instance of the client, use:

XmlStorageSystem.new(<usernumber>,<md5-hash-of-password>,<root-directory>)

For instance, I use:

XmlStorageSystem.new('122027','8c8034f5c9d68564e155c67a6d4e4612','/0122027/')

although that's not my actual password.

Actually, my local copy of xmlStorageSystem.rb has these value specified as the defaults, so I just user XmlStorageSystem.new, and I'll use that form in the rest of these examples. The constructor also accepts an number of arguments that should allow one to connect to the Python Community Server and others, although Radio is the only server I've tried.

To get a listing of the files currently stored on the server, use:

XmlStorageSystem.new.getMyDirectory

To download all those files to a local directory

XmlStorageSystem.new.backupMyDirectory 'backupdir'

To upload a file (or files) to the server, use:

XmlStorageSystem.new.saveMultipleFiles( 'local-base-dir', [ 'file1', 'file2', 'etc' ])

To delete a file (or files) from the server, use:

XmlStorageSystem.new.deleteMultipleFiles( [ 'file1', 'file2', 'etc' ])

Finally, the really handy one:

XmlStorageSystem.new.updateFromLocalDirectory 'localdir'

Which will compare the list of files in the local directory with those on the server, delete the ones that don't appear locally, and load/update the rest.

Since this is Ruby, it's easy to set up little shell scripts that invoke those commands in ways useful to your personal work style.

If one wanted to be clever, there is metadata available via XmlStorageSystem.getMyDirectory that would allow one to determine whether or not a file has changed since it was last uploaded, but I haven't gotten around to that yet.

I'm still pretty much a Ruby neophyte, so there's probably substantial room for improvement there. In particular, (1) there's no error handling present just yet and (2) the current implementation supports hackablity (changing the script itself) more than extensibility. Nevertheless, it's neat that a Ruby neophyte can write a basic XmlStorageSystem client in 150 lines of readable code.

On Programming Idioms

Fri, 14 Nov 2003 17:36:09 GMT

Two things I'm always keen to learn when picking up a new programming language are:

How does one organize large projects? In other words, how does one partition responsibilities and types across namespaces, modules and files?
What are the common idioms in the language?

I've been doing some string processing work with Ruby recently, and it's got me thinking about examples of the latter.

For example, in Java, the String class doesn't have a direct, boolean-valued method that will tell you whether or not a String contains another String, i.e., there's nothing like:

if(someString.contains(anotherString)) { ... }

Instead, most Java developers will write:

if(someString.indexOf(anotherString) != -1) { ... }

where String.indexOf(String) returns the index of the first occurrence of the argument String, or -1 if the given String isn't found. Most Java developers will immediately recognize that as the "String.contains" idiom, and won't miss a beat.

This idiom is so strong in the Java community that it's almost counter-productive to write a custom utility method:

public class StringUtils {
  public static boolean contains(String a, String b) {
    return (a.indexOf(b) != -1);
  }
}

since many developers who see

if(StringUtils.contains(someString,anotherString)) { ... }

are likely to wonder whether the StringUtils.contains method really does what it is implied--Is this equivalent to the String.indexOf idiom? Is that someString.contains(anotherString) or vice versa? How are null's handled? etc. Unless the developer is already comfortable and familiar with the StringUtils class being used, this code is probably less readable to an experienced developer than the "indexOf != -1" formulation.

(This is not to say that "String.indexOf(x) != -1" is actually preferable to "String.contains(x)", but rather that in the absence of String.contains, the idiom is more widely recognized than a custom utility method. Why Sun can't at some point introduce a String.contains method, say in JDK 1.5, isn't entirely clear to me.)

Now, in the Ruby scripting I've been doing recently, I keep needing to determine whether a String begins with a given prefix. In Java, that's the String.startsWith method, of course. The Ruby String class does not have a startsWith method, but one of the neat things about Ruby is that it's possible to literally add such a method the the String class, as follows:

class String
  def startsWith str
    return self[0...str.length] == str
  end
end

after which everything behaves exactly like the built-in String class contained that definition. For example, one can then write:

if someString.startsWith(anotherString) ...

or even

if "a literal string".startsWith(anotherString) ...

etc.

Of course, the implementation I used for String.startsWith (self[0...str.length] == str) is just one of several possible implementations. Regular expressions provide one way of implementing such a check. The Java-like indexOf function provides another (e.g., self.indexOf(str) == 0).

Since there is no built-in String.startsWith (or for that matter String.contains) in Ruby, I wonder if there is some common idiom that experienced Ruby developers find more readable than adding a custom method to String? If not for String.startsWith, how about String.contains?

The impotence of functional programming (by Sjoerd Visscher)

Wed, 12 Nov 2003 01:12:06 GMT

Sjoerd Visscher writes:

"Take for example the function zip (or zip3, zip4, exponential ugh-ness), a very popular function in f.e. Haskell. Firstly, if you have two lists, with values that go pairwise together, those values shouldn't have been apart in the first place. Secondly, what you end up with is a list of tuples. Values in a tuple always have more context than just being together. All this information is lost."

[Via Sjoerd Visscher's weblog]

Apache Jakarta Commons Primitives 1.0 Released

Thu, 06 Nov 2003 17:08:37 GMT

I've blogged a bit about the Primitives Component in the past, and I'm happy to note that commons-primitives now has an official 1.0 release (binaries) (source).

Currently Primitives provides Java-primitive based versions of various Collection interfaces, which is substantially smaller and faster than working with the Object-wrapper equivalents. For instance, an ArrayShortList requires 1/10th the space of an ArrayList of Shorts, and an ArrayLongList requires 2/5ths the space of an ArrayList of Longs. There's a substantial time savings as well, in that one isn't constantly "boxing" and "unboxing" the primitives and their Object equivalents when moving them in and out of the Collection. There are readability and other advantages as well.

(And, no, as I understand it, neither the auto-boxing nor the generics features of JDK 1.5 will obviate the need for this library. You could use auto-boxing to emulate the syntax of list.add(int), but the internal representation will still be Object based.)

We use the primitive collections fairly extensively within the Axion database project for things like indices and lists of row identifiers or data file offsets, indeed the code was originally developed there.

Custom Radio UserLand Clients?

Tue, 04 Nov 2003 21:24:32 GMT

Radio is a reasonably nifty weblog client, sporting an easy-to-use, web-based content management tool, a workable server-based news aggregator and a fairly robust weblog hosting infrastructure for USD$3.33 a month.

Things I find annoying about Radio include the fact that it won't run on a Unix-based OS (although there are reports of folks who've got the Windows version up and running under Wine on Linux), it's got a fairly large footprint, and it uses a relatively cumbersome and under-documented scripting framework.

All in all, the biggest appeal of Radio Userland to me is that I can get quick, easy and generally adequate weblog hosting for monthly cost less than a trip to Starbucks. If only I could continue to use the hosting service and replace the Windows-based client--I'd be happy to generate the weblog HTML via some homegrown software and simply upload the content via FTP or SCP.

Well, it's not quite as easy as FTP, but after poking around a bit, I've figured out how to "manually" update a Radio hosted site, or more generally, any of a number of servers that supports the xmlStorageSystem XML-RPC protocol.

Here's a relatively crude but functional example using HttpClient.

First, the basic framework for processing a request and it's response. This method takes a request "body" and returns the response. Typically both the request and the response will be XML-RPC documents.

String processRequest(String body) throws IOException, HttpException {
    PostMethod method = new PostMethod();
    method.setRequestHeader("Content-type","text/xml");
    method.setRequestBody(body);
    method.setPath("/RPC2");
    method.setRequestContentLength(body.length());
         
    HttpClient client = new HttpClient();        
         
    HostConfiguration host = new HostConfiguration();
    host.setHost("radio.xmlstoragesystem.com");
 
    client.executeMethod(host,method);
   
    return method.getResponseBodyAsString();
}

Typically we pass an xmlStorageSystem request to this method as the String parameter. Here's one simple example, a sort of "status" query:

<?xml version="1.0"?>
<methodCall>
  <methodName>xmlStorageSystem.getServerCapabilities</methodName>
  <params>
    <param><value><!-- your radio user number, e.g., mine is "122027" --></value></param>  
    <param><value><!-- the md5 hash of your radio password as ASCII bytes --></value></param>
  </params>
</methodCall>

To create or update a file (or files), we use saveMultipleFiles, for example:

<?xml version="1.0"?>
<methodCall>
  <methodName>xmlStorageSystem.saveMultipleFiles</methodName>
  <params>
    <param><value><!-- your radio user number, e.g., mine is "122027" --></value></param>  
    <param><value><!-- the md5 hash of your radio password as ASCII bytes --></value></param>
    <param>
      <value>
        <array>
          <data><value>/rpctest/test.html</value></data> <!-- location of file -->
        </array>
      </value>
    </param>
    <param>
      <value>
        <array>
          <data><value>Hello World!</value></data> <!-- contents of file -->
        </array>
      </value>
    </param>
  </params>
</methodCall>

To delete a file (or files), we use deleteMultipleFiles, for example:

<?xml version="1.0"?>
<methodCall>
  <methodName>xmlStorageSystem.saveMultipleFiles</methodName>
  <params>
    <param><value><!-- your radio user number, e.g., mine is "122027" --></value></param>  
    <param><value><!-- the md5 hash of your radio password as ASCII bytes --></value></param>
    <param>
      <value>
        <array>
          <data><value>/rpctest/test.html</value></data> <!-- location of file -->
        </array>
      </value>
    </param>
  </params>
</methodCall>

etc.

See Dave Winer's writeup for more procedures, details and examples.

Now that I know how to manipulate my radio.weblogs.com site without using the Radio client, I'm tempted to roll my own weblogging software, if only so I don't have to keep a Windows box around to support my blog. As I mentioned above, there are a number of servers that support the xmlStorageSystem protocol. Anyone know of any non-Radio clients for xmlStorageSystem? At minimum, something that syncs an arbitrary local directory with a remove xmlStorageSystem server should be easy enough to implement, which would allow anything that you can "publish" locally to be used as a client to a Community Server implementation.

The Jakarta Commons Problem in a Nutshell

Thu, 30 Oct 2003 15:26:23 GMT

Exhibit A, Exhibit B

Wiki + Blog = PIM

Wed, 22 Oct 2003 15:40:46 GMT

I'm one of those people who's so obsessed with organization that he's poorly organized. I'm constantly trying out new ways to manage the notes I scribble on various physical and electronic media, so much so that the frequency of change leads to greater disorganization.

I've tried a number of "information management" tools, including index cards, sticky notes, notebooks, a Franklin planner, Microsoft Outlook, and various outlining, mind-mapping and PIM programs, and found them all wanting in one way or another (maybe it's me). Paper based systems were too hard to search and too static in their organization. Lacking a PDA, electronic systems weren't portable enough. No matter what the system, I'd always spend more time trying to set up organizational categories than I ever saved by using them. In fact I had pretty much broken the habit, falling back to a collection of loosely organized text files that I could easily scp to and from whatever workstation I was currently using.

About a year ago, I got a Sharp Zaurus, and since that addresses one of my recurring PIM complaints--having something that's always readily available, my hopes for finding the perfect PIM system were renewed. I've played around with a number of the PIM systems available. Using the bundled Address Book/Calendar/To Do List/Email Client was too jumbled. The "outliner" IQNotes was promising, but a single hierarchy is inflexible and using it to create dated journal entries requires too much manual effort. In the end, I came back to a loosely organized collection of text files, this time stored on the Zaurus.

Back in my first post, I mentioned that one of the reasons I was interested in blogging is that I had become interested in journaling in general. These two concepts--the search for the perfect organizational system and interest in journaling--interrelate.

As I'm sure I'm not the first to suggest, it seems that the ideal PIM system presents at least two perspectives on the information to be managed:

1) A time based view, answering such questions as "what have I been working with recently?" or "what else was I working on around that time?"

2) A "concept network" view, answering such questions as "what's related to this topic?" or "what else do I know about this subject?"

Clearly, I presume, blog-like software is an excellent starting point for the former, and wiki-like software is an excellent starting point for the former. Fortunately, a number of projects have begun to explore combinations of the two, either by implementing a blog-as-wiki (like SnipSnap does) or by adding blogging features to a wiki engine (like JspWiki and no doubt others do).

Having a fair amount of Java experience (indeed, I've built a little wiki or two in Java), my first thought was to get a Java based wiki up and running on my Zaurus. JspWiki looks nice, and like most Java-based wikis is Servlet/JSP based, so I endeavored to get a WAR-friendly Java Servlet engine up and running on my Zaurus. Having tried both Tomcat and Jetty under both the Jeode evm and Personal Java cvm, I came close, but fundamentally failed to do this. (I note that my SL-5500 had just barely enough disk space or memory to do this as well.)

I briefly considered setting up a stand-alone (i.e., non-Servlet/JSP) Java wiki engine on the Zaurus, but without a moderate amount of development that seems like an awkward environment for templating and modification, and non-servlet Java web development feels like dead-end development. (Its only a matter of time before a reasonably micro servlet engine is available.) That's when I stumbled across PhlIp's Mini Ruby Wiki, which implements a featureful, stand-alone wiki server in around 1000 lines of Ruby code.

(For the record, one could readily build a similarly featured wiki in Java, and probably a number of languages, in roughly the same number of lines, it might not be as much fun.)

It still took me a while to get a Ruby interpreter up and running on the Zaurus (after trying a few different builds, I finally got the one from Vincent Fiack working), but the resulting binaries are smaller in terms of disk and memory footprint than the JVMs I was playing with. Perhaps more importantly, as a purely interpreted language, the engine itself is easier to tweak. And this gives me a reasonably good excuse to hack out some Ruby code. I've already added/modified a few minor features, adding some simple blog-like features to the calendar features that already existed. If I come up with something I'm not to embarrassed to share, I'll post it here or submit it to PhlIp's RubyForge project.

My only problem now is resisting the temptation to continually tweak the wiki engine code.

A Timely Bug

Fri, 17 Oct 2003 22:23:12 GMT

I'm a sucker for quirky bugs. Recently, I've been hacking around with PhlIp's excellent Mini Ruby Wiki (more on that later), and discovered one such bug.

There's a block of code in MiniRubyWiki that renders a small calendar in HTML, not unlike your typical blog calendar widget. Here's a small subset (the actual code is more interesting that this) which is largely although not completely unchanged from the original miniWiki.rb source. See if you can spot the bug.

  # here's a constant, declared elsewhere in the program
  SecondsInDay = 60 * 60 * 24
 
  # ...
 
  # In this loop we're rendering a four-week calendar,
  # from the beginning of last week until the end of the week after next.
  # "beginningOfLastWeek" is a Time indicating the Sunday of last week
 
  today = Time.now()
  (0...28).each do |day|
    dayOnCalendar = beginningOfLastWeek + day * SecondsInDay
    # break the row every seven days
    calendar += '</tr><tr>' if day.divmod(7)[1] == 0
    calendar += '<td'
    # highlight today
    if today.mday() == dayOnCalendar.mday() then
      calendar += ' bgcolor="#88ff88"'
    end
    calendar += '>' 
    # show the month for the first day in the calendar or in the month
    calendar += dayOnCalendar.strftime('%b') + ' ' if dayOnCalendar.mday == 1 or day == 0
    calendar += dayOnCalendar.mday.to_s() 
 
    # ...snip some stuff displayed within the calendar...
 
    calendar += '</tr>'
  end

I've been running this code on my Zaurus, sometimes launching it from the EmbeddedKonsole on the Zaurus itself, and sometimes from an ssh terminal running on a remote machine. Eventually I noticed something funky about this. When I launch the server from the ssh terminal, 1 November 2003 is listed, correctly, as a Saturday. When I launch the server from a terminal on the Zaurus itself, 1 November isn't listed at all (the last day on the calendar is 31 October 2003, listed, incorrectly, as a Saturday). In fact, when I launch the server from the Zaurus, Sunday, 26 October 2003 is listed twice.

Adding a "puts curDay.to_s" line inside that loop makes the problem obvious. This output like the following:

Sun Oct 19 00:00:00 CDT 2003
Mon Oct 20 00:00:00 CDT 2003
Tue Oct 21 00:00:00 CDT 2003
Wed Oct 22 00:00:00 CDT 2003
Thu Oct 23 00:00:00 CDT 2003
Fri Oct 24 00:00:00 CDT 2003
Sat Oct 25 00:00:00 CDT 2003
Sun Oct 26 00:00:00 CDT 2003
Sun Oct 26 23:00:00 CST 2003
Mon Oct 27 23:00:00 CST 2003
Tue Oct 28 23:00:00 CST 2003
Wed Oct 29 23:00:00 CST 2003
Thu Oct 30 23:00:00 CST 2003
Fri Oct 31 23:00:00 CST 2003

That's right, sometimes 24 * 60 * 60 is the wrong value for SecondsInDay. Due to the switch from CDT to CST, 26 October 2003 has 25 hours, so adding SecondsInDay to the midnight time isn't sufficient to advance the day.

For whatever reason, when launched via remote ssh terminal, the interpreter doesn't seem to be aware of the locale (it shows the times as UTC), but when launched from EmbeddedKonsole, Time.now() is aware of the locale and manages the switch from CDT to CST automatically, leading to the bug.

Cute, no?

Kudos to Jakarta Commons FileUpload

Mon, 15 Sep 2003 15:37:40 GMT

Recently Ted and Jason blogged on an old complaint: open source projects aren't well documented.

In a curious coincidence, at the same time I had an very positive experience with open source documentation. Being a Jakarta Commons committer, and having done my share of multipart/form-data parsing in the past, I was vaguely familiar with Commons FileUpload but hadn't had much cause to look into it until recently. Not only does it feature an easy-to-use API that works like a charm, the Using FileUpload page tells one everything he needs to know to get up and running in minutes.

Granted FileUpload addresses a pretty limited domain, and not all the documentation is as complete, but I can't remember the last time I picked up a Java library and started using it without even cracking open the JavaDocs.

Everything new is old again

Thu, 28 Aug 2003 01:26:57 GMT

Recently, Daniel, Ted, Andy and James all point to a post by Tom Lord over on the arch-users mailing list.

Tom's comments are made in the context of Red Hat's agreement to distribute Sun's JVM as part of their "Enterprise Linux" offering, a story that (although I regularly program in Java and I'm writing this post on Red Hat box) quite frankly I haven't been following at all. Much of what is interesting about Tom's comments may be a function of reading them outside of this context.

I find this a striking post, among other reasons, because it raises several distinct responses from me:

1. What are we building?

Tom suggests:

There has been a sort of tension in the commercial free operating system world:

(a) Are we building a free alternative to proprietary software?

or

(b) Are we building a commodity, $0-price OS component to lower the cost of proprietary applications?

For my part, this is a false dichotomy. In my open source efforts I'm not trying to do either of these, but rather to simply build something useful. Whether this something is an alternative or adjunct to proprietary software is incidental at best. For that matter, whether this something is truly innovative or just a more useful variation of a component that already exists isn't terribly significant either (although the utility of copying something that's already readily available is limited of course).

I wonder if this a symptomatic of differing perspectives between the BSD-style and GPL-style camps of open source development.

2. How do we build it?

Tom writes:

If the goal is (a), then we need an architect. We need to come up with an inexpensive-to-develop architecture that, nevertheless, contains viable solutions for the needs of our markets.
If the goal is (b), then we need an anti-architect. We need to come up with impossibly-expensive-to-fully-develop clone projects of proprietary software to draw off the energy of volunteer contributors who might otherwise undermine the value of the proprietary applications we expect to drive revenues for our distro.

Here option B is obviously a straw-man, so of course I'll follow option A. But while I agree we need "an inexpensive-to-develop architecture", and the word "architect" appears in my job title, I'll suggest a evolutionary approach is the best way to get there. Inexpensive-to-develop systems of any interesting size, let alone federations of such systems, are rarely "architected" in a traditional sense. What we need is an environment where diverse ideas are allowed to compete, cooperate and breed.

3. How hard is a Java implementation, really?

Running throughout Tom's post is the notion that a reasonably complete Java environment is simply too complicated to implement in an open source fashion:

For example, under (a), we would probably have to admit that trying to clone all the Java libraries _and_ build a competitive Java implementation is too expensive a course of action. While we might be perfectly happy to ship a low-end, incomplete implementation to help the low-end of the market, in the long run, we'd want to look for a more clever solution: something that can compete with Java and Java libraries on functionality, but that is cheaper to build in the first place (and cheaper and more effective to apply, of course).

[...]

[Y]ou can also make things expensive to develop by structuring them as an object oriented framework that you then spend zillions on filling out with subclasses, or by making really hard components (like finely tuned JIT compilers and garbage collectors) critical to implementations.

[...]

Some architectures, such as the Java environments and the view-tree/component-based GUI frameworks, are ideal for large proprietary software companies with large command-and-control armies of developers and QA practitioners. Those architectures are a terrible fit for the loose confederation of generally underresourced developers in the free and open source software world.

Putting aside questions of productivity and effectiveness, and the "enterprise" libraries for the moment, how hard is the core Java environment to implement, really? More difficult than a Lisp implementation, for example, but I suspect it's not substantially more difficult to implement than, say, a C implementation, and probably on par with something like Ruby.

4. Where are the alternative Java implementations then?

Well, there are several actually, although relatively few complete or robust ones. Why? Perhaps one compelling reason is that proprietary yet free (as in beer) Java runtime environments are readily available for most platforms.

Besides, who really wants another Java platform anyway...

5. "Yep, time to stop copying"

Like Daniel and Ted, I find the general call for innovation dead on:

For a long time, the right strategy for GNU was to build a basic unix replacement differentiated primarily by licensing. [...]
Well, that part's done and the strategy won.
[...]
If the goal is still "(a) build a free alternative to proprietary software", then a new strategy is called for: competition on software architecture, not just licensing.

Here Tom and I are in violent agreement. If I were building an open source compiler and/or language-platform, I'd certainly think twice about doing it on Sun's plantation--not because it's too hard; in part (as before) because the utility of copying something that's already readily available is limited; but mostly because I think you could construct more useful environments.

It's worth applying this, independently, to the "enterprise libraries" I set aside earlier. Are J2EE implementations like Apache Geronimo or JBoss building something "useful" or a "$0-price OS component to lower the cost of proprietary applications"? From my perspective, I've found some pieces of the J2EE suite to be quite useful, and others seem to fit Tom's "proprietary vendor" strategy.

6. Except when it isn't

It's interesting to observe how quickly the arch-users thread evolves from "we need a new architecture" to "we need a Lisp platform". (It's doubly interesting to note how often that seems to be the case.) And perhaps that really is what we need. But there's a big difference between "time to stop copying" and "time to stop copying Java".

(Actually, although I think one could do much better than to copy Java, you could also do much worse. It would be naive to think that Java's success is purely coincidental or purely the result of marketing muscle. They must be doing something right.)

More generally, it would be naive to think that what we need is innovation for innovation's sake, and I think deciding we need to resurrect a 40 year old platform is evidence of this fact.

7. What are we building, revisited.

Tom writes:

Nowadays, the proprietary competition is about databases, and productivity apps, and browsers, and middleware layers. The software we're competing against is not like unix: it isn't simple; it wasn't built by a small number of people; it's a moving target. It isn't a tractable project to clone this proprietary software under different licensing.

This point is puzzling. Certainly we don't mean to assert that it is impossible to successfully build open source databases, web browsers, or middleware servers, do we? Do we assert it is a bad idea to do so?

While I certainly think we should look for innovative ways to solve the sorts of problems these projects do, it would be a mistake to believe that the existing approaches don't offer something of value simply because they have strong proprietary implementations as well--just as it would have been a mistake for the GNU project to reject a pipe-and-filter architecture simply because a strong implementation was once controlled by Bell Labs. I don't think it is tractable to create a wholly new software paradigm--one that doesn't contain variations of n-tier, database and web technologies--out of thin air. We need new ideas, but we need old ones too.

Adaptive vs. Predictive Planning

Mon, 18 Aug 2003 17:13:05 GMT

At my day job, we develop what seems to be a wide variety of systems: several subscription based web sites, a suite of desktop applications, an e-commerce system for selling both subscriptions and physical products, and various financial, administrative, content management, editorial and reporting systems for supporting them. We've been developing most of those systems under a locally adapted form of XP for about three years now.

For a lot of what we do an agile process works well. We're often making "small" modifications to a running (production) system, so a process that allows the customer to say "here's a set of changes, implement and deploy them in the next couple of weeks" fits our needs rather well. Even the least imaginative customer is typically able to break down large changes into an incremental series of small ones, and is often happy to see the intermediate steps in a running (although not always production) system.

Allow me to call this approach "adaptive" planning. (I don't think I'm the first to do so.) An interesting feature, perhaps the defining feature, of this adaptive approach is that customer is often defining new requirements as we're implementing the old ones, so that we're not always exactly sure where we're going when we start. It's sort of like a hill climbing approach: we take a few steps in what seems to be the right direction and then reevaluate. Our plan adapts to changing business requirements and generally doesn't try to look very far ahead in any detail.

It doesn't take a lot of insight to see that an adaptive approach is well suited to what might be called "product" as opposed to "project" management. If one conceives of the system as long-lived, constantly improving entity, it's easier to be comfortable with an iterative development process. No one can really foresee where the system is going to end up anyway, so the urge to ask "when is it going to be done?" is lessened.

On the other hand, there are times when the customer wants, or at least thinks he wants, a more predictive planning process: "Here's a large set of requirements, when are they going to be met?" This question is important because the answer will often drive business deals, strategic planning, and revenue projections. Depending upon the time sensitivity of the opportunity, the answer may determine whether the project is worth initiating at all.

Frankly, I don't think our process is very good at answering this sort of question. In practice, the way we approach such a question is quite similar to most "traditional" methodologies: we'll collect coarse grained requirements and prepare coarse grained estimates for them. This process is often time consuming--the customer team will spend weeks struggling to define and communicate a "base" set of requirements for estimation purposes. The result is often unsatisfactory--poorly understood and poorly articulated requirements lead to poor estimates.

One solution is to somehow dissuade the customer from believing that he needs detailed planning or long term predictions. Sometimes it really is a distinction without a difference--we're going to do the work anyway, so knowing precisely what features make it into the version 1.0 release or precisely when this release will occur are details that can be determined later. Most projects end with some negotiation of scope and schedule anyway. The trouble is, sometimes accurate predictions really do matter.

Another solution is to accept the limitations of this predictive approach--we don't really know how long it will take to complete the project, or what "complete" really means in the first place, so the best we can do is take an educated guess and know that the margin of error (and therefore the risk) is high. True as it may be, this answer is unsatisfactory at best, and unacceptable at worst. The fact that traditional planning approaches don't fare much better is little consolation.

We've been through this "predictive" planning exercise perhaps four times in the past three years, and have found it to be successful, but only moderately so. Increasingly I am beginning to believe that the adaptive process may be a more effective way of arriving at accurate predictions anyway. Through a lack of information, insight, or political capital, I've never seen it played this way, but I'm beginning to think this is the right approach:

Rather than spending a couple of weeks (or more) trying to determine a rough but "complete" estimate for the total cost of the project, spend that time developing an implementation of some of the base features (call it a "spike solution" or prototype if that helps, but it's really neither of those things). The requirements will certainly be incomplete, maybe even wrong. The implementation is likely to be wrong too, or will be when the requirements are better understood. I think that that may not matter.

Part of what leads me to this conclusion is that this is what we really do anyway. Although the thought is "let's not go down this road until we're sure that's what we want to do", in practice what we'll struggle to determine what the "total cost" of the project is going to be until it becomes clear that if we don't initiate the development soon, we won't be able to hit the requisite dates anyway. We may only be 60% sure that the project is worth pursuing, but the schedule eventually makes the decision for us. Indeed, this is how we first adopted an XP practice: as a way out of the "analysis paralysis" threatening a major product initiative.

(This last-minute adaptive approach is expensive in at least a couple of ways. Firstly, by the time a deadline is looming, we no longer have the luxury of slowly ramping up size the development team, we need to throw all our developers at the system immediately. This is a chaotic way of doing green-field development. Secondly, when we follow this approach, it's rarely the most important features we attack first, it's the ones that are easiest to define.)

I think this "just get started" approach may bring several benefits over a predict-then-implement one. Having a working system, however rudimentary...

makes the total cost estimate more accurate simply by virtue of having some of the requirements already implemented. We don't need to guess what those will cost--they're already done.
makes the development team more confident about their ability to address the remaining requirements.
makes the development team better able to predict what's going to be tricky and what isn't. We have some measure of the team's velocity when working on the system in question. As a result, estimates should be more accurate.
provides the customer team with a reference point to work from. Rather than saying "here's what the final system will look like", they can be begin to express requirements as "here's how I want to you change the existing system."
may give the customer team enough confidence in the development team's ability to deliver the end product to follow an iterative/evolutionary approach. Once it's clear that we can deliver something by the deadline, it may be less important precisely what that something is.

Of course, the customer team can in parallel continue to define and refine the requirements and seek a "total cost" estimate, but at least for these first few iterations, we don't need to worry too much about the total scope of the project

There is some risk here, namely that after a couple of weeks of development it becomes clear that this is not a project worth pursuing and hence the development effort in those first few weeks is wasted. I can't imagine that this cost is significantly greater than the cost of the predictive analysis, but for better or worse in my environment there is a general impression that development resources are precious and hence need to be conserved--it's better to waste "analysis" time than "development" time. Overcoming this impression may be the key to selling this approach.

Looking over what I've written here, it looks like I'm describing a straight-forward XP process, and perhaps I am. In some ways, the distinction may be in the reasons for the adaptive approach. In our typical agile project, we sell the just-in-time requirements definition process as a way to get development moving while the business is still working out requirements. When predictive questions are asked, the business often believes that the requirements are already defined (although in practice they're often not as well defined as the customer may believe), only the technical cost needs to be determined. In this case the iterative development cycle may be more valuable to the technical team than to the business.

It may be interesting to explore how to use an adaptive planning process to answer predictive planning questions.

Commons Logging was my fault

Fri, 15 Aug 2003 18:59:08 GMT

I'll come right out and admit it: commons-logging, at least in its initial form, was my fault, though probably not mine alone.

Back in 2001 I did a fair bit of work refactoring and debugging commons-httpclient. As part of that effort, I replaced a custom setDebug()/System.out.println based logging system with log4j. I did this for several reasons but all of them come down to the fact that a fully-fledged logging system is often quite useful. In fact, that debugging effort probably wouldn't have happened without it.

This change turned out to be controversial, and a runtime dependency upon log4j was in fact vetoed. After an enormous amount of discussion, not all of it pretty, a compromise was finally agreed to. This compromise basically said (a) use a thin wrapper around either log4j or the System.out logging initially implemented and (b) allow the user to specify whether to use log4j or System.out.println based upon a method call (inversion-of-control style I guess) or an external property. I did the first implementation of this. It was later suggested, probably reasonably so, that this "log wrapping" package be extracted from httpclient for use in similar circumstances. In a more robust or complicated form (depending upon your perspective) this eventually became commons-logging.

For all its warts, this brief history of Commons Logging shows the "bazaar-style" of open source development working as it should: the operative word here being "compromise".

Personally, I don't believe it to be especially important that Jakarta Commons committers be able to express their creativity by the selection of logging frameworks. I'd have been happy simply using log4j, and I think Ceki's work with the "light" log4j-ME would have been sufficient to address most of the technical concerns raised. Yet others disagree, and I'll respect that position. Certainly respect for that position has encouraged advocates of alternative logging implementations to participate more fully in Jakarta Commons.

That said, I think Hani and Glen miss the point entirely. The purpose of Commons Logging is not to isolate your code from changes in the underlying logging framework. (That's certainly easy enough to do on your own, and not really worth doing in the first place given the ease of switching from one logging framework to another.) The purpose of Commons Logging is not to somehow be more useful than actual logging frameworks by being more general. The purpose of Commons Logging is not to somehow take the logging world by storm. In fact, there are very limited circumstances in which Commons Logging is useful. If you're building a stand-alone application, don't use commons-logging. If you're building an application server, don't use commons-logging. If you're building a moderately large framework, don't use commons-logging. If however, like the Jakarta Commons project, you're building a tiny little component that you intend for other developers to embed in their applications and frameworks, and you believe that logging information might be useful to those clients, and you can't be sure what logging framework they're going to want to use, then commons-logging might be useful to you.

On a slightly unrelated note, I think that Hani's post fails to adequately explain how the problems caused by trying to integrate systems that use incompatible versions of the same component are unique to Commons Logging. If you want a discussion of the technical issues with Commons Logging, which are quite genuine, there are much better analyses available.

A chemist, a physicist and a computer scientist were traveling in a car...

Thu, 14 Aug 2003 13:17:35 GMT

I have to give a product demo this morning. Although I hope I don't need to use it today, I'm reminded of a little joke I used to tell back in my consulting days to keep the customer occupied when a problem forced me to do an unexpected restart, reboot or reinstall in the midst of a demonstration:

A chemist, a physicist and a computer scientist were traveling in a car when the engine suddenly puttered out and they were forced to pull over to the side of the road.

"We must be out of fuel", the chemist said, "but I just saw a gas station. I'll walk back and pick some up."

"No, no", said the physicist, "that clearly sounded like a mechanical problem. Let me pop the hood and take a look."

"Wait!", said the computer scientist. "I have an idea. Let's all get out of the car . . . and then get back in."

Quickie Radio Tidbits

Mon, 11 Aug 2003 17:26:32 GMT

Radio now supports TrackBack. See TrackBack for Radio for details, but the gist of it is this: (a) enable TrackBack through your TrackBack preferences page then (b) invoke the <%trackbackLink%> macro somewhere within your item template.
As you can see, I've got trackback links turned on in this blog now. I've also added direct (non-JavaScript) links to my comments and trackback pages. I did this by adding <%radio.weblog.getCommentLink(<%itemNum%>, adrblog)%> and <%radio.weblog.getTrackbackLink(<%itemNum%>, adrblog)%> calls to my item template.
I've finally figured out where many of the built-in Radio macros are stored. From the Radio application, go to Window:Radio.Root:system:verbs:radio:weblog. The system/verbs tree has a number of nifty macros, including radio.weblog.getUrl, which generates the absolute URL of your Radio homepage. This is the answer to a question I had previously asked.
Another pair of interesting macros I discovered there are radio.macros.previousDayLink and radio.macros.nextDayLink, which should generate day-to-day links in your archive pages. I've got this set up in my archives now, but there seem to be minor problems here and there. For example, the previous day link from Wednesday, 16 July 2003 goes to Monday, 14 July 2003, skipping over the post on Tuesday, 15 July 2003.

When I get a chance, I'll type these up for my Radio Tidbits page.

PS: I'd love it if someone could point me to a definitive reference to Radio's scripting language.

In Defense of XML

Fri, 08 Aug 2003 17:38:26 GMT

Recently I've read a number of comments (some old and some new) that take issue with the use of XML for one purpose or another (in the above examples, Ant and Jelly scripts, respectively). In fact, it seems all the cool kids are XML detractors these days, although the coolness may have peaked a few months ago, when even Tim Bray was admitting that XML may not be as nifty as first thought.

Now, I'll certainly admit that XML has its drawbacks. Is it overly verbose? Often. An equivalent s-expression syntax would be more concise and (for Lisp developers at least) more useful. Is it hard to read? At times. It's certainly better suited for documents where the ratio of text to tags is high. Is XML often used for developer convenience at the expense of user convenience? Is there information that has no business being in an XML format, but that developers or vendors insist on making XML anyway? Yes, yes, of course.

(On a related note, if "executable XML" is such a bad idea, how does one explain the longevity of Lisp?)

Despite these limitations, there is some value in selecting XML over, say (as Andy Hunt suggested for Ant) some arbitrary context-free grammar.

Part of this value is the ease with which a developer can drop in an XML parser, but that's only an indirect source of value. Part of this value is the comfort that users familiar with popular SGML applications (read "HTML") have with the angle bracket notation, but that may only explain the quick adoption.

The real value of XML is in the tool chain.

Suppose Ant had been based upon some custom, non-XML grammar. What would we lose?

Well for one, nearly every programmer's editor has a syntax coloring, well-formedness-checking mode for XML (certainly emacs and vi, or at least vim, and nearly anything that calls itself an IDE). Moreover, many editors support DTD or Schema validation as well, perhaps even tag and attribute completion. Using an XML format means a host of editors can handle Ant scripts smartly. The same would probably not have been true, at least initially, with some arbitrary grammar.

How does this come about? Well in part because XML is a popular format, but also because of the ease with which a developer can drop in an XML parser and other libraries.

For another advantage, consider the plethora of XML-based or XML-extending specifications. Many commentators have looked at that dense diagram and scoffed: not everything is well suited for an XML representation. Yet few would deny that at least some of those technologies do something truly useful.

Similarly, consider the number of tools, libraries and technologies that implement those specifications or provide other XML utilities. Want a pretty printer? There's a tool for that (indeed you're probably using one right now). Want an API for processing arbitrary XML? There's several, in nearly any language you can name. Want portable validation? Want to combine dialects? Want to translate one schema to another? Want simple macro support? Want to embed or link sub-documents? Want to generate hyperlinked documentation for a script or it's syntax? There's a tool for that too.

Sometimes it's sufficient to be adequate and popular, when that means a strong tool chain comes along for the ride. Sometimes worse really is better, or at least good enough.

Clover Plug-in for Eclipse (and NetBeans)

Fri, 08 Aug 2003 12:45:59 GMT

Matt Quail writes "We have just released (beta) an Eclipse plugin and a NetBeans plugin for Clover, allowing you to instrument your code and view coverage results all from with the those Java IDEs."

Haven't tried it yet, but the screenshots look like exactly what I was hoping for. Sweet.

The Silent Majority Pays for Open Source

Wed, 30 Jul 2003 18:18:19 GMT

When you think of a writer, you probably imagine a book author, a journalist, an essayist, or even a script writer of some sort, but it's generally someone who makes a living selling their words. But the majority of text isn't produced for direct or even indirect sale: it's used to remind your spouse to pick up a gallon of milk at the store, or to thank Aunt Rita for that lovely sweater, or to point your friend to that really funny website you saw this morning. Even the majority of text someone gets paid to write isn't written for sale: it's in business correspondence, or restaurant menus, or on mortgage applications, or bus schedules, or in instruction manuals, or on the back of cereal boxes. Practically everywhere you look you can find the written word. Someone was paid to write much of it, and yet hardly ever are you buying the words themselves.

Most of the time text isn't the end product, it's a tool for communication. Lots of folks are paid to write something, very few of them view the text itself as the product. It's a means to an end.

Yesterday Alan Williamson asked "Who pays for open source?" and answered his own question with "The great belief, as sung by pretty much all companies involved in open source, is 'we charge for support'."

Andy Oliver suggests this is but one open source business model, but goes on to define four "forms" that all pretty much come down to "we charge for support".

I think Steven Berkowitz, in a comment to Alan's post, gets much closer to the truth: "Open source projects that somehow make money for someone, be it through support, consulting, etc., are the exceptions." I'll take that comment one step further: Modulo outsourcing, software projects that directly make money for someone, be it through sales, support, etc., are the exceptions.

I'm going to make an assumption here, but I think it's an assumption I can safely make. The majority of software developers aren't selling software, or even software support and services.

Living as I do, in Chicago, where I'm told both the per capita and absolute number of software developers is higher than the Silicon Valley, I think this observation is a bit more obvious. I know a lot of software developers, but practically none of them build software for direct sale to consumers or businesses. When a "software development" firm is hired, it's often for custom (or at least customized) development. Even when we sell shrink-wrapped software, as my company does, it's not the software that customers are really buying.

For most companies and for most developers, software isn't a business, it's a tool for getting the real work done.

In this scenario, it's easy to see why a company might use open source software: just find me the best tool for the job. It's also easy to see why a company might allow its IT staff (internal or outsourced) to contribute to open source development. Critical as it might be, and as disastrous as it can be when it fails, few companies rely upon software for their competitive advantage. I don't care if my competitors use the same web or database server that I do, or for that matter the same XML parser, caching engine, unit testing framework, or database connection pool that I do. Indeed there is some advantage to me if the non-proprietary parts of my infrastructure become commodities.

While the public face of open source software might be the folks that are trying to build a business around it, I suspect that's because, well, they're selling something. Who pays for open source development? My guess is that the answer is the same as it is for most software development: the folks who are trying to get something else done.

Wanted: GUI Wrapper for Text-Based Java Applications

Wed, 30 Jul 2003 00:10:38 GMT

I've got a simple console-based (i.e., text interface) Java application that I'd like to expose via Java Web Start or as an Applet. I can't, however, because System.in and System.out aren't attached to anything useful, so my application runs invisibly and can't collect any input.

What I need is a simple wrapper application that creates a basic text i/o frame that looks like stdin/stdout to my application's main method. (Or, failing that, to a custom main(String[] args, InputStream stdin, PrintStream stdout, PrintStream stderr) method.) More elaborate interfaces, such as color coded output, distinct stdout and stderr frames, spool to file, etc., are easily imagined.

I believe such an application would have fairly broad utility. A web search finds quite a few elaborate telnet/ssh oriented terminal emulators, which one might be able to pare down to what I'm looking for, but it seems like someone would have already tackled this problem. Lazy web to the rescue?

A is for Axion ... Z is for Zaurus

Thu, 24 Jul 2003 18:11:39 GMT

I've got Axion up and running on my Zaurus.

If you'd like to do the same, here's how:

Obtain a build of the Axion HEAD (1.0M3-dev). See the instructions on building Axion for details.
Obtain binaries of Axion's runtime dependencies, namely Jakarta Commons Collections (currently a nightly build is required) and Jakarta Commons Logging (you'll want release 1.0.3, earlier versions don't work on the Jeode VM).

If you want to use the LIKE operator, you'll also need Jakarta Regexp.

If you want to use the BASE64ENCODE or BASE64DECODE functions, you'll also need Jakarta Commons Codec.
Ensure that you have the java.util Collections and java.sql JDBC packages available on your platform. Since the Insignia Jeode VM installed on the Zaurus by default is JDK 1.1 based, I had to copy over the JDK Collections manually. I had thought the Collections package was available as a standalone JAR to drop in to JDK 1.1 environments but as I was unable to find it. I just copied over the rt.jar from a JDK 1.3 installation, but I think the java.util package would suffice.
Obtain a copy of axiondb.properties. You can pull this out of the Axion JAR or from CVS. Normally Axion loads this file out of the JAR automatically, but the Jeode VM seems to always return null for getClassLoader so we had to add a mechanism for specifying the configuration as an external file.

Now simply copy those files over to the Zaurus, add the JARs to your classpath, and you are ready to go. For example to run the Axion console I use:

evm 
 -Dorg.axiondb.engine.BaseDatabase.properties=axiondb.properties 
 -classpath axion-1.0-M3-dev.jar:
            commons-collections.jar:
            commons-logging.jar:
            commons-codec.jar:
            regexp.jar:
            rt.jar 
 org.axiondb.tools.Console $1 $2

Similar steps should get Axion up and running on other micro or J2ME platforms or for that matter, other JDK 1.1 environments. If the console is any indication, Axion seems to run rather well on the Zaurus (even running off a compact flash card rather than the RAM disk).

If folks were interested, it wouldn't be difficult to create an IPK installer for Axion, although my interest here, and others as well I imagine, is in using Axion within other apps on the Zaurus, rather than playing with Axion via the console.

If you're curious, I did have to make a few minor changes to Axion to get it run on the Jeode/JDK 1.1 VM. Here's a brief list of what I had to change:

The Jeode VM always seems to return null for getClassLoader (rather than throwing a security exception), so I had to add checks for null, and provide an alternative mechanism for loading the properties file.
We had used File.createNewFile in several places, which is a JDK 1.2 method. These calls turned out to be unnecessary anyway, so I simply removed them.
Although it's easy to add java.util.Comparable and java.util.Comparator to the classpath, none of the core objects (Number, String, etc.) actually implement Comparable in JDK 1.1, so I had to add custom Comparators replacing ComparableComparator for those DataTypes.

For reasons I don't understand, in several places where we had an hierarchy like this:

interface Foo {
  void someMethod();
}
 
abstract class AbstractBar implements Foo {
  void anotherMethod() {
    doSomething();
  }
}
 
class Bar extends AbstractBar {
  void someMethod() {
    doSomethingElse();
  }
}

I had to declare the interface methods in the abstract class:

abstract class AbstractBar implements Foo {
  abstract void someMethod();
 
  void anotherMethod() {
    doSomething();
  }
}

to make various AbstractMethodErrors go away.

These changes have already been checked into the HEAD version of Axion, and should be part of the Milestone 3 release.

There are few Axion-based apps I've been thinking of tinkering with on my Zaurus, some for personal use, others for my day job. That may shake out a few issues I haven't yet encountered, but so far I've got feature I've tried working without too much trouble.

The Great Wall of China

Wed, 23 Jul 2003 18:25:07 GMT

Although I'm neither old enough nor wise enough to pull it off effectively, I'm a fan of management by narrative: using a brief story or parable to illustrate a point. Often I find it is sufficient to present the narrative without explicitly connecting the dots--I'll just tell the story and then launch into what I was otherwise going to say. (As always, you need to understand your audience to use this approach effectively. I've noticed developers often respond well to leading with the narrative. Suits, whether management, business or marketing folks, often respond best to hearing the explanation and then the metaphor.)

Yesterday, in the context of the early stages of a fairly large scale development project, I had the opportunity to tell the following.

When the emperor decided to build the 10,000 Li Wall, he didn't amass an army of men to start at one end and build to the other. Nor did they start at both ends and build toward the middle, the way one might dig a long tunnel. Rather, they assembled small teams of men and spread them far and wide. Each crew was responsible for constructing a small section of the wall, separated by many li from the neighboring sections.
Like all ancient construction, the work was difficult, and by the time the section was complete the crew was greatly discouraged. They had worked so hard, and seemed to have little to show for it but a single, isolated tower.
Yet as the crew made the long journey back to their homes, they would come across other sections of the wall--some complete, others being busily attended to by other work crews. They came to discover that the wall stretched, section by section, from the Yellow Sea deep into the western desert. They arrived home in awe of the enormity of what had already been accomplished, and more inspired than ever to complete the construction.

I have no idea if this story is true (never let the facts stand in the way of a good story), my source is a half-remembered parable written by Franz Kafka. In my limited understanding of Chinese history, the story is a half-truth: the Great Wall was assembled a bit more organically out of a series of smaller fortifications.

I was a little hesitant to tell this story as I think the metaphor can be easily misinterpreted. I'm not suggesting that large scale systems be built by teams with only local knowledge of the overall system. Nor am I suggesting that if "isolated towers" aren't useful to you (or your customer) that this approach is a good one. (This latter case is an example of where the reality makes a better metaphor than the story. In practice Shih huang-ti connected a number of existing, independently useful fortifications to form the Great Wall.) In fact, I think laying out a grand vision of the project and working on little bits here and there is a remarkably bad way to approach the development unless those little bits are useful to you in isolation. Otherwise you're likely to end up with bits that aren't useful to you at all. But with a little bit of discipline about which "towers" to build first, I think this serves as a useful metaphor for an effective (and not uncommon) development practice. We're implementing a large and complex system via a series of user stories. Rather than building out a major subsystem in its entirety and then moving on to the next one, we're implementing the skeleton of the end-to-end system, to which we will add meat (that is, complexity) in subsequent iterations. We know any given story describes an incomplete subsystem at this stage, but each describes an cohesive unit of functionality, to which we can add complexity and functionality to fill out the walls between the towers.

Curly braces, Pipes, Escape and other characters on the Zaurus

Mon, 21 Jul 2003 18:56:30 GMT

If you've ever tried to type code, pseudo-code or shell scripts on the Sharp Zaurus, you may have noticed that the little slide-out keyboard is missing some useful keys. If you've done much typing with this thumb-keyboard, you may have noticed that when you fat-finger a couple of keys, you can get some of those extra characters. For my reference as much as yours, here's a short list:

keys	character	unicode value	name
[Fn]+z		0x005A(?)	undo
[Fn]+[Shift]+c	€	0x20AC	euro symbol
[Fn]+[Shift]+[Backspace]	[	0x005B	left square bracket
[Fn]+[Shift]+,	]	0x005D	right square bracket
[Fn]+[Shift]+.	{	0x007B	left curly brace
[Fn]+[Shift]+[Enter]	}	0x007D	right curly brace
[Fn]+[Shift]+'	^	0x005E	caret/circumflex
[Fn]+[Shift]+[Space]	`	0x0060	tick/backquote
[Shift]+[Space]	\|	0x007C	pipe
[Shift]+[Tab]	\	0x005C	backslash (note that this is listed incorrectly in the sharp doc)

In the table above, "[Fn]" means the purple "function" key, "[Shift]" means the arrow-up "shift" key, "[Backspace] means the white back-arrow/delete key, "[Enter]" means the purple "return" key, "[Space]" means the space bar, "[Tab]" means the purple tab key, and a "+" means hit these keys in combination, typically by holding down the "meta" keys first.

Also notice that the "Cancel" button works like "Escape", which makes VI usable without resorting to the on-screen (virtual) keyboard.

There's a full keycode mapping table available on Sharp's site.

LGPL and Java: More confused than ever

Fri, 18 Jul 2003 17:35:52 GMT

After reading the various comment threads, and seeing Andy and David's exchange in a bit more context, I'm more confused than ever.

As quoted by Andy, Roy wrote:

What the FSF needs to say is that inclusion of the external interface names (methods, filenames, imports, etc. defined by an LGPL jar file, so that a non-LGPL jar can make calls to the LGPL jar's implementation, does not cause the including work to be derived from the LGPL work even though java uses late-binding by name (requiring that names be copied into the derived executable), and thus does not (in and of itself) cause the package as a whole to be restricted to distribution as (L)GPL or as open source per section 6 of the LGPL.

and Andy asked:

Is this statement true with regards to the use of LGPL Java libraries by non-LGPL Java libraries?

To which David replied:

If I understand the statement correctly, yes -- that's exactly what section 6 is for.

Seen in context, my reading is that LGPL does not infect Java code that simply references, invokes methods of, or extends an LGPL'ed class. Brad Kuhn's comments mentioned in the slashdot "update" seem to be reiterating that position. I see this as saying all of my examples are not infectious. Joe's comments on that post focus on the word "distribute" in what may be a clarifying way (i.e., the in-memory, late-bound version of code that combines LGPL and non-LGPL code is not distributed, it is assembled at runtime, and therefore not in violation).

I hesitate to ask but the question seems inevitable: what does it mean to apply aspects to LGPL'ed code? If I hook in code to a cut-point in some LGPL'ed code, is my cut-point code infected? Does it matter if my AOP-system is implemented with byte-code manipulation, reflection or composition? Does it matter if this happens at runtime or at compile time? It seems that it would, and that feels like a fairly arbitrary distinction in some ways.

IANAL and this makes my head spin. I'm going back to writing code. Someone let me know when there's a clear answer here. (Unfortunately I think it may take a court to decide that. Let's hope it never comes to that.)

The [L]GPL, Java and Asymmetry

Wed, 16 Jul 2003 17:35:04 GMT

Andy has finally tracked down some answers on applying the LGPL to Java. Dave and Sam had some additional comments.

A few points:

In his comment thread Dave suggests that "Class.forName() might also be a workaround", echoing questions on the LGPL in Java I previously mentioned. The answers to those questions still aren't entirely clear to me, although the position of simply steering clear of LGPL/GPL code for Java is seems increasingly rational.
Sam writes "An example of something that does solve the issue: JDBC" but I'm not sure I follow. Does this mean Class.forName? In other words, I can use a LGPL'd database code as long as I never directly invoke the LGPL code?
Andy writes "Thus we need a license for Java that guarantees contributors will donate back to the library that does not infect the outside code."

Do we? Why? I think there is sufficient incentive to release most derivative works anyway, and if someone doesn't, who cares? The open source project still gets more users, more support, more field testing, mo' better, than it otherwise would. If you make a few bucks selling proprietary extensions to my open source code, more power to you. (You run the threat, of course, of the community re-implementing your proprietary extensions in an open source fashion, thus destroying your business model. This disincentive to selling proprietary derivatives augments the incentives for releasing those derivatives to the community for further extension, testing, maintenance and support.)

A big issue I have with the [L]GPL is the asymmetry of the license. If Company X releases code under the [L]GPL, I'm less troubled by the need to similarly open my derivative works (although the people who sign my paycheck quite reasonably have a different perspective on that--there is some work that is best kept proprietary, if only because it pays the bills) than by the fact that Company X, typically acting as the "umbrella" copyright holder (to both their original work and my contributions) now has more rights to my contributions than I do. Under the viral GPL, I cannot use the larger work under any terms but those provided by the GPL, and in isolation my contributions are probably pretty much useless. Yet Company X (as the holder of the collective copyright) is free to take my (donated) work and theirs and release (or sell) it under whatever terms they choose. I wonder if Stallman had foreseen this consequence when constructing the GPL.

The "umbrella" copyright holder on a BSD- or ASF-type license has the same rights of course, but the license grants me the ability to do pretty much whatever I want with the larger work, short of claiming it all to be my own. There is very little that the copyright holder can do that I, as a user let alone contributor, cannot also do. It seems to me that the license with the least restrictive terms is the one that is most "free".

my pet bug, or another example of how sun doesn't get community development

Tue, 15 Jul 2003 20:02:07 GMT

I've submitted a few insignificant bugs to Sun's Bug Parade over the years, most recently "Collections.ReverseOrder.equals method is lacking", which I submitted back in January of 2003 and was finally accepted yesterday (14 July).

This isn't a critical bug, of course: It's easy to work around (that is, replace) and frankly it's been an annoyance but not a real problem in my development efforts. Moreover, it seems like Sun has been pretty busy over the last few months working on other things. Nevertheless, this whole experience has been frustrating.

First, in my limited and anecdotal experience the time to respond to bug reports is getting much worse. It took six months for the Java team to acknowledge this simple, and if I may say reasonably well documented (including a "patch" and unit test), bug.

Second, this is a good if trivial example of how Sun's community efforts fail in execution. Here's a trivially simple (it's clear from inspection alone) and readily demonstrated (a complete unit test is provided) defect. A trivially simple (one line, two if you add hashCode()) patch is provided, and yet it took six months to get the issue acknowledged. Now I'll wait an indeterminate amount of time to see the problem fixed. All that and this problem could have been analyzed, patched and regression tested in less time than it took me to write this post. Even the slowest moving of open source projects would have had this problem patched, if not released, in a matter of weeks. If Sun can't find a way to move more quickly (and stop chasing the most superficial features of C#) Java seems destined to be eclipsed by community developed languages.

I'll uncharacteristically paraphrase Jerry Seinfeld here: "Sun knows how to take community input, they just don't know what to do with it. And that's really the most important part of community: the doing." (text, audio)

PS: Also note that the second bug I linked to above, "Repackage sun.net.www.content.* Classes (java.net.ContentHandler)", was already addressed by the time I submitted it, I personally added a comment to this effect nearly two years ago, and yet the bug sits marked "In progress" (and with 3 votes, none of them mine).