Jeff Potts: XML

Mon, 03 Oct 2005 02:24:30 GMT

I'm trying out a new approach (new for me, anyway) to writing doc. Over the years I've gotten so tired of messing around with Word. Assuming you get your styles to cooperate the format you wind up with is the format your doc will be in forever. Sure, you can go to PDF using Distiller, but what about other formats, like HTML? Ever seen the HTML Word produces? Yuck!

I've always wanted to write my doc in a simple XML vocabulary and then transform it into the desired format. I took a look at Docbook, and that looked promising, but it seemed like too much for what I needed. Plus, I didn't have a structured authoring tool.

Thursday I came across Apache Forrest. I had seen Forrest before while messing around with Cocoon but had never taken the time to explore it. As it turns out, it was just what I was looking for. It is a simple XML-based publishing system. Under the covers it uses the power of Cocoon to turn a simple XML format into a basic web site, PDF, or whatever format you need. Forrest is simple to get up-and-running, but the format, as simple as it may be, is a bit too painful for unassisted plain-text editing. So I still needed a structured authoring tool.

A colleague reminded me that James Clark wrote a mode for Emacs called nxml that works with RelaxNG Schemas. He also wrote a tool called trang that converts RelaxNG schemas into RelaxNG compact syntax, which is something nxml can understand. Both are easy to install.

So now, thanks to Apache Forrest, Emacs' nxml-mode, and trang, I've got a sweet little XML publishing setup.

Fri, 06 May 2005 22:52:48 GMT

Evaluating XML editors. Check out this nice review of XML editors (thanks to Owen Ambur of XML.gov). We would have liked to see some other options included, such as Ektron's browser-based XML editor and the well-regarded tool from Xopus, along with perhaps more emphasis on usability. In any case, the long list of criteria here suggests that there is more than first meets the eye when looking at tools for managing structured content...... [CMSWatch Trends and Features]

Fri, 08 Apr 2005 20:48:05 GMT

Lenya Releases Version 1.2.2.

Today the Apache Lenya team announced the release of version 1.2.2. This release includes upgrades to the WYSIWYG editors (Kupu and BXE) and some minor UI improvements. Also, a migration to Cocoon 2.1.6 leverages new features such as more rubust caching, multi-channel output, and improvements to form handling.

By noemail@noemail.org (Seth). [Enter Content Here]

Tue, 28 Dec 2004 23:14:13 GMT

Got charting working. The XMLDB pieces I noted in Step 3 and Step 4 of this post were actually very easy. The syntax for getting XML into Xindice is simple as is the querying. Once I got that going it was just a matter of hooking of the pieces of my pipeline to do what I wanted to do. I did have to tweak the XSLT that produces the SVG. I didn't build it to handle enough data points (bars were too wide, not enough graph area, etc.).

The cron and xmldb samples were really helpful in getting this working, both from a code perspective and from a functional perspective. As I stored XML in Xindice, I'd pop over to the xmldb browser sample and browse my collection to verify that it worked as expected. I used the cron sample OOTB to set up a task to run the DQL queries against Documentum on a schedule. Going forward, I'll need to incorporate an admin/config interface into my app for creating the cron task and browsing the xmldb collections.

Mon, 27 Dec 2004 06:31:31 GMT

When I dusted off my Documentum-Cocoon integration stuff I had to do a bit of a fix up. It seems that my WDK install had either rearranged some classpath entries (maybe different versions of JARs Cocoon dependend on behind its own) or made the classpath too long. In any case, I had to update the catalina.bat file to remove the WDK entries as a temporary fix.

I then noticed that when I ran any pipelines that used my Documentum-Cocoon components, they didn't seem to be getting called. My loggers weren't showing any entries and the page was just coming up blank. It turned out I had taken a little too much out of my classpath. Obviously, Tomcat needs to be able to find the Documentum DFC classes because my components rely on those. It was frustrating that no one was returning a helpful message to alert me to my blunder.

Something helpful in this situation is the Cocoon Status page in the Samples area. On that page you can show the classpath. If it doesn't see the DFC JAR and the Documentum config directory, you could be in trouble.

Mon, 27 Dec 2004 06:14:57 GMT

Made some progress on the Documentum-Cocoon stuff over the Christmas break. I'm working on a piece that will allow me to dynamically build charts and graphs based on data stored in the docbase. Here's what I've done and what's left to do:

Step 1. Figure out how to build a bar chart in SVG

First, I needed to figure out how to build a chart using SVG. I found some Java code that builds charts using SVG but what I was looking to do was build the SVG programmatically using XSLT to transform my source data which will be XML coming from my Documentum Transformer.

I needed something I could use to graphically draw the chart to get me started on the SVG. I downloaded a drawing tool called Sodipodi for this. It worked great. It stores everything as SVG. So all I had to do was draw my chart and save it to get a nice XML file to start with.

Step 2: Create the SVG with XSLT

Now that I had a static SVG file I needed to create XSLT that would take source data XML and transform it into the bar chart SVG. First, I copied my static SVG file into my Cocoon install to make sure it was happy rendering it into JPEG and PNG. It was. Next, I downloaded the Batik binaries. I wanted a tool I could use to quickly view my SVG file as I tweaked it. Batik comes with an SVG viewer called Squiggle. In WSAD, I created a test XML source data file, copied my static SVG file from Step 1 into a new XSLT file, tweaked, and then viewed the results in Squiggle.

The first thing I did was add comments so I could figure out which part of the SVG did what. In Sodipodi I had set every object's "id" attribute, so correlating the XML to the object it was describing was pretty easy. Adding comments made it easy to rearrange blocks of XML to work how I needed it to in the stylesheet.

The next thing I did was clean up the SVG by rounding off all of the numbers Sodipodi had used for the coordinates of my shapes. Obviously with a design tool everything has got to be very precise, but for what I needed, whole numbers worked just fine and it made the XML easier to look at.

I then figured out the algorithms I'd need to use to figure out how to dynamically size and position the bars and bar legends in the chart.

Finally, I replaced the hardcoded values with XSLT variables. I then updated my source XML data with new values, transformed, and voila: Dynamically generated bar chart.

Step 3: Grab the data from Documentum on a schedule

The data I need to chart is in Documentum. It changes over time. Because it's historical data I need to get a snapshot from Documentum on a schedule and store the result somewhere. Then, my chart will be built on-demand via a Cocoon pipeline using the data that's been captured for a given time period.

I already have a pipeline that gets the snapshot from Documentum. All I need to do is execute that pipeline on a schedule and store the result somewhere. I've tested out the "cron" feature of Cocoon and it works great. It allows you to execute a pipeline on a schedule. A perfect fit.

Now what I need is to put my snapshot data somewhere. I'm planning on using Xindice. It's an XML database embedded in Cocoon (you can use a separate Xindice install if you want). My pipeline will pull data from Documentum, transform it (I really only need to store a subset/summarized cut of the data), and store it in Xindice.

To complete my work for this step, I need to look at the XML:DB examples in Cocoon to learn how to stick data in Xindice from a pipeline.

Step 4: Build the chart based on the historical data

Now that I've got my historical data summarized and sitting in Xindice, I need to create a pipeline that will query Xindice for the data and transform the query result using the SVG I build in Step 2.

To complete my work for this step, I need to look at the XML:DB examples in Cocoon to learn how to query XML data from Xindice using XPath. The rest should be a piece of cake.

The result

Once this is in place, I'll have an approach and some reusable code I can use to capture and chart source data from Documentum.

Fri, 12 Nov 2004 02:25:01 GMT

Here's a new minor release of the Documentum-Cocoon integration components. It includes the reader and generator components I referenced a while back and a minor tweak to how I'm handling namespaces. (I'm not convinced I'm handling them the way I should, BTW). I've mentioned needing to develop a Documentum-specific protocol but now that I have a reader and a generator, that doesn't seem as important--readers and generators could be used as "internal" resources within a pipeline to retrieve things like XSL from the docbase to perform transformations.

I'm a little torn on what to work on next regarding this integration. I'd like a more elegant way to handle sessions. I can add single sign-on fairly easily so that when called from a WDK application the credentials would be passed to Cocoon and then used to create a session as the currently logged in Documentum user.

I'd also like to migrate the administrator dashboard prototype I've developed to use the Cocoon Portal UI framework.

Another idea is to use Cocoon scheduler to wake up periodically, execute a pipeline that reads data from Documentum, and persist that data either back to Documentum or into the Xindice XML database. This could be used to run historical reports, for example.

At Momentum I ran across a presentation by a fellow doing XML pipelining with Cocoon and Documentum. We exchanged information. If he posts his components for public consumption I'll link to them.

Mon, 25 Oct 2004 14:45:06 GMT

XML play-doh. A universal canvas requires a universal way to represent data. Other solutions are conceivable, but let's accept for now that XML is a reasonable one, and that it's here to stay. How does XML become Play-doh? The first answer, for me, was Perl with its XML::Parser module. Then came XSLT (XSL Transformation), which traded away procedural idioms to gain declarative transformational power. Then came Python with its libxml/libxslt modules, which married the procedural and declarative styles in a highly interactive way. That's been my weapon of choice lately, but now there's a new contender: E4X (ECMAScript for XML). ... [Jon's Radio]

Mon, 27 Sep 2004 03:28:00 GMT

Introduction to Structured Content Management with XML. Kay Ethier and Scott Abel offer an introductory primer on how to think about and define structure, then apply XML for better management of your documents...... [CMSWatch Trends and Features]

Tue, 24 Aug 2004 03:21:26 GMT

Ektron Updates XML Authoring Tools. New Hampshire based Ektron, has recently updated their popular eWebEditPro+XML web-based editing component. Of note is an enhanced API, giving developers further abilities to manipulated data and behavior in server-side code. eWebEditPro+XML integrates with Web content management systems and other Web-based applications to enable non-technical users to create “smart” forms and capture and validate data based on specific criteria, all... [CMSwire]

Tue, 24 Aug 2004 02:15:13 GMT

Got a Reader and a Generator added to my Cocoon-Documentum integration package last week. I also tested out the sendmail transformer that comes with Cocoon. Now, I can grab any file from Documentum from Cocoon using the Reader and then stream it to the browser or email it. If I want to transform an object stored in Documentum, I can use the Generator to snag it and then use it as I normally would in any other pipeline. For both I'm using the same protocol Documentum uses to reference documents through their "virtual link" support which is /<docbase>:<full path to content>.

I haven't made the Reader or the Generator available yet. I'll add it to the Transformer package when I get a chance.

What's left to do is create a protocol. That will allow me to use any Documentum object anywhere in a pipeline rather than just being restricted to Readers and Generators. That means I could store my XSL stylesheets as objects in the Documentum repository instead of on the file system if I wanted/needed to, although that would degrade pipeline performance.

Tue, 10 Aug 2004 03:36:54 GMT

Making progress on "Stupid Documentum Tricks with Cocoon". I've added stylesheets so I can see Documentum query results as Microsoft Excel spreadsheets or PDF. I also just got the mail action working--that means I can have Cocoon send out these files via email if needed.

Wed, 28 Jul 2004 16:10:49 GMT

Introducing o:XML o:XML is an innovative object-oriented programming language in which XML is a first class type and also provides the concrete syntax. [xml.com]

Wed, 28 Jul 2004 15:44:57 GMT

Updated list of eForms resources. [Ideas in Technology and Publishing]

Fri, 23 Jul 2004 22:48:28 GMT

Finally got the Documentum Cocoon Transformer cleaned up in WSAD to make it easier for me to share with others. Here is version 0.1. I've released it under the Apache License.

It isn't anything fancy. Very little code. There are probably other Cocoon-Documentum integration points that would be useful but I haven't knocked any out yet.

Sat, 03 Jul 2004 03:36:47 GMT

I've had some interest in the Documentum Cocoon Transformer. I'll probably make it available to whoever wants it.

Tue, 15 Jun 2004 04:50:53 GMT

Finally got my Cocoon Documentum Transformer fully functional this evening thanks to the good folks at the Cocoon Users Mailing List. My problem was that the Documentum xDQL query was returning a full XML document. When I tried to parse that, the extra startDocument and endDocument calls were causing the runtime exception (archived mailing list thread).

Now that I have my transformer working, I can configure any number of XML docs I want, each with any number of xDQL queries against any docbase my server can see. This is real handy because there are times when you want to run the same query against multiple docbases. I've currently got the pipeline configured to style the query results using a generic xDQL-to-HTML stylesheet, but now that I've got everything working, the sky is the limit. Here are some ideas...

Documentum admin portal

Go to one page and see very quickly what the status is of key metrics across all Documentum servers, docbases, and environments (Dev, QA, Prod). This is easily done using Cocoon's aggregation of my styled query results. If I really wanted to get fancy I could use the Cocoon portal framework and create an honest-to-goodness portal with multiple users and profiles (different users might want to monitor different things on different docbases) as well as security.

But, at some point, this thing starts to approach the out-of-the-box Documentum Administrator client and I'm definitely not up for reinventing that wheel. So, this one is a lower priority.

Documentum RSS alerts

Create one or more RSS feeds that will syndicate alerts for things in the environment that need attention. To do this, I'll add another transformation in my pipeline that will apply business logic to the query results and format "alerts" as XML. I will then add an alert-XML-to-RSS stylesheet. (I'm separating the two because I might want to format alerts in HTML). Once that is in place, I can use a news aggregator like Amphetadesk or NewsMonster to poll my Documentum RSS alerts periodically for things that need attention.

This one is nice because it does not force the admins to go to a web page--they just leave their aggregator running on their desktop.

Return Documentum Query Results in a Spreadsheet

Sometimes you'd like to be able to query multiple docbases and have the results in a spreadsheet so you can then do some analysis or whatever. Today you have to query each docbase and then cut-and-paste the results. With my transformer in place, I ought to be able to spit my query results into an Excel spreadsheet, PDF, SVG, whatever.

Thu, 10 Jun 2004 05:07:51 GMT

RSS vs. Atom Explained. You probably know about the schism of sorts within the headline-syndication community between RSS (which has endured a variety of irredentist movements itself) and Atom. But if yo... [CMSWatch Trends and Features]

Thu, 10 Jun 2004 04:49:32 GMT

Two things are stealing time away from the time budget previously allocated to blogging:

1. Netflix. Having a constant supply of movies on-hand that have been on my to-be-watched list for some time is awesome, and I love that I'm getting caught up. But, the blog has suffered.

2. Integrating Documentum with Cocoon. I got my DQL Transformer working some time ago but I ran into a problem transforming the DQL query result with my stylesheet. What sucks is that I had to upgrade my DFC and all of my WDK apps because of my current project, and after that, I discovered the problem. So now I've got to figure out if it was the upgrade or something else.

Until now I hadn't been able to get logging to work from my custom transformer but this problem forced me to work through that. It turned out to be a simple thing that just wasn't documented clearly. The solution was to use the logger attribute on my transformer declaration in sitemap.xmap like this:

<map:transformer name="dqlquery" src="com.navigatorsystems.cocoon.dctm.transformation.XdqlTransformer" logger="sitemap.transformer.dqlquery">

The category declaration in logkit.xconf matches up with that like this:

The DQL Transformer works like the SQL Transformer. It receives SAX events looking for a query element. When it finds it, it executes the query against Documentum as an XDQL query (ie, the DQL query results get returned as a well-formed XML document). The XML that gets returned gets sent to the next component in the pipeline (like another transformer or a serializer).

I am now back to running on Cocoon 2.1 instead of the version that came with the Cocoon book. And I'm running it on Tomcat 4.1.

Wed, 05 May 2004 05:06:44 GMT

XML databases move to the middle.

It's true that you can use native XML databases to manage the growing number of business documents created by the new generation of XML-savvy end-user applications. It's handy, for example, to search an insurance database for incident reports that match some structured pattern of in-line metadata. But hybrid SQL/XML databases can do that too, and they can also join the structured XML content with relational columns -- a powerful combination. So XML databases are migrating into a niche that SQL/XML can't and won't occupy. They're becoming the high-performance pumps that push XML traffic around on the emerging services web. [InfoWorld.com]

This short piece is a companion to Sean McCown's excellent cover story which surveys the XML features of leading relational databases: Oracle, DB2, SQL Server, Sybase. ... [Jon's Radio]

Mon, 03 May 2004 04:43:21 GMT

Still working through the Cocoon text. The book covers 2.0. I was having trouble getting some of the PDF examples and sub-map examples to work right so I went ahead and installed 2.0 instead of trying to get their examples to work on my 2.1 install. Once I'm through I'd like to circle back, read the 2.1 release notes and figure out why their 2.0 examples were broken.

Tonight I got MySQL installed, loaded with some test data, and configured for access by Cocoon. It was extremely straightforward. I want to code up a little Cocoon app for managing the MySQL test data. Then, I'm going to hook in my Xindice install and play with that through Cocoon. Next, it'll be writing a simple generator and serializer for Documentum.

Thu, 29 Apr 2004 19:25:40 GMT

XML Forms and XML Editors. Quick thought. Where do XML-based eForms end and XML-based editors begin? There is clearly some overlap. Certain kinds of content entry are well served by electronic forms—we wouldn't have built all of these industrial-strength Web sites if this weren't true.... [Ideas in Technology and Publishing]

Thu, 29 Apr 2004 19:25:05 GMT

The State of the eForms Market. I gave my opening presentation at the Seybold eForms summit. It went well, though I have to admit to being a bit in awe of the audience, which included many of the real thinkers and doers in the XHTML and... [Ideas in Technology and Publishing]

Thu, 29 Apr 2004 19:20:53 GMT

XML and the Airlines. I have a new article in Transform magazine about how Contintal Airlines has used XML and Web Services to automate some routine publishing and document review tasks. To quote briefly from the introduction: Aircraft maintenance is an especially content-centric process.... [Ideas in Technology and Publishing]

Cocoon texts

Wed, 07 Apr 2004 04:11:45 GMT

I opened up Java & XML recently because it had some references to Cocoon. The Cocoon examples are pre 2.0, so instead I worked through some of the XML-RPC chapters which were good. I'd recommend working through the whole book to anyone who's just getting started using Java and XML together.

Before I made it to my intended destination (the SOAP and web services chapters), I decided to pick up Cocoon: Building XML Applications. I had previously seen this one but passed up. So far, it is pretty good--I'm glad I came back around. They seem to do a better job explaining the architecture than some of the articles I've read recently. I like the examples I've worked through so far. The authors spend a little too much time on the evolution of the net and web applications--I wish they would have devoted that space to some of the more advanced features. It definitely gets me excited about using Cocoon on some real projects. This is a good book for anyone starting out with XML applications with Cocoon.

I'm noticing there are a few other Cocoon texts out there but I haven't taken a look at them yet. They are:

Cocoon 2 Programming: Web Publishing with XML and Java, Bill Brogden
Professional XML Development with Apache Tools: Xerces, Xalan, FOP, Cocoon, Axis, Xindice, Theodore W. Leung
Cocoon Developer's Handbook, Lajos Moczar, Jeremy Aston