David Black: Architecture

An approach to monitoring and managing clustered servers as a single entity

Tue, 23 Jan 2007 14:01:00 GMT

Often we need to be able to monitor and manage clustered servers as a single entity. This is an outline of an approach I’ve used before to monitor and manage a clusterable enterprise server product which had very demanding RASP and throughput requirements. It is based on JMX, but allows for an aggregated view of the mbeans (monitoring attributes and management operations) in the cluster i.e. it allows for things to be examined in a meaningful way at the cluster (or service) level, while at the same time being able to view an individual nodes value for that attribute.

It also enables an operation to be executed once for the whole cluster e.g. flush a cache, rather than having to manually do that for each server instance in turn. This can be achieved using a standard single server aware mbean console (a limitation of most off the shelf open source ones).

The kind of aggregation appropriate for an individual attribute can be easily configured, and a view can be presented at the cluster level and at the individual servers (node) level. Different attributes require different kinds of aggregation. For example, the aggregation of the number of items in a given cache across the cluster would be the sum of all the individual cache sizes, but at the same time we also want to be able to see the value for an individual node. In the case of message throughput the aggregation might be an average of all the individual servers throughputs, and we might also want a count across the cluster of messages processed to date, as well as those statistics on an individual node basis. For another attribute it might not be sensible to aggregate it, so we might just want the individual values for each node to be shown together. The point is that a sensible “cluster” view of a given attribute differs.

The basic idea is to instrument the classes that house the stuff you want to expose as normal using an MBean interface. So each node in the cluster exposes all its stuff as an individual server. Additional metadata is added to the MBean interface using variables (static final Strings), which describe what is to be done with the individual attributes and operations. I say static final Strings because this annotation is Java 1.4.x compatible, but if you're using Java 5 then of course you can use "real" annotations.

The mbeans are registered with the mbean server via a façade – lets call it MBeanServerFacade. The facades job is to interrogate the mbeans and extract metadata using introspection. A simple convention is followed for the variable names in order to associate them with the monitoring attributes and management operations. The values of the strings then describe how the attribute is aggregated (and other useful things). The metadata could be described somewhere else, but the advantage of putting it on the mbean interface is that its easy to develope, and it lives with the attributes, so when the mbean changes the metadata can be easily changed with it. Also, it is really simple to add these annotated mbeans so there is no barrier to adding instrumentation to the server (and for clustered servers setups you can never know enough about what is going on under the covers). MBeanServerFacade writes the metadata to a simple database table, if that particular type of mbean hasn’t already been registered. It also writes the host address of each node in the cluster to another table.

The next element in the architecture is a broker process. This should (but does not have to) live separately to the cluster, so on a standalone server instance. The MBeanServerBroker reads (and periodically reloads) the descriptions in the tables populated by MBeanServerFacade. So the broker has a full description of all the relevant mbeans available and all the servers where they can be found. MBeanServerBroker is used as an intermediary between the various MBeanServers and the calling application. Using the metadata MBeanServerBroker creates JMX dynamic mbeans which expose an appropriate aggregate view of the individual mbeans across the cluster. Because it uses dynamic mbeans, the aggregate mbeans can also expose the individual attribute values (and which server they belong to). When an aggregate operation is invoked from the console, MBeanServerBroker runs that operation on each node in the cluster. The other important job of the broker is caching the returned mbean information. This is because JMX requests are expensive both in terms of the caller (waiting/blocking) and the component exposing the mbeans (doing the work of gathering the mbean info etc.). The console remains responsive because its hitting a cache, and the broker updates the cached values frequently. It’s the dynamic mbeans exposed by MBeanServerBroker that the console looks at.

Here is an example of an annotated mbean interface:

interface MyStatsMBean {

/** attributes */

int getCacheSize();
int getAvgJmsReadsPerSecond();
int getMaxInsertMessageTime();
int getMinInsertMessageTime();
int getUniqueValue();

/** annotations */

String getCacheSize_AM = AggregateMethod.SUM;
String getAvgJmsReadsPerSecond_AM = AggregateMethod.AVG;
String getMaxInsertMessageTime_AM = AggregateMethod.MAX;
String getMinInsertMessageTime_AM = AggregateMethod.MIN;
String getUniqueValue_AM = AggregateMethod.LIST;
}

In this example we want to see the sum of cache sizes across the cluster, the average JMS reads per second across all nodes, the slowest message insert into the database, as well as the quickest, and a list of individual values for an attribute that it does not make sense to aggregate.

The OSGi bandwagon effect

Tue, 16 Jan 2007 14:01:50 GMT

No one could ever accuse me of not jumping on a well spotted bandwagon. Good introductory article on OSGi and bundle development with Eclipse.

OSGi-everywhere

Mon, 15 Jan 2007 18:10:23 GMT

There was a lot of buzz in 2006 around OSGi and it looks like 2007 will be the OSGi-everywhere year. One of the more interesting posts I've come across is this one, a reference to a talk about the Newton open source project and dynamic service grids in the enterprise and OSGi as a basis for SOA. I'm hoping to try some of this out for large-scale (and large scaling) distributed systems shortly. More on that in due course.

Cape Clear 7.0 BETA is now available

Wed, 03 Jan 2007 08:23:57 GMT

Happy New Year and Happy New BETA! The Cape Clear 7.0 BETA is now available for download. Check out the brief summary of some new features, participate in the BETA on developer.capeclear.com and have a look at the detailed What's New. I reckon its going to be a great 2007 :-)

Cape Clear ESB keeps on winning with IT Week Magazine Editors Choice Award for 2006

Wed, 09 Aug 2006 09:50:07 GMT

"One of the best ESB products available and it wins our Editor's Choice Award for functionality and performance." - IT Week, July 31, 2006

"Clear's Cape Clear ESB 6.6 provides an intuitive and capable platform for creating, managing and deploying services, processes and data for an SOA infrastructure."

And Cape Clear will just keep getting stronger - we have a host of goodies in store for later this year. I can't wait! :)

SCA v0.95 - some much need simplification

Fri, 04 Aug 2006 16:16:13 GMT

I am so glad to see that in the latest SCA draft specification the assembly composition has been greatly simplified using a recursive model. I thought the whole module, fragment, sub-system and system thing was hopelessly confusing (how many times did I read the 0.9 spec?!). Now we have Composities (which are just the same as Components) that include other Components and so on. So its basically composite services and services.

New Cape Clear ESB University courses starting shortly

Thu, 03 Aug 2006 08:31:09 GMT

The Cape Clear ESB University is an educational and training initiative aimed at providing software developers, systems integrators and IT architects with the information and tools they need to ensure the success of their Service-Oriented Architecture (SOA) projects. These live educational webcasts, which will be held during the month of August, will support organizations in the design, development, deployment and management of SOA and ESB projects, using relevant business case examples.

You can register for these courses here.

The first course covers the Principles of SOA Design Using Enterprise Service Bus. This session focuses on the SOA creation process, principles of good SOA design, and the use of an ESB to enable SOA projects.

During this event you will learn:

- Service-Oriented Architecture: The accepted architecture for reducing development costs, increasing IT agility and improving IT/business alignment
- Enterprise Service Bus: The new technology platform for realizing SOA
- Standards: What are the standards that relate to SOA?
- True ESB: The standards-based ESB platform for SOA

On these courses you can learn how Enterprise Service Bus can Leverage your SOA Initiative.

The Cape Clear ESB University has been designed to accommodate the diverse needs of those who are responsible for creating a SOA strategy, design and implementation, helping to ensure projects result in reduced development costs, and increased business agility.

The ESB University webcasts will include sessions on the following topics:
- Course 1: The Principles of SOA Design Using Enterprise Service Bus
- Course 2: SOA--Introduction to Process Orchestration and Top Down Service Design
- Course 3: Mapping and Transforming Diverse Data Formats Using SOA
- Course 4: Service Enablement and Quality of Service with SOA

Sign up now!

---------------------------------------

Course 1, August 9, 2006

8 AM PDT/11 AM EDT/4 PM GMT

The Principles of SOA Design Using Enterprise Service Bus

---------------------------------------

Course 2, August 16, 2006

8 AM PDT/11 AM EDT/4 PM GMT

SOA - Introduction to Process Orchestration and Top Down Service Design

---------------------------------------

Course 3, August 23, 2006

8 AM PDT/11 AM EDT/4 PM GMT

Mapping and Transforming Diverse Data Formats Using SOA

---------------------------------------

Course 4, August 30, 2006

8 AM PDT/11 AM EDT/4 PM GMT

Service Enablement and Quality of Service with SOA

---------------------------------------

OSGi tutorial at ApacheCon Dublin

Wed, 28 Jun 2006 21:37:47 GMT

ApacheCon Europe 2006 is being held in Dublin this week. I couldn't attend the whole week, but I did get to go to a half-day OSGi tutorial given by longtime OSGi evangelist Peter Kriens. Peter, who is the current editor of the OSGi specification, says:

"OSGi is kind of SOA"

"OSGi is kind of a Java Operating System" or at least a "mini-kernel"

OSGi is "the Dynamic Modularity Layer for Java"

OSGi is a Java framework for developing remotely deployed service applications. OSGi started life in 1998 with the original use case of home automation. OSGi applications can run on different software and hardware architectures. There is a lot of buzz around it right now, particularily in the enterprise space, partially because of Eclipse Equinox, which is OSGi R4 based and the technology underlying Eclipse bundles (actually Equinox goes beyond pure OSGi). But there are a bunch of other implementations, including the Apache Felix project. The specification is freely available and royalty free.

OSGi applications adhere to a basic SOA, the contract between client and service is based on a Java interface and discovering and binding available implementations is done through the framework. Client and service are softly coupled. So it provides an in-VM service model. Declarative services, introduced in R4, are akin to having dependency injected components. Applications declare their service dependencies in an XML file, and are notified by the framework at runtime as services becomes available and go away. The last is an important point: services can go away at runtime, just as they can be dynamically discovered. The next major release of Spring is adding support for OSGi. One of the tutorial attendees made the point that OSGi is a generic event notification framework.

OSGi applications are packaged as bundles, handled by the OSGi Module Layer. The Module Layer addresses packaging, class loading modularization, protection and versioning (multiple versions of the same package) - things Java does not do well, or at all, by itself. In addition, OSGi provides basic Life Cycle Management. An API for managing bundles allows them to be installed, resolve their dependencies, started, stopped, refreshed, updated and uninstalled. So basically, bundles register one or more services with the framework service directory, and other bundles can use those services. Service contracts are just Java interfaces, so service implementations are just POJOs and no special classes need be extended nor interfaces implemented.

Its a pity OSGi wasn't somehow used as the basis of JBI, there are so many similarities.

Its also a pity that there are currently two rival JSRs attempting to establish a dynamic modularity layer for Java SE. JSR-291 is the OSGi Alliance backed one that aims to "define a dynamic component framework including component lifecycle for existing Java SE platforms". Peter is on the expert group. JSR-277 aims to do more or less the same thing, expect not as well according to Peter - who is is not on the expert group despite his best efforts! Richard Hall was at the tutorial, he is on both groups. According to Richard he is trying to steer the 277 folks in the right direction. It will be interesting to see what happens. Richard was giving a couple of OSGi related talks in Dublin, I wish I had been able to attend them.

And I think the Eclipse Rich Server Platform proposal deserves some serious attention. There is so much happening in the module/component/service packaging, assembling and deployment space (the McSPAD space) (ok, I guess thats several kind of overlapping spaces). We've got JBI, SCA (though don't hold your breath) and now (yes it has been around for quite some time) OSGi. What we really need to do is get everyone in an aircraft hanger (big enough?), seal the doors and tell them they are not getting out until there is only one. One McSPAD to rule them all.

AMQP: an open specification for queue based messaging

Sun, 25 Jun 2006 20:13:40 GMT

The Advanced Message Queuing Protocol (AMQP) 0.8 spec has been released. AMQ is an open specification for queued messaging that was born at JPMorgan Chase under the direction of John O'Hara. JPMorgan have formed the AMQP Working Group, together with a bunch of companies including Red Hat, Cisco and Iona. They aim to create an alternative to expensive proprietary messaging middleware like IBMs MQSeries. IBM aren't the only vendor selling an overpriced queuing system, I reckon TIBCO fall into that category too. I understand this has been brewing behind the scenes for a while, but a lot of folks weren't sure it would ever see the light of day. As Brian McCallister puts it this is a space badly in need of an open standard, but has concerns about whether it will be properly opened up. According to InfoQ JPMorgan have an implementation in production now. I've worked on a number of messaging systems for investment bank trading floors (and beyond) over the years, going back to NetBIOS days, reliable broadcast with UDP/IP, point-to-point based systems with TCP/IP, including HTTP tunneling based Internet trading infrastructure. Back in those days the messaging systems were the product. These days they should just be part of the woodwork, and hopefully AMQP, OpenWire and implementations like ActiveMQ can bring scalable, performant and reliable asychronous messaging to all enterprises.

Enabling effective BAM with the ESB

Sat, 10 Jun 2006 12:44:24 GMT

By now most folks understand the advantages of running and orchestrating key business processes using SOA principles and Enterprise Service Buses, like the web service standards based Cape Clear ESB. There is a lot of business activity taking place on the ESB, and it'd be great to be able to monitor that, gather the intelligence and analyze what our businesses are telling us. In other words putting the ESB and Business Activity Monitoring (BAM) together.

One of the key issues when considering BAM is being non-invasive. A BAM should be able to collect the data it needs to present a real-time dashboard of business state and Key Performance Indicators (KPIs), without you having to engineer that into your applications. Another key issue is correlation. Lots of bits of data and process state go into creating a holistic picture of a business process. A BAM needs to be able to tie all those bits together. Different systems often use different keys to identify a particular instance of a process. Sometimes the interaction between systems is terse e.g. in a synchronous request/response scenario the response message might not even have a process identifier. This can make it difficult for the BAM to create that holistic view and show you everything you want to see.

Fortunately, both of these issues can be addressed very effectively by the ESB. When business services are exposed on an ESB, the bus is handling all the messages as they flow between services. These messages and their orchestrated flow between services define a large part of the process. The ESB itself can event out these messages to the BAM, there is no need for invasive engineering - so collection is free. The ESB is already doing correlation, internally it knows what process instances it is dealing with, so it can add that information to the messages it sends to the BAM, together with a whole bunch of other useful stuff like timestamps and source and destination services.

Sometimes of course, things happen inside of applications that are invisible to the bus, and may be difficult to collect, but that are important to know. Again the ESB can help by exposing its BAM collection function as just another service that applications can invoke. With the Cape Clear ESB we're enabling effective BAM. We're doing a whole bunch of other great stuff to make BAM easier too, I'll post more on that soon.

In the meantime, how about a new TLA: SAM - SOA Activity Monitoring :-)

Landed at the Cape

Sat, 06 May 2006 20:35:11 GMT

I've wanted to get into the SOA and Enterprise Service Bus space for some time now. When an opportunity arose to join Cape Clear Software, the leading ESB vendor, I was excited by the prospect and impressed by the people I met. From a technical perspective, it fit nicely with my keen interest in Eclipse tooling on the one hand, server technology, and enterprise architecture in general. So pretty much the whole kit and kaboodle then! This is clearly a red hot space right now, and rightly so, SOA is changing the way we do so many things. Cape Clear have a stellar SOA platform that is ready to go out of the box, giving you tons of loosely coupled integration options that work first time. So I charted a course, and I've just landed at the Cape. One of the things I'm going to be helping Cape Clear do is to define extension points for BAM (Business Activity Monitoring). More on that later.

Farewell PaceMetrics

Sun, 23 Apr 2006 17:43:52 GMT

In a few days time I'm finishing up at PaceMetrics, the company I've been with for the last three years. Leaving Pace was a tough enough decision for me. I joined on an up curve just as the company was starting to build its fourth generation BAM (Business Activity Monitor), and winning a huge deal with a tier one investment bank. To deliver that project and productise it, I was lucky enough to quickly assemble one of the best software development teams I've ever had. This team was world class, all experienced, seasoned, innovative, agile engineers. We had a really stellar project manager and business architect in Colm Toolan, and a super professional services team, and the result was PaceMaker 4:

- fully Java, J2EE (POJOs of course!) and JMS based, leveraging best-of-breed open source components and tools
- loosely coupled components communicating asynchronously using XML-based messagess
- a simple maven based build
- a super-scalable, fully reliable, and architecturally innovative BAM engine - thanks James :-)
- a scalable web front end built on some pretty cool caching technology
- enterprise ready monitoring and management through JMX from our own custom web based admin console
- Eclipse based tooling (courtesy of yours truely)

Its been a great few years, but in the end I've been tempted away by the shimmering lights of SOA and ESBs. I'll post more on that in a couple of days.

Weblogic Workshop? I don't think so

Thu, 18 Dec 2003 22:02:18 GMT

Have you seen the new Weblogic Workshop released with Weblogic 8.0? Are you impressed by the flash demo, cool graphics and seamless BPM workflow visual editor based development?

I've just had quite a discussion with some of my companies product management people. It was suggested we ditch the open source and open standards based stuff we're using to build a major web application for a big customer we have, in favour of this new workshop and BEAs webflow technology. So right now we're using Eclipse for development, Struts, Tiles, JSTL and EL. Hmmm, let me think about that for just a nanosecond - no way dude! First of all, why didn't they just build on Eclipse? Why reinvent an inferior, Swing-based (barf) IDE wheel? And all this jpd stuff is totally proprietary. Even if BEA push to standardize some of this it'll be years before there are other implementations (and it'll be different). I just can't see too many people ditching Eclipse and best use open source for this. I guess BEA have deep pockets, but I think this is a mistake.