Tuesday, January 23, 2007

An approach to monitoring and managing clustered servers as a single entity Often we need to be able to monitor and manage clustered servers as a single entity. This is an outline of an approach I’ve used before to monitor and manage a clusterable enterprise server product which had very demanding RASP and throughput requirements. It is based on JMX, but allows for an aggregated view of the mbeans (monitoring attributes and management operations) in the cluster i.e. it allows for things to be examined in a meaningful way at the cluster (or service) level, while at the same time being able to view an individual nodes value for that attribute. It also enables an operation to be executed once for the whole cluster e.g. flush a cache, rather than having to manually do that for each server instance in turn. This can be achieved using a standard single server aware mbean console (a limitation of most off the shelf open source ones). The kind of aggregation appropriate for an individual attribute can be easily configured, and a view can be presented at the cluster level and at the individual servers (node) level. Different attributes require different kinds of aggregation. For example, the aggregation of the number of items in a given cache across the cluster would be the sum of all the individual cache sizes, but at the same time we also want to be able to see the value for an individual node. In the case of message throughput the aggregation might be an average of all the individual servers throughputs, and we might also want a count across the cluster of messages processed to date, as well as those statistics on an individual node basis. For another attribute it might not be sensible to aggregate it, so we might just want the individual values for each node to be shown together. The point is that a sensible “cluster” view of a given attribute differs. The basic idea is to instrument the classes that house the stuff you want to expose as normal using an MBean interface. So each node in the cluster exposes all its stuff as an individual server. Additional metadata is added to the MBean interface using variables (static final Strings), which describe what is to be done with the individual attributes and operations. I say static final Strings because this annotation is Java 1.4.x compatible, but if you're using Java 5 then of course you can use "real" annotations. The mbeans are registered with the mbean server via a façade – lets call it MBeanServerFacade. The facades job is to interrogate the mbeans and extract metadata using introspection. A simple convention is followed for the variable names in order to associate them with the monitoring attributes and management operations. The values of the strings then describe how the attribute is aggregated (and other useful things). The metadata could be described somewhere else, but the advantage of putting it on the mbean interface is that its easy to develope, and it lives with the attributes, so when the mbean changes the metadata can be easily changed with it. Also, it is really simple to add these annotated mbeans so there is no barrier to adding instrumentation to the server (and for clustered servers setups you can never know enough about what is going on under the covers). MBeanServerFacade writes the metadata to a simple database table, if that particular type of mbean hasn’t already been registered. It also writes the host address of each node in the cluster to another table. The next element in the architecture is a broker process. This should (but does not have to) live separately to the cluster, so on a standalone server instance. The MBeanServerBroker reads (and periodically reloads) the descriptions in the tables populated by MBeanServerFacade. So the broker has a full description of all the relevant mbeans available and all the servers where they can be found. MBeanServerBroker is used as an intermediary between the various MBeanServers and the calling application. Using the metadata MBeanServerBroker creates JMX dynamic mbeans which expose an appropriate aggregate view of the individual mbeans across the cluster. Because it uses dynamic mbeans, the aggregate mbeans can also expose the individual attribute values (and which server they belong to). When an aggregate operation is invoked from the console, MBeanServerBroker runs that operation on each node in the cluster. The other important job of the broker is caching the returned mbean information. This is because JMX requests are expensive both in terms of the caller (waiting/blocking) and the component exposing the mbeans (doing the work of gathering the mbean info etc.). The console remains responsive because its hitting a cache, and the broker updates the cached values frequently. It’s the dynamic mbeans exposed by MBeanServerBroker that the console looks at. Here is an example of an annotated mbean interface: interface MyStatsMBean { /** attributes */ int getCacheSize(); int getAvgJmsReadsPerSecond(); int getMaxInsertMessageTime(); int getMinInsertMessageTime(); int getUniqueValue(); /** annotations */ String getCacheSize_AM = AggregateMethod.SUM; String getAvgJmsReadsPerSecond_AM = AggregateMethod.AVG; String getMaxInsertMessageTime_AM = AggregateMethod.MAX; String getMinInsertMessageTime_AM = AggregateMethod.MIN; String getUniqueValue_AM = AggregateMethod.LIST; } In this example we want to see the sum of cache sizes across the cluster, the average JMS reads per second across all nodes, the slowest message insert into the database, as well as the quickest, and a list of individual values for an attribute that it does not make sense to aggregate. 2:01:00 PM    comment []

Tuesday, January 16, 2007

The OSGi bandwagon effect No one could ever accuse me of not jumping on a well spotted bandwagon. Good introductory article on OSGi and bundle development with Eclipse. 2:01:50 PM    comment []

Monday, January 15, 2007

OSGi-everywhere There was a lot of buzz in 2006 around OSGi and it looks like 2007 will be the OSGi-everywhere year. One of the more interesting posts I've come across is this one, a reference to a talk about the Newton open source project and dynamic service grids in the enterprise and OSGi as a basis for SOA. I'm hoping to try some of this out for large-scale (and large scaling) distributed systems shortly. More on that in due course. 6:10:23 PM    comment []

Wednesday, January 03, 2007

Cape Clear 7.0 BETA is now available Happy New Year and Happy New BETA! The Cape Clear 7.0 BETA is now available for download. Check out the brief summary of some new features, participate in the BETA on developer.capeclear.com and have a look at the detailed What's New. I reckon its going to be a great 2007 :-) 8:23:57 AM    comment []

Wednesday, August 09, 2006

Cape Clear ESB keeps on winning with IT Week Magazine Editors Choice Award for 2006 "One of the best ESB products available and it wins our Editor's Choice Award for functionality and performance." - IT Week, July 31, 2006 "Clear's Cape Clear ESB 6.6 provides an intuitive and capable platform for creating, managing and deploying services, processes and data for an SOA infrastructure." And Cape Clear will just keep getting stronger - we have a host of goodies in store for later this year. I can't wait! :) 9:50:07 AM    comment []

Friday, August 04, 2006

SCA v0.95 - some much need simplification I am so glad to see that in the latest SCA draft specification the assembly composition has been greatly simplified using a recursive model.  I thought the  whole module, fragment, sub-system and system thing was hopelessly confusing (how many times did I read the 0.9 spec?!). Now we have Composities (which are just the same as Components) that include other Components and so on. So its basically composite services and services. 4:16:13 PM    comment []

Thursday, August 03, 2006

New Cape Clear ESB University courses starting shortly The Cape Clear ESB University is an educational and training initiative aimed at providing software developers, systems integrators and IT architects with the information and tools they need to ensure the success of their Service-Oriented Architecture (SOA) projects. These live educational webcasts, which will be held during the month of August, will support organizations in the design, development, deployment and management of SOA and ESB projects, using relevant business case examples.   You can register for these courses here. The first course covers the Principles of SOA Design Using Enterprise Service Bus. This session focuses on the SOA creation process, principles of good SOA design, and the use of an ESB to enable SOA projects. During this event you will learn: - Service-Oriented Architecture: The accepted architecture for reducing development costs, increasing IT agility and improving IT/business alignment - Enterprise Service Bus: The new technology platform for realizing SOA - Standards: What are the standards that relate to SOA? - True ESB: The standards-based ESB platform for SOA On these courses you can learn how Enterprise Service Bus can Leverage your SOA Initiative. The Cape Clear ESB University has been designed to accommodate the diverse needs of those who are responsible for creating a SOA strategy, design and implementation, helping to ensure projects result in reduced development costs, and increased business agility. The ESB University webcasts will include sessions on the following topics: - Course 1: The Principles of SOA Design Using Enterprise Service Bus - Course 2: SOA--Introduction to Process Orchestration and Top Down Service Design - Course 3: Mapping and Transforming Diverse Data Formats Using SOA - Course 4: Service Enablement and Quality of Service with SOA Sign up now! --------------------------------------- Course 1, August 9, 2006 8 AM PDT/11 AM EDT/4 PM GMT The Principles of SOA Design Using Enterprise Service Bus --------------------------------------- Course 2, August 16, 2006 8 AM PDT/11 AM EDT/4 PM GMT SOA - Introduction to Process Orchestration and Top Down Service Design --------------------------------------- Course 3, August 23, 2006  8 AM PDT/11 AM EDT/4 PM GMT Mapping and Transforming Diverse Data Formats Using SOA --------------------------------------- Course 4, August 30, 2006 8 AM PDT/11 AM EDT/4 PM GMT Service Enablement and Quality of Service with SOA --------------------------------------- 8:31:09 AM    comment []

Wednesday, June 28, 2006

OSGi tutorial at ApacheCon Dublin ApacheCon Europe 2006 is being held in Dublin this week. I couldn't attend the whole week, but I did get to go to a half-day OSGi tutorial given by longtime OSGi evangelist Peter Kriens. Peter, who is the current editor of the OSGi specification, says: "OSGi is kind of SOA" "OSGi is kind of a Java Operating System" or at least a "mini-kernel" OSGi is "the Dynamic Modularity Layer for Java" OSGi is a Java framework for developing remotely deployed service applications. OSGi started life in 1998 with the original use case of home automation. OSGi applications can run on different software and hardware architectures. There is a lot of buzz around it right now, particularily in the enterprise space, partially because of Eclipse Equinox, which is OSGi R4 based and the technology underlying Eclipse bundles (actually Equinox goes beyond pure OSGi). But there are a bunch of other implementations, including the Apache Felix project. The specification is freely available and royalty free. OSGi applications adhere to a basic SOA, the contract between client and service is based on a Java interface and discovering and binding available implementations is done through the framework. Client and service are softly coupled. So it provides an in-VM service model. Declarative services, introduced in R4, are akin to having dependency injected components. Applications declare their service dependencies in an XML file, and are notified by the framework at runtime as services becomes available and go away. The last is an important point: services can go away at runtime, just as they can be dynamically discovered. The next major release of Spring is adding support for OSGi. One of the tutorial attendees made the point that OSGi is a generic event notification framework. OSGi applications are packaged as bundles, handled by the OSGi Module Layer. The Module Layer addresses packaging, class loading modularization, protection and versioning (multiple versions of the same package) - things Java does not do well, or at all, by itself. In addition, OSGi provides basic Life Cycle Management. An API for managing bundles allows them to be installed, resolve their dependencies, started, stopped, refreshed, updated and uninstalled. So basically, bundles register one or more services with the framework service directory, and other bundles can use those services. Service contracts are just Java interfaces, so service implementations are just POJOs and no special classes need be extended nor interfaces implemented. Its a pity OSGi wasn't somehow used as the basis of JBI, there are so many similarities. Its also a pity that there are currently two rival JSRs attempting to establish a dynamic modularity layer for Java SE. JSR-291 is the OSGi Alliance backed one that aims to "define a dynamic component framework including component lifecycle for existing Java SE platforms". Peter is on the expert group. JSR-277 aims to do more or less the same thing, expect not as well according to Peter - who is is not on the expert group despite his best efforts! Richard Hall was at the tutorial, he is on both groups. According to Richard he is trying to steer the 277 folks in the right direction. It will be interesting to see what happens. Richard was giving a couple of OSGi related talks in Dublin, I wish I had been able to attend them. And I think the Eclipse Rich Server Platform proposal deserves some serious attention. There is so much happening in the module/component/service packaging, assembling and deployment space (the McSPAD space) (ok, I guess thats several kind of overlapping spaces). We've got JBI, SCA (though don't hold your breath) and now (yes it has been around for quite some time) OSGi. What we really need to do is get everyone in an aircraft hanger (big enough?), seal the doors and tell them they are not getting out until there is only one. One McSPAD to rule them all. 9:37:47 PM    comment []

© Copyright 2007 David Black. Last update: 2/17/2007; 4:04:04 PM. This theme is based on the SoundWaves (blue) Manila theme.