Tuesday, January 23, 2007

An approach to monitoring and managing clustered servers as a single entity Often we need to be able to monitor and manage clustered servers as a single entity. This is an outline of an approach I’ve used before to monitor and manage a clusterable enterprise server product which had very demanding RASP and throughput requirements. It is based on JMX, but allows for an aggregated view of the mbeans (monitoring attributes and management operations) in the cluster i.e. it allows for things to be examined in a meaningful way at the cluster (or service) level, while at the same time being able to view an individual nodes value for that attribute. It also enables an operation to be executed once for the whole cluster e.g. flush a cache, rather than having to manually do that for each server instance in turn. This can be achieved using a standard single server aware mbean console (a limitation of most off the shelf open source ones). The kind of aggregation appropriate for an individual attribute can be easily configured, and a view can be presented at the cluster level and at the individual servers (node) level. Different attributes require different kinds of aggregation. For example, the aggregation of the number of items in a given cache across the cluster would be the sum of all the individual cache sizes, but at the same time we also want to be able to see the value for an individual node. In the case of message throughput the aggregation might be an average of all the individual servers throughputs, and we might also want a count across the cluster of messages processed to date, as well as those statistics on an individual node basis. For another attribute it might not be sensible to aggregate it, so we might just want the individual values for each node to be shown together. The point is that a sensible “cluster” view of a given attribute differs. The basic idea is to instrument the classes that house the stuff you want to expose as normal using an MBean interface. So each node in the cluster exposes all its stuff as an individual server. Additional metadata is added to the MBean interface using variables (static final Strings), which describe what is to be done with the individual attributes and operations. I say static final Strings because this annotation is Java 1.4.x compatible, but if you're using Java 5 then of course you can use "real" annotations. The mbeans are registered with the mbean server via a façade – lets call it MBeanServerFacade. The facades job is to interrogate the mbeans and extract metadata using introspection. A simple convention is followed for the variable names in order to associate them with the monitoring attributes and management operations. The values of the strings then describe how the attribute is aggregated (and other useful things). The metadata could be described somewhere else, but the advantage of putting it on the mbean interface is that its easy to develope, and it lives with the attributes, so when the mbean changes the metadata can be easily changed with it. Also, it is really simple to add these annotated mbeans so there is no barrier to adding instrumentation to the server (and for clustered servers setups you can never know enough about what is going on under the covers). MBeanServerFacade writes the metadata to a simple database table, if that particular type of mbean hasn’t already been registered. It also writes the host address of each node in the cluster to another table. The next element in the architecture is a broker process. This should (but does not have to) live separately to the cluster, so on a standalone server instance. The MBeanServerBroker reads (and periodically reloads) the descriptions in the tables populated by MBeanServerFacade. So the broker has a full description of all the relevant mbeans available and all the servers where they can be found. MBeanServerBroker is used as an intermediary between the various MBeanServers and the calling application. Using the metadata MBeanServerBroker creates JMX dynamic mbeans which expose an appropriate aggregate view of the individual mbeans across the cluster. Because it uses dynamic mbeans, the aggregate mbeans can also expose the individual attribute values (and which server they belong to). When an aggregate operation is invoked from the console, MBeanServerBroker runs that operation on each node in the cluster. The other important job of the broker is caching the returned mbean information. This is because JMX requests are expensive both in terms of the caller (waiting/blocking) and the component exposing the mbeans (doing the work of gathering the mbean info etc.). The console remains responsive because its hitting a cache, and the broker updates the cached values frequently. It’s the dynamic mbeans exposed by MBeanServerBroker that the console looks at. Here is an example of an annotated mbean interface: interface MyStatsMBean { /** attributes */ int getCacheSize(); int getAvgJmsReadsPerSecond(); int getMaxInsertMessageTime(); int getMinInsertMessageTime(); int getUniqueValue(); /** annotations */ String getCacheSize_AM = AggregateMethod.SUM; String getAvgJmsReadsPerSecond_AM = AggregateMethod.AVG; String getMaxInsertMessageTime_AM = AggregateMethod.MAX; String getMinInsertMessageTime_AM = AggregateMethod.MIN; String getUniqueValue_AM = AggregateMethod.LIST; } In this example we want to see the sum of cache sizes across the cluster, the average JMS reads per second across all nodes, the slowest message insert into the database, as well as the quickest, and a list of individual values for an attribute that it does not make sense to aggregate. 2:01:00 PM    comment []

Tuesday, January 16, 2007

The OSGi bandwagon effect No one could ever accuse me of not jumping on a well spotted bandwagon. Good introductory article on OSGi and bundle development with Eclipse. 2:01:50 PM    comment []

Monday, January 15, 2007

OSGi-everywhere There was a lot of buzz in 2006 around OSGi and it looks like 2007 will be the OSGi-everywhere year. One of the more interesting posts I've come across is this one, a reference to a talk about the Newton open source project and dynamic service grids in the enterprise and OSGi as a basis for SOA. I'm hoping to try some of this out for large-scale (and large scaling) distributed systems shortly. More on that in due course. 6:10:23 PM    comment []

Friday, August 04, 2006

SCA v0.95 - some much need simplification I am so glad to see that in the latest SCA draft specification the assembly composition has been greatly simplified using a recursive model.  I thought the  whole module, fragment, sub-system and system thing was hopelessly confusing (how many times did I read the 0.9 spec?!). Now we have Composities (which are just the same as Components) that include other Components and so on. So its basically composite services and services. 4:16:13 PM    comment []

Wednesday, July 12, 2006

The scruffs cleans ratio I really like this post from Steve Northover. I think great engineers are about 70% scruffs and 30% cleans. 8:09:34 AM    comment []

Tuesday, October 04, 2005

Comment First Development Most folks are now familiar with Test First or Test Driven Development. Write a failing test, make it pass. Having software based tests that exercise specific units of functionality is one of the foundation stones of agile development in general and XP in particular. But to be honest (and I suspect I'm not alone) I don't really do stuff in this order too much. Once I'm happy with the architectural aspects of whatever it is I'm working on, I often start by sketching out some interfaces and empty classes, putting together a rough framework. This is a cheap exercise with Eclipse, as is refactoring if I don't quite get it right first time. At the same time I start to outline some tests, and I jump back and forth between code and test. Often what I do is something I've come to call Comment First Development. I create a class, it could be the functional one or its unit test, and then I start writing comments. These comments are my way of working out the details, and they will (with some minor rework) become the comments for the code I have written. I suppose this is kind of like writing your psuedocode in days of yore - before stamping out the punch cards :) - but looser and quicker and a lot more flexible I reckon. I find CFD works even better when it comes to changing code I haven't written. I write comments where I think stuff should go, and I tag them with some sort of short string code that'll be easy to find (e.g. DWZH). When I think I've outlined all the bits I'm going to need, I can do a search for my code and step forwards and back through my comments in the Eclipse search results view, checking what I've said. Then the coding is easy, and hey it all works first time (yeah right). DWZH = David woz here.   2:00:06 PM    comment []

Monday, September 05, 2005

Test Perfect Development Write a failing test, make it pass. Test Driven Development. Beginners Tip: write the test to the "perfect" test API - don't even think about how the test is going to work. Then go down a level, code the details to another perfect API. At some point you'll feel you cannot go any further. Then worry about implementing the API. 4:45:45 PM    comment []

Thursday, October 14, 2004

Losing the Facade I seem to have lost the plot with the Facade design pattern over the last year. I can’t help it, its just such a commonly occuring pattern. But every second class I’m writing seems to end up a facade. Maybe I need a FacadeFacade. The GoF define facade so: Provide a unified interface to a set of interfaces in a subssytem. Facade defines a higher level interface that makes the subsystem easier to use. I wonder is it time for some additional granularity here, or is it that I’m seeing facades where they don’t exist? 8:02:31 AM    comment []

© Copyright 2007 David Black. Last update: 2/17/2007; 4:04:05 PM. This theme is based on the SoundWaves (blue) Manila theme.