A Test Automation Framework I'd Like To See - Some History

In this test automation framework project I want to learn from my experiences with test automation in the past.  Those experiences include working with two different test automation frameworks.

The first test automation framework I was exposed to was built in MFC and included a library that wrapped the Visual Test DLL, using it to drive the UI of the program under test.  This framework include an executable program that loaded tests and did all the launching and terminating of the program under test.  The tests themselves were MFC extension DLLs.  (Since MFC extension DLLs weren't designed to be loaded on demand, this framework had to do a lot of work to implement the ability to do the loading and unloading of tests.)

The tests didn't drive the UI of the program under test directly.  There were two layers of code in between the tests and the Visual Test DLL.  The lower of the two layers was made up of what were called "UI components".  These components were designed to provide an object-based wrapper of the procedure-based Visual Test DLL.

The second layer of code in between the tests and the Visual Test DLL was called the CO layer (CO standing for "component object").  This layer was designed to be a higher level abstraction of a particular feature (editor, debugger, project system, etc.) in the program under test.  Components in this layer called methods on UI components to do their work.

One of the main reasons for having these support layers in between the test and the program under test was to isolate changes in the program under test from the tests themselves as much as possible.  The idea was that tests, once written, shouldn't have to be rewritten when the UI of the program under test changed.  Instead, the changes would be isolated to a single place in the support layer and all tests would get updated to the new behavior without having to be changed themselves.  This worked pretty well in practice.  We were able to reuse our tests across multiple versions of our product with minimal changes related to changes in the product's UI.

There were some issues with this framework:

• Almost all of the testers that worked on and used the framework were also developers.  This was actually good in one way because the product we were testing was a developer tool.  On the downside, the framework sometimes became a testbed for "cool" development ideas that may not have been necessary and took time away from some of the required testing on the product.  Many of the people that worked hard on the framework used it as a resume item to move from testing to development.  (This may not really have been a bad thing in our product's case because it lead to a lot of "real-world" customer scenario work with our product while it was still in development.)
  
• Even though the framework was written in C++ it really didn't take advantage of all the object-oriented features of the language and the libraries it was built with.  In particular, inheritance and encapsulation weren't used as effectively as they could have been.  The lack of encapsulation in the implementation of the framework was particularly problematic.  Many support layer components made use of other components at the same level of abstraction.  When changes were made to one component they frequently had a "ripple effect" on other components.  In effect, the components weren't components at all but rather one monolithic block of code divided into multiple DLLs.

This first test automation framework had a lifespan of about 7 years.  (I was only involved with it for the last 2 of those years so I really didn't get to see the framework develop from the beginning.)  In the last years of the framework's lifespan, COM and ATL were being developed and many people were moving from MFC to ATL as their development library of choice.  Our product was also moving from the IDE that it had lived in all its life to a new IDE that would be implemented by a different team and shared among multiple developer tools.  Our test automation framework would have to be updated to target the new IDE which would involve significant changes.  We made the decision to abandon our old test automation framework and write a new one from scratch based on COM instead of MFC extension DLLs.

The second test automation framework was designed to be component-based and implemented in COM.  We didn't mandate a particular implementation library.  That led to some of the framework being implemented in MFC, some of the framework being implemented in ATL, and some of the framework being implemented with plain Win32 calls.  For many people on the team, this framework was their first experience using COM so many of the components implemented early on did not follow the rules or conventions of COM.

One of the benefits we thought we'd get by using COM was language independence.  To make sure we thought about this we wrote example tests in multiple langauges before we started using the framework for real testing.  We had tests in C++/ATL, Visual Basic, and Visual J++.  Later on when .NET became real some teams wrote tests in C# (.NET's COM Interop support made this possible).

Another thing we thought about early on was minimizing the locality of change.  In the first automation framework this had worked well in the support layer components (the part of the framework between the tests and the program they were driving).  In the second framework, we implemented the whole framework with that in mind.  This fit in well, too, with the fact that we were implementing the framework to be component based.  Some benefits that we got from adopting this approach:

• Some external teams that ended up adopting our framework were testing different programs than the one we targetted the framework at.  They were able to write support layers to drive the program they tested and plug them into the framework.
  
• One team that was using our framework had different requirements for running their tests in their test lab.  They were able to implement a new version of the program that launched tests in a different way than ours did and still reuse the rest of the framework.
  
• One team that used our framework did away with the support layer entirely and used the framework to implement API-based tests.
  
• One team testing a similar developer tool on a different hardware platform built a support layer to target their platfrom and didn't write any tests at all.  They used our framework, their support layer, and one of our feature team's existing tests to test their product.

The second framework was not without problems, though.  I already mentioned that the folks implementing the framework weren't really up to speed on COM and the implementation suffered because of that.

Another issue was that the framework was designed by a few test leads but implemented by everyone else on the team.  We didn't do a good job of building a shared vision of what the framework should look like and we also didn't have time to watch over everyone implementing the framework.  This led to inconsistent factoring of the components that made up the support layer.  For example, the component interface that drove the project system functionality had single methods to do the major things you needed to do in the project system.  You could call BuildProject() and it would handle everything and return you a result.  In contrast, the component interface that drove the hierarchical view of the classes in a project had much more atomic methods for doing things in that area.  You would have to call ActivateWindow() before you did anything with the feature, then you'd have to make single calls to open hierarchy nodes, then you'd have to make a call to bring up the context menu, and finally you'd make a call to do the thing you wanted to do.  Trying to stitch all these differently-factored components together in a test was not very intuitive or easy.  It made writing tests more difficult than it had to be.

Another problem we had was due to the fact that, in the beginning stages of framework development, pretty much anybody could party on the source code.  We didn't have coding standards, we didn't enforce code reviews, and we didn't have any concept of component ownership.  All that anarchy brought us to a state where the framework was not very robust.  We recovered by strictly enforcing coding standards, code reviews, and working to get owners of the various components.  This greatly improved the robustness of all new code and fixes that went into the framework but we didn't have time to review existing stuff that wasn't posing an immediate problem.

The second framework has been in use for 3 years now and we are still working to improve it.  Many of the changes needed to really do the right thing would require major architectural work and so they will probably not get done.  It did (and is doing) the job but I can't help thinking there is a better way.

So, I am going to try to implement the test automation framework that I'd like to see and report on it in future articles in this series.  Actually, the next few articles will be on some of the major requirements and design decisions for the Framework Project.  The topics I will explore will include:

• The Scope of the Framework
• The Technology Used to Implement the Framework
• Test Topologies
• Retargetting the Frameworkb To Test Different Programs
• Testing Distributed Scenarios
• Logging
• Test Automation Error Handling

I'm hoping that by doing the project this way, I'll be laying out the vision for a great test automation framework a piece at a time.  I am also looking forward to the fact that I won't be under any time constraints on when this will need to be done so the decisions I make can be considered carefully.

Although I have a general idea of how this will go, I can't predict the future.  I think the project will be exciting (well, for me anyway) and I like the idea that I will be documenting it so the experiences I have doing this won't get lost.

Note: due to changes in my job this series did not continue.  I have retained it on my web in response to requests.  -ronpih