A Test Automation Framework I'd Like To See - The Problem

Assume your job is to test the UI of a product.  This product runs on 4 different operating systems.  It is also localized into 9 languages.  You are required to run you tests against 2 of the languages and foreign subsidiaries are required to run your tests against the localized version of the product that they are doing the localization work for.  Those foreign subsidiaries have allocated 1/2 person to testing your product and that person must also verify localization.  During its lifecycle, your product will release 6 service packs and testing those service packs is outsourced to a team that knows nothing about your product (or the other 8 products they test as an outsourcing partner).  The operating systems that it runs on will also get some number (typically 5) of service packs applied to them, 3 of which will be released before your product.  Your product also ships in 6 different configurations (an enterprise edition, a professional edition, a standard edition, an academic edition, an edition that ships in books that teach folks how to program using your product, and a trial edition that expires after 90 days).  Management has decreed that every test case you have must run on every configuration that the product can run on at least once before you ship.

This is my world.  And now you know why I spend a lot of time thinking about UI test automation.

Another issue: since our product is a heavily used developer tool, there aren't really any third-party test automation tools that we can use for our testing. Typically, the vendors of those tools will release a version of their product that supports our product when we release our product.  We need to have all our testing done before that.  So, we must build our own test automation framework.

While watching our test automation efforts I have formed opinions about what is good, what is bad, and what an ideal UI test automation framework would look like.  I would like to try out my ideas in a real-world implementation of a test automation framework built according to my thinking.  But I can't do that at work.

Because of the heavy investment in both our current framework and our large body of existing tests, we can't construct a new test automation framework from scratch at work.  There just isn't enough time (and there isn't likely to be time for this in future).

It's also too risky.  Maybe I don't know what I'm talking about.  If we commit the large amount of resources necessary to do this in a timely manner and it doesn't work the way I envision it, well, it would be bad...

However, since my hobby is writing code, I can dedicate some of my own time trying to put together a UI test automation framework based on my ideas.  By doing it on my own time the stakes aren't as high and I can make mistakes that lead to learning without affecting other people who have work to do.  That's what this series of stories will be about.

Note: due to changes in my job this series did not continue.  I have retained it on my web in response to requests.  -ronpih