UK Tester Forums

There’s a mildly rollicking little discussion going on the in the Software Testing Club at the moment, in which Rob Lambert observes, “I’ve seen a couple of conversations recently where people are talking about red, green and yellow box testing.” Rob then asks “There’s the obvious black and white. How many more are there?” (For [...]

I have four or five blog posts in the hopper, each almost ready to go. I’m working on a whole book and a chapter of another one, and I’m on a deadline that I’m about to blow. The kids are still out of school, and I really should be cooking dinner right now. And yet… [...]

One of the most amazing things about working in the technology and software world is that this industry advances so fast that I have no real idea what software/technology I'll be testing on and writing about in 10 years time.

10 years ago I hadn't really heard of cloud computing, now I'm testing call centres hosted in the cloud. 10 years ago I was just starting to test on web apps, now it seems everything is available online.

Almost every article I read is about testing web apps, using web testing tools or some form of web based testing service. Combine this with the general mobile phone uptake trends you have a very mobile, very web, very technology heavy future. Add to this the lowering of barriers of entry (cost and availability of technology) and it wont be long before we are facing option paralysis when deciding which configuration to start testing on first.

As more people adopt modern technology, whether willingly or not, it's clear a wider array of technology and applications will be moving online and mobile providing us with numerous challenges of not only testing the functionality but also the human factors (or ergonomics) side of our apps and the devices they run on.

Self service (by end users) is also becoming a big selling feature and with mobile devices being just as powerful as many desktop devices it's clear we have a future dominated by mega massive selections of devices to test against, all running on varying degrees of network capability used in a mind melting variety of contexts. Add to that the seemingly natural way in which the future generations have perfected the art of managing multiple devices and we can quickly see how testing may need to evolve to keep up.

So my domain knowledge of the product and industry will need to evolve to keep up with technology and trends, this hasn't changed much. It's always been the case.

But also my approaches, test tools and ideas about testing will need to grow, shift and evolve too. But also too my test environments will need to evolve to keep up with the relentless progress of technology and communications. In fact, some test environments may just need ripping apart and rebuilding. And for some, this could be the biggest testing challenge they will face.

So how do you see your test lab or your test "approach" evolving over the next 10 years?

Image courtesy of : http://www.flickr.com/photos/levitateme

Permalink | Leave a comment  »

I was asked the other day how does Citrix improve response times. The simple answer is that it cuts down on the number of times the user has to wait for information to be transfered across the network. For example the diagram below shows an application on a client PC communicating with the DB. In this case it is Oracle forms 6i and this can take between 20-60 network hops to get the data needed to display a screen.

If you are on the same LAN as the DB then you may not notice the delay but it you access across a WAN then the time to cross the WAN all adds up to a slow response time.

With Citrix the citrix server is placed in the datacentre close to the DB. The client then makes one request to the citrix server, the citrix server makes all the requests to the DB and then when the screen is complete send a picture of the screen back to the user. As can be seen in the diagram below.

As the citrix server and DB are closely located then the network hops needed to get the data to build the screen happen quickly and the user has to suffer the delay across the WAN only once. Where applications cross the WAN many times and the delay from the users to the server is high then Citrix is likely to help improve performance. However, you need to also consider the bandwidth of the pipe between the client and the server.

To do this you can create a performance model. The following presentation contain I performance model I built for determining when to deploy citrix to various locations as part of a large upgrade project. In that project the application made 7 trips across the network to generate the display for the user. Pages 12-21 provide the model and some results.

You can download the presentation here.

Eric Jacobson has a problem, which he thoughtfully relates on his thoughtful blog in a post called “How Can I Tell Users What Testers Did?”. In this post, I’ll try to answer his question, so you might want to read his original post for context. I see something interesting here: Eric tells a clear story [...]

In a recent blog post, Rahul Verma suggests that all testing is confirmatory. First, I applaud his writing of an exploratory essay. I also welcome and appreciate critique of the testing vs. checking idea. I don’t agree with his conclusions, but maybe in the long run we can work something out. In mythology, there was [...]

I wrote an article for EuroStar last year about creating value with testing. Some of you have asked for more specific ideas about determining whether your testing is creating value or not. In the article, I talk about getting feedback from stakeholders, but that isn't always easy or possible. One of the most important stakeholders on any project is you, so how do you go about satisfying yourself with your testing value?

The easiest way to get feedback is from other stakeholders. What does your manager think about your testing? How about the programmers, business analysts and customers (users) of your software?

The hard part with that answer is you may not be able to talk to all of those stakeholders. Or, they may not know what good testing looks like so they won't have answers that satisfy you. In some cases, the stakeholders around you may have such low expectations that their feedback might not help you at all. They may expect you to provide testing work that you might consider shoddy and negligent. In that case, you have to show them what great testing looks like. When that happens it's like graduating from a cheap box of wine to the good stuff. Once they've tasted the good stuff, it's hard for them to go back to expecting poor testing.

Even if you have good direction from other stakeholders, I recommend asking yourself some questions to help determine if you are creating value or not. This is hard to do, and will result in work for you over the long-term, much like personal growth endeavours. Don't expect quick fixes, but if you work in these areas over time, you will see changes in your testing. Here are some things to think about:

• Is my testing work defensible? (Cem Kaner talks a lot about this.) Think of a court case. What would a jury think if you testified and described what you did as a tester and why. How did you determine priority? Why did you test some things and not test others? (100% complete testing is impossible, so you have to make decisions to optimize your work. Are those decisions well thought out, or more subconscious? What sorts of things might you be missing that you haven't thought of?)
• James Bach talks about how important it is to have thought out and varied approaches to testing. What kind of approach do I have to testing? Do I consciously choose to have a varied approach using as many models of coverage as I can to discover important information about the product? Do I make the best use of tools, testing techniques and management approaches that I can? Or do I just do what the programmers or someone else tells me to do?
• In the absence of getting real feedback from real people on my project, what would happen if a well-known consultant came to visit me? Could I answer their questions about why I chose to test this way? What kinds of holes might they spot in my thinking? Would they see weak spots? More importantly, would I be proud to have Cem Kaner or someone else I look up to see what I actually do? Have I used ideas from testing thought leaders in my work and found out what works well for me and what might not work so well? Could I communicate my work to an expert outsider clearly and thoughtfully? If so, what might they think?
• Do I adapt my test plans and strategies from project to project based on the risks and rewards our project environment has at a particular point in time, or do I just copy and paste what I did last time, and repeat the same thing over and over?
• Do I track down and find repeatable cases for important intermittent bugs, or do I just file them and forget about them?
• Do I feel energetic, creative and proud of my work as a tester, or do I just feel like I am doing the same boring things over and over and filling in paper work and forms to please a manager?
• Can I look at a released product and identify ways in which my testing has improved the product experience for our end users?
• Do others on my team feel better with me around? Do they miss me and my creative input when I am away, or do they welcome the break from my negativity? Do they request that I work with them on other projects?
• Is my testing service in demand? Am I the person team members come to when they need help solving a particular problem that I am really good at helping solve?
• Am I aware of other approaches to testing that challenge my favorites? Do I understand approaches that I may not favor or I may even dislike, or do I just dismiss anything unfamiliar and threatening out of hand? Do I have an open-mind and look to challenge my ideas in testing to help improve?
• Am I learning about different ways I could improve my work? Am I aware of recent changes in testing techniques and tools? Do I know where to find information to learn from?
• Do I consistently try to do better than I did last time?

These are the kinds of questions I ask myself regularly. I don't always have the best answers to my own questions, but as time goes on, I feel much more confident about both my own answers to those questions, and more importantly, the value I know my testing work provides.

As a community, we’re very guilty of using technical terms and confusing business users. If we want to get them more involved, we have to use the right names for the right things and stop confusing people. This lesson is obvious in acceptance tests and we know that we need to keep the naming consistent and avoid misleading terms, but we don’t do this when we talk about the process. For example, when we say continuous integration in the context of agile acceptance testing, we don’t really mean running integration tests. So why use that term, and then have to explain how acceptance tests are different from integration tests? Until I started using Specification Workshops as the name for a collaborative meeting about acceptance tests, it was very hard to convince business users to participate. But a simple change in naming made the problem go away.

By using better names, we can avoid a lot of completely meaningless discussions and get people started on the right path straight away. So here is what I propose:

One of the biggest issues teams have with acceptance testing is who should write what and when. So we need to come up with a good name for the start of the process that clearly says that everyone should be involved, and that this needs to happen before developers start developing and testers start testing, because we want to use acceptance tests as a target for development. Test first is a good technical name for it, but business users don’t get it. I propose we talk about Specifying Collaboratively instead of test first or writing acceptance tests.

It sounds quite normal to put every single numerical possibility into an automated functional test, why wouldn’t you do it if it is automated. But such complex tests are unusable as a communication tool. So instead of writing functional tests, let’s talk about Illustrating with Examples (thanks, Dave) and expect the output of that to be Key Examples to point out that we only want enough to explain the context properly.

Key examples are raw material, but if we just talk about acceptance testing then why not just dump all those complicated 50-column 100-row tables into an acceptance test without any explanation, it’s going to be tested by a machine anyway. But these tests are for humans as well for machines, so let’s talk about a whole new step after this, about the process of extracting the minimal set of attributes and examples to specify a business rule, and about adding a title, description and so on. I propose we call this Distilling the specification.

I just don’t want to spend any more arguing with people who already paid a license for QTP that it is completely unusable for acceptance tests. As long as we talk about test automation, there is always going to be a push to use whatever horrible contraption testers already use for automation, because it’s logical to use a single tool. Acceptance testing tools don’t compete with QTP or things like that, they address a completely different problem. Instead of talking about test automation, let’s talk about automating a specification without distorting any information – Literal Automation. Literal automation also avoids the whole scripting horror and using technical libraries directly in test specifications. If it’s literal, it should look as it looked on the whiteboard, it should not be translated to selenium commands.

After Literal Automation, we get a specification that can be checked against the code automatically, an Executable Specification.

We want to run all the acceptance tests frequently to make sure that the system still does what it is supposed to do (and equally more important to check that the specification still says what the system does). If we call this regression testing, it’s very hard to explain to testers why they should not go and add five million other test cases to a previously nice, small and focused specification. If we talk about continuous integration, then we get into the trouble of explaining why these tests should not always be end-to-end and run the whole system. On the top of that, for some legacy systems we need to run acceptance tests against a live, deployed environment. Technical integration tests run before deployment. So let’s not talk about regression testing or continuous integration, let’s talk about
Continuous Validation (or even Frequent Validation).

The long term pay-off from agile acceptance testing is having a reference on what the system does that is as relevant as the code itself, but much easier to read. That makes development much more efficient long term, facilitates collaboration with business users, leads to an alignment of software design and business models and just makes everyone’s life much easier. But to do this, the reference really has to be relevant, it has to be maintained, it has to be consistent with itself and with code. we should not have silos of tests that use terms we had three years ago, and those we used a year ago, and so on. Going back and updating tests is a very hard thing to sell to teams who are busy, but going back to update documentation after a big change is expected. So let’s not talk about folders filled with hundreds of tests, let’s talk about a Living Documentation system. That makes it much easier to explain why things should be self-explanatory, why business users need access to this as well and why it has to be nicely organised so that things are easy to find.

What do you think about this? Does it make sense? Will it help? Do you have a better name for one of these concepts, that explains it more clearly?

Donna Reed generously offered to organise a follow-up webinar to the one I did recently on effective specifications for agile teams. I’ll run through an interactive exercise of creating an effect map and then do Q&A on all the topics we did not have time to answer the last time.

The webinar is tomorrow. Register now

I came up with this yesterday while running my agile acceptance testing workshop for a client.

Sounds familiar? Read How to implement UI Testing without shooting yourself in the foot

The next meeting of the UK agile testing user group is on the 3rd of August in central London. Here are the details of the talk:

Dan Leong on Clean Acceptance Tests

This presentation discusses how our agile team renewed our focus and understanding of our acceptance tests when the team members changed. Our group found some core shared values in the context of acceptance testing which we expressed in the style of the agile manifesto. We then looked at our existing tests to find bad test smells that we could learn from. The whole exercise was a good experience and we encourage you to try something similar in your teams.

Dan Leong is a team lead at Sky Network Services, where they have been using agile/XP techniques for over 4 years to deliver the company’s broadband and voice provisioning system. He has over 10+ experience working in companies ranging from small .com start-ups to global advertising and media companies. Like the rest of us, he’s trying to figure out how to do things better.

The event is free to attend, but up front registration is required. Register now

When I use TestNG for my Selenium tests, I really like the BeforeSuite and AfterSuite annotations because then I can share a Selenium session amongst the tests. With nested JUnit suites I can achieve a similar effect, and I can also go some way to grouping my tests to make it easier to create browser [...]

The Times has an interesting article on investors’ attitudes to financial risk tolerance. Data suggest that investors’ perceived and actual risk tolerances can markedly differ, affecting the suitability of their investment portfolios.

So why’s this relevant? Well, as testers we always make risk-based calls on what to test and to what extent. We therefore need to understand how our perception of our own risk tolerance maps to that of the business so that these decisions are in line with what the delivery, and business, needs.

For a short while The Times has teamed up with finametrica.com who provide an online risk tolerance questionnaire. It’s obviously geared towards financial risk, but it’s well worth a look to see how your own risk tolerance maps to that of the adult population.