A big thanks to Paul Gerrard and Susan Windsor for organising yesterdays UK Test Management Forum in London. Also a big thanks to the sponsors for the event; SQS UK, Original Software and Tricentis Technology & Consulting

It was a cracking turn out with about 100 people there. Great to see so many familiar faces but also lots of new faces too.

Paul started the afternoon with news of the merger between his company, Gerrard Consulting, and Susan Windsor's company, WMHL. They are keeping the Gerrard Consulting name and are looking to expand in to the main stage of the consulting world. They both talked about a new tool called Testela which sounded interesting. Looking forward to finding out more when it's released.

BTW - Stephen Hill's blog is an excellent write up of the afternoon...here, so I wont do any more other than to include some photos.

There was a lot to agree with and a lot to disagree with about Jonathan Pearson's talk on "The Dark Side of Application Quality Management Ten Black Holes to Avoid For Successful Application Delivery" Some lively discussions took place around metrics and agile and many other topics. Jonathon also remained dressed as Obi Wan for the entire talk. Impressive. I must admit that their tool is looking increasingly like a tool capable of satisfying many needs within the testing community. It appears to be flexible, incredibly extensible and doesn't empose restrictions on how and what methodology or approach you can take. It's also got support for exploratory testing, which sounds very interesting indeed.

Second up for me was James Wilson. I worked with James many years ago and his topic of choice was very interesting. Why is testing in an Agile with Scrum environment hard? (And what can we do about it?)

It was one of the best discussions I've had for a long time. The presentation essentially got hijacked but there were some interesting ideas put forward about how to run soak tests in short sprints and how to manage the quality of your release, when time and cost are fixed. Good stuff.

As usual, some after conference drinks :)

 

See and download the full gallery on posterous

 

Permalink | Leave a comment  »

The 26th Test Management Forum took place on Wednesday 28 July 2010

Many, many thanks to the facilitators, delegates, the sponsors SQS UK, Original Software and Tricentis Technology & Consulting and of course our venue Balls's Brothers, Minster Pavement. for making the latest Forum a big success.

The web page for the event, and facilitator slides etc. can be found here.

Gerrard Consulting is delighted to announce its merger with WMHL Consulting.

After 25 years of successful trading, we are scaling our business with the aim of becoming a significant industry player. WMHL and Susan Windsor bring extensive experience in business development and service delivery. Steve Purchase, having 40 years as an IT professional with business and service experience joins the board as Non-Exec Director.

We will continue to provide our testing assurance and consultancy services but we will offer an exciting portfolio of new services starting with Tested Trusted Requirements and Improved Business Acceptance.

Susan says “I’m really passionate and excited about our new services and to be involved in scaling a service company once again”. Steve commented “In my view, requirements are the key to ensure that software fully meets user expectations, too often projects are late or functionality is somehow missed or watered down”.

WMHL brings us marketing, business development and leadership capability that complements Gerrard Consulting's reputation for innovation in the testing industry. We will market services to support businesses wishing to improve the way they manage systems, suppliers and software acceptance. These services are founded on our new Business Story Method and are supported by our test tool, Testela. This is an exciting time for the new company.

The full news release can be found here

read more

Inspired by a post by Michelle Smith about what's on her walls when she is working I thought I'd open something up to the testing community and see if we can't get some insights in to testers workspaces going. To continue the mantle from Michelle I've posted some pictures of my walls and home office. I work from an office at New Voice Media but when I'm at home writing blogs yada yada this is my inspiration.

Mine, it would appear, is not quite as neatly laid out or as decorative as Michelle's and I couldn't photograph the whole study as I have a bath side panel, some tiles and my DIY tool kits laid out...I must find some time to fix that bathroom.

I'm a big fan of the NBA. I'm not entirely sure how I got in to it, because here in the UK it's by no means a popular sport but I'm a fan none-the-less. I was also lucky enough to get to see a NBA match when I was in New York a few years back. It was excellent. The game itself was poor but the atmosphere was fantastic. Nicks versus Golden State Warriors......

Here in Britain it's got some core followers but it barely registers in the newspaper and you'd be hard pushed to find someone to talk to about basketball. Although you can't see in this picture, but the NBA image is made up of lots of little images of basketball players from the past. Also some modern art from my son :)

 

A map of the British Isles. I find it intriguing that many people in Britain are striving to explore the world and dismiss Great Britain as boring and dull. One look at this map and it's clear I haven't explored 99% of the British Isles so I can't join them in saying it's boring and dull.

And some books.

So why not upload your photos to flickr and tag them with "softwaretestingclub" so we can collate them together, maybe for a funky ebook. Maybe just as an experiment.

Permalink | Leave a comment  »

There's a growing number of people in the testing world doing something I've been calling Method Testing, although that's probably not the right term. Named after Method Acting (Wikipedia : Method actors are often characterized as immersing themselves in their characters to the extent that they continue to portray them even offstage or off-camera for the duration of a project.) I'm using this term to describe the testers who emerse themselves in their end users environment to learn how their customers use their product and in what way and on what devices. They then bring this back to the lab and test against it, often in situ, often with real end users.

In my opinion this is taking testing to a higher level than many can realistically go, but it all comes down to your goals, budgets and aims. Unfortunately, those who do Method Testing don't seem to write much about it..... In the examples I hear about it's not uncommon for the test team to prepare a "real life" environment (not just a real life computer configuration) in which to test, often using real end users, sometimes using actors and often using props.

This differs from UAT in that they are doing this as part of their day to day testing, not just at the end, when to be honest..it's often far too late.

A friend of mine tests medical equipment and it's associated software. He has a fake emergency room kitted out with dummies (in various medical emergency positions), medical equipment and costumes, he even uses fake blood. Proving the software works (or doesn't) is one thing but making sure it works under pressure, in situ and is usable is something completely different. Usability testing at it's best. He even employs actors once a week to act as patients adding extra drama and realism. They often bring in real surgical teams to run through a typical procedure. The whole development team observe the behaviour and adjust accordingly. He films the sessions to learn more about the actions performed in surgery and potential flaws with design. Sure it's expensive...very expensive, but it's invaluable for him. The software industry he is in demands working kit..all the time. I'm trying desperately to get him to write about it.

At the almost completely free end of the spectrum, another friend of mine tests call centre software and he has a small call centre environment set up with a seating plan in the corner of the office, so he and his team can role play a typical call centre scenario. He's not only testing the software, but testing in sudo-situ, with terminals, seats, wallboards and background chatter.

He suggests that it "tricks the mind" in to thinking differently about the software. I think he's on to something.

I know of several testers who test their software whilst out and about on trams, trains, at the beach, from a cafe etc on a variety of mobile devices using free wi-fi or shared connections. That's because they know roughly how and where their software is to be used. They are testing in situ. They are exploring, learning, feeding back. They are immersing themselves in the environments their customers will use their product in. They are doing method testing.  They are also extremely lucky :)

But what ties all of the above is that they are all reporting remarkable results in customer satisfaction <-- the ultimate feedback on product acceptability?

They are making an effort to emulate the end users real environment, to make sure the softwares original purpose is right, for the right people and under the right contexts.

But it's too tricky to find out who uses our software and how they use it..isn't it?

Not always. For a small few this might be true but in reality the majority of us just don't bother to seek out this information or we look in the wrong place. Why not try one or all of the following:

1. Speak to support desk or customer services. They are on the front line of complaints and feedback and can give you invaluable insight. They often maintain lots of records on system configurations and usage patterns. They don't stash it away from you to be mean, they probably just don't know you want it.

2. Speak to sales or accounts teams. They sell the stuff and maintain relationships with the customers. They will have more information than you would imagine.

3. Attend a sales pitch. You will learn loads, trust me. You'll also see the types of questions your customers ask about the product.

4. Attend (or give) a training session on the softwared to real users. You can observe them first hand and see which bits trip them up or confuse them.

5. For webapps or websites use Google Analytics or other website traffic monitoring tools or build in some performance/usage measuring. The amount of information Google Analytics gives you is insane. Browsers, screen resolutions, locales, referal sites, devices. You can find out bounce rates, pages navigated to/from, time on page, links hit etc etc etc etc Mind blowing stuff that really will make a difference to your test approach.

6. Talk to the customers....(might be worth checking with key people in your company first though :) )The majority of customers are more than happy to show you around their offices and show you the software in use, especially if it's not working too well!

7. Give customers easy and simple ways of reporting back problems from your software. Encourage them to be clear about the issues and log pertinent information.

8. Actively encourage customer involvement by hosting forums and chat rooms where they can discuss your product. Encourage them to be truthful and honest. Engage with them. You'll be amazed at how receptive customers are to honest and upfront conversations. Ask for feedback on what environments your product gets used in. Ask for the truth about your product. You can't blindly push on releasing if people are hating on your product. Or you can, but it won't be for long :)

The information is out there..everywhere. It's a case of widening your awareness fields and grabbing it. Analysing it. Using it. Reacting to it. Learn how your users use your product and you are gaining invaluable feedback on how to test it. It's never a replacement for real end users but it's a great addition to testing your software in a test lab with controlled environments and no idea about how your users use your software.

It might be time to step outside of this test lab comfort zone and do some in situ testing. Or should that be Method testing? Customer emulation testing? End user situ testing? No doubt someone will have a good name for it. I personally like method testing. And although your managers may think your mad for wanting to test in the same environments as your users, they'll appreciate it in the long run. There is, after all, some method in your madness.

Permalink | Leave a comment  »

The next meeting of the UK agile testing user group is on the 3rd of August in central London. Here are the details of the talk:

Dan Leong on Clean Acceptance Tests

This presentation discusses how our agile team renewed our focus and understanding of our acceptance tests when the team members changed. Our group found some core shared values in the context of acceptance testing which we expressed in the style of the agile manifesto. We then looked at our existing tests to find bad test smells that we could learn from. The whole exercise was a good experience and we encourage you to try something similar in your teams.

Dan Leong is a team lead at Sky Network Services, where they have been using agile/XP techniques for over 4 years to deliver the company’s broadband and voice provisioning system. He has over 10+ experience working in companies ranging from small .com start-ups to global advertising and media companies. Like the rest of us, he’s trying to figure out how to do things better.

The event is free to attend, but up front registration is required. Register now

I am very new to this forum and wanted to introduce my services to you in the best possible way so here goes-

I specialise in Contract Recruitment in the Software Testing Market.

For the moment I am offering free training on our InterSkill eLearning platform for anyone that registers a live requirement with me.

Once we have established a working relationship and have a contractor on site with you, I can give your team access to the platform which could potentially save you thousands of pounds and you could also use it to reward your staff.

If you want to deal with someone that knows the market and good contractors, then do get in touch as I know my network could help and access to our training platform could only have a positive impact on you.

I look forward to speaking to you all soon.

You can reach me on:
T: 01223 365197
E: mohammedali@computerpeople.co.uk

I've been a bit too busy on various side projects recently and realise I've not given this blog the attention I used to. Unfortunately the rest of the year is going to be mad too. So this blog will see fewer posts than normal..but I am still working on some new content for it. Here's why it's taking a back seat though:

 

Firstly I'm going to be taking a short break to get my Agile Testing Days presentation finished. I've got lots of research that needs collating and some more interviews to do with peeps in the testing community. The super cool organisers of Agile Testing Days also sent me a special speakers logo. How neat is that?

Secondly, I'll also be starting to ramp up my contribution to The Software Testing Club blog, which, by the way, you can all do too here.

Thirdly, I need some sleep :)

 

And for those who have not been in the loop with all things Software Testing Club then I should let you know we have officially disapproved this cheeky little ebook.

http://blog.softwaretestingclub.com/2010/07/ebook-by-cbpbos-building-a-test-team/

And we had nothing to do with it  :)

We've also started posting out some printed copies of the forthcoming Testing Planet (the Software Testing Clubs very own newspaper)

Some people will have already received the printed version and we are releasing the digital version very soon indeed. It's slightly later than expected, but hey....it will be worth the wait.

I must admit it looks amazing (I am biased though). Don't forget you can help support the work that the peeps here at The Software Testing Club do by purchasing your printed copy. Wink Wink.

Oh yeah. I'll also be at Eurostar later this year. Hopefully I'll get to see some of you there.

Rob

Permalink | Leave a comment  »

There is a discussion on the UK agile testing mailing list on driving the development of an administrative application with BDD. It illustrates a problem that many teams have, so I’ll post my response here as well.

Although we have a long way to go things are going OK at the moment and we feel it is bringing some real focus to the development process. However a lot of the early stories are largely admin type CRUD – for
example functionaility to set up user defined entities and their user defined properties within the system and to provide a mechanism for relating these entities to each other
…
Does anyone have any advice about how to write tests for this sort of stuff or any experiences in starting out with BDD they can share.

CRUD is not a user story, it’s a screen. It’s not a business function, but implementation detail. Why do the business users need a particular CRUD screen? (I know it sounds as a stupid rhetorical question, but I’m serious) What does it allow them to do?

Often you don’t need to implement an entire CRUD screen and deliver them one by one. Sometimes there is value in releasing two screens that allow you to set a subset of properties but together they bring value. You can then automate these tests through the UI and use the CRUD screens, but that will be hidden in the automation layer. Say for example that we have a risk report that lists users and their card numbers:

Scenario: Only regular customers with a specific risk category and 
              card type show up in risk reports
Given the following users
|name| type| card type| card number| risk category|
|Mike |VIP | Mastercard |53111 11111 11111 1111|X|
|Tom |VIP| Visa |41111 11111 11111 1111|Y|
John |Regular |Mastercard |51111 11111 11111 1111|X|
|Steve |Regular |Mastercard |52111 11111 11111 1111|Y|
Then the risk report for Mastercard and category X contains the following data
|John | 51111 11111 11111 1111|

This could, for example, invoke the user CRUD to save a user name, type and risk category and a completely different CRUD to save card details for that user. Any other information that would go on that CRUD (addresses, card expiry dates etc) aren’t part of this story or criteria because they are not important for this particular report.

Start with the outputs, the reports that the system produces, instead of data entry operations. this ensures that you have the data you need to produce the reports at the end, and that you don’t have superfluous data that nobody really cares about. if you don’t do that, the resulting data schemas are often overcomplicated and contain many things that simply aren’t used at the end at all.

The eagle eyed amongst my regular readers seem to have noticed that I am indeed from the future.

I posted my last blog post from the future, exact date: August 28th, 2010

I've been rumbled. Now....is the future dark or bright? http://www.developsense.com/presentations/e2008twofutures.pdf

That would be telling wouldn't it?

Permalink | Leave a comment  »

nFocus was lucky enough to attend the Microsoft Worldwide Partner Conference in Washington DC this year and there was loads of great keynotes, sessions and networking. We were even treated to a keynote by President Bill Clinton at the Verizon Center

Brad Brooks (Corporate Vice President, Windows Consumer Marketing) introduced some of the Personal cloud stuff and I have to say it sounds great. All your consumer devices interconnected with your own personal cloud, syncing seamlessly, it’s like live mesh on steroids! They demo’d Kinect™ for Xbox 360 which went down a storm with the audience but I’m glad I have mine on pre-order, can’t wait for November, this device really rocks!!

Brad Books – Personal Cloud

Well, we managed to find one chat on Software testing by the one and only Brian Keller (Technical Evangelist for Microsoft Visual Studio), he demo’d Visual Studio 2010 testing tools in a rather small room with a poxy little screen but still managed to pull it off – I just had to get a photo with him :)

Brian Keller from Channel 9

My personal fav, Kevin Turner COO for Microsoft, I always love this guy and this time round he did not disappoint. He attacked their competitors Apple, VMware, Linux, Oracle and Google and painted a picture of how Microsoft's strategy for mobile, desktop and cloud will allow them to beat their top competitors. Turner also described market share gains against Google, VMware and even Apple (picture below taken on my Apple iPhone 4)!

Kevin Turner COO for Microsoft

I have to say one of the highlights for me was the UK party, a much sought after event with people from all over the world trying to gatecrash (they did let Jon Roskill in though). It was a 007 Casino Royal theme and everyone had a great night although I wasn’t to sure why they had a Dame Edna Everage impersonator roaming round, that was somewhat surreal!

Danny Crone (Technical Director for nFocus) and Jon Roskill (Corporate Vice President for the Worldwide Partner Group) having a good time at the UK Bond Party.

So roll on WPC11 which is due to be hosted in Los Angeles…

The Unite union (of the let’s screw BA travellers for several weeks every few months fame) created a
Facebook group to stop jobs in the technology department being outsourced and offshored at Guardian News Media Ltd, publisher of the Guardian, the Observer and guardian.co.uk. According to the group web site, the board of Guardian News Media is meeting tomorrow to make a decision on outsourcing a large part of their IT department.

Without taking a position on who’s right or wrong in this case, I’m very interested in how this whole thing is going to play out. A recent major rewrite of their flagship web site is one of the most publicised apparently successful IT projects in the UK.

Amazon uses them as a case study for cloud computing.

Phil Willis spoke at several conferences, including Qcon London last year, on how applying domain driven design on this project helped domain experts to get involved in software development, and how they maintained a deep, malleable domain model, whilst meeting deadlines. This is one of the key case studies used by the Domain Driven Design community to prove that DDD works.

ThoughtWorks use them as a reference of how they help big companies implement agile processes. On the back of that project, they got to do the same with AutoTrader, ran by a Guardian subsidiary Trader Media. Their web site quotes Tom Turcan, General Manager for Digital: “A multi-million pound project running on time, and absorbing growth in scope – remarkable”.

A common thread here is that DDD, clouds, agile apparently gave better service more efficiently and produced more business value for the same investment. If the Guardian News Media board is now thinking of pulling the plug on all that, then that is seriously casting a shadow on those claims. Cloud computing aside, both Domain Driven Design and Agile development rely heavily on on-site close collaboration of business users and IT development teams. I guess a multi-million pound project running on time, and absorbing growth in scope, is less remarkable than the money they expect to save by sending the jobs abroad. Or maybe this is a confirmation that their new web site runs on its own and doesn’t need that many people to maintain it.

Update (2:30 PM) The Register picked up the story this morning as well

You've surely been in the same situation? You're sat using an app or site that requires some form of data entry from you, the problem is, you have no idea what it wants. There is no advanced information, communication or guidelines on what rules are going to be applied to your data. There's sometimes an ambiguous or tricky to find error message. Sometimes there's nothing.

As a software tester these types of bugs in our apps are low hanging fruit. They are quick wins, easy gains and an incredible booster of usability. I refer to them as "clarity bugs". Features, functions or capabilities that just need a little bit of extra "clarity" to understand them or make them work. This may come in the form of on-screen prompts or help files or documentation or maybe through training, but sometimes there just needs to be that little bit of something extra.

 

Image courtesy of : http://www.flickr.com/photos/rdolishny/

 

Help text that isn't helpful is pointless and confusing. None existent help is assuming a level of knowledge by the end user that might not be right. Errors that aren't descriptive are frustrating and time wasting.

To only explain what the user did wrong after they've actually got something wrong is wasteful and irritating. Communication is key.

When there is no explanation about what criteria the data and subsequent validation must meet it's is an easy usability defect to grab. (think about max and min lengths on fields, or password validations or date formats etc) Don't become blind by your pre-existing knowledge and sail straight past the fact it's not explained anywhere. Are we sure our end users know about our specific data validation?

It can end up with the user sat there screaming "what do you want from me?" whilst hammering on the keyboard hoping they submit the perfect data.

These usability bugs are bugs. Plain and simple. Even if the application works, if it's not usable then the chances are your customers will simply stop using it :(

Want to know some more about good form design and general usability?

Check out :

• ELMER - form design (excellent source of general principles of good form design) - http://en.wikipedia.org/wiki/ELMER_guidelines
• OR the actual ELMER guidelines (i.e. the suggestions and principles - this is very good stuff) - http://www.elmer.no/retningslinjer/pdf/elmer2-english.pdf
• UXBooth - http://www.uxbooth.com/
• Ten Usability Heuristics -  http://www.useit.com/papers/heuristic/heuristic_list.html
• 30 Usability concepts - http://stuffapproved.com/blog/2009/11/08/30-usability-concepts/

 

 

Permalink | Leave a comment  »

I don't normally post these types of blog where I include a screen shot of an error I've found in the wild on someone else's site. There's plenty of others doing that and with great effect. But this one really made me think.

On the Hampshire Library Search page there appears to be either:

• Left over test data

Or

• A lazy data ordering technique where everything no longer needed is prefixed with a Z to shift it to the end of the list. (why not delete it?)

 

Sadly though, some languages seem to have been discontinued (do the people who speak them know?) yet have double entries making the list somewhat confusing. To make matters worse the search form itself didn't appear to work. When searching for books in Winchester I get books from everywhere but Winchester. It's a reverse search feature.

How many of us go through the software once it's released to the wild? Would we have drilled down this low and spotted these issues? Do you think these issues were present in testing?

 

Permalink | Leave a comment  »

Creating an Executable Specification from existing manual test scripts might seem as a logical thing to do when starting out with Specification by Example and Agile Acceptance Testing. Such scripts already describe what the system does, and the testers are running them anyway, so automation will surely help. Not really — this is in fact one of the most common failure patterns.

The problem is that manual and automated checks are affected by a completely different set of constraints. With manual testing, the time spent preparing the context is often a key bottleneck. With automated testing, people spend most time on understanding what is wrong when a test fails.

For example, to prepare for a manual test that checks user account management rules, you might have to log on to an administrative application, create a user, log on as that new user to the client application, and change the password after first use. To avoid doing this several times during the test, manual scripts often reuse the context. So you would create the user once, block that account and verify that the user cannot log on, reset the password to verify that it is re-enabled, then set some user preferences and verify they change the home page correctly. This approach helps a tester run through the script quicker.

With automated testing, time spent on setting up the user is no longer a problem. Automated tests generally go through many more cases than manual tests, and when they run correctly nobody is really looking at them. Once a test fails, someone has to go in and understand what went wrong. If a test is described as a sequence of interdependent steps, it will be very hard to understand what exactly caused the problem, because the context changes throughout the script. The fact that a single script is checking ten different things also makes it more probable that the test will fail because it is affected by lots of different areas of code. In the previous example with user account management, if the password reset function stops working, we won’t be able to set the user preferences correctly. If we had ten different, smaller, focused and independent tests instead of one big script, a bug in the password reset function won’t affect the test results for user preferences. That makes tests more resilient to change and reduces the cost of maintenance. It also helps us pin-point the problems quicker.

Instead of plainly automating manual test scripts, think about what the script is testing and describe that with a group of independent, focused tests. This will significantly reduce the automation overhead and maintenance costs.

The fourth in a series of articles on Anti-Regression Approaches has been posted here: Part IV: Automated Regression Testing

Part I: Introduction & Impact Analysis can be found here.

Part II: Regression Prevention and Detection Using Static Techniques can be found here.

Part III: Regression Testing csan be found here.

The final article will cover management of anti-regression and sumarise the messages in this article series.

When I use TestNG for my Selenium tests, I really like the BeforeSuite and AfterSuite annotations because then I can share a Selenium session amongst the tests. With nested JUnit suites I can achieve a similar effect, and I can also go some way to grouping my tests to make it easier to create browser [...]

It was a pleasure to do the Agilista PM Webinar on Effective Specifications for Agile Teams today. This is the first time I did a webinar, so I’m sorry if it was a bit rough, but I hope you enjoyed it. Here are the links and slides.

• download slides
• Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing
• Effect Managing IT – also see my review of the book
• Exploring Requirements: Quality Before Design
• Sources of Power: How People Make Decisions
• Challenging Requirements (video)
• Other articles and resources mentioned in the webinar

John D. Cook recently issued a blog post, How many errors are left to find?, in which he introduces yet another silly quantitative model for estimating the number of bugs left in a program. The Lincoln Index, as Mr. Cook refers to it here, was used as a model for evaluating typographical errors, and was [...]

The Times has an interesting article on investors’ attitudes to financial risk tolerance. Data suggest that investors’ perceived and actual risk tolerances can markedly differ, affecting the suitability of their investment portfolios.

So why’s this relevant? Well, as testers we always make risk-based calls on what to test and to what extent. We therefore need to understand how our perception of our own risk tolerance maps to that of the business so that these decisions are in line with what the delivery, and business, needs.

For a short while The Times has teamed up with finametrica.com who provide an online risk tolerance questionnaire. It’s obviously geared towards financial risk, but it’s well worth a look to see how your own risk tolerance maps to that of the adult population.