The number 1 reason your test automation fails

Test automation should not feel like a nasty video game that feels like it has been created only to smash your spirit and the last vestiges of your team's willpower. Sadly, it often feels like this is the case.

Being at the coalface of software testing, and particularly test automation, we see first hand where the industry is getting it wrong when implementing test automation. Interestingly, for something that should be so intrinsic to modern, digital application delivery programs, most people who fail with test automation the same wrong things.

I think this is because the software testing industry as a whole has not been forced to innovate and, as a result, been forced to adapt to change. So when the "experts" moved from manual testing to automated software testing, they simply applied lipstick to a pig did the same things with a renewed flair. Such practice reminds me of a very famous line that is very commonly misattributed:

Insanity is doing the same thing over and over again and expecting different results.

Said by a very sage person

Why do I know this secret?

First, it's not a secret. It is an unfortunate consequence of a lack of effective and practical knowledge-sharing about how to move from manual testing to automated software testing. We do this from some of the world's largest companies and, frankly, we do it well; we deliver results, so I do know what I'm talking about.

You want proof? Well we recently finished the first round of automation for a customer in Europe where we cut their testing time for a major release down from 4 weeks to 36 hours! What could such results do for you?

Why are you automating your tests?

When discussing test automation with a customer we often rephrase this quesiton to, "do you NEED to automate your software testing?" Let's assume that you have made this decision for the right reasons, you would most likely identify with this finding of the World Quality Report:

If you identify with this significant group of IT decision makers then you are most likely in pursuit of a solution that allows you or application development effort to be more nimble and responsive to the needs of your business. Ideally, automated software testing should enable an application delivery process that helps the company achieve not just its IT-specific goals, but more importantly, it's commercial / business goals.

Check out our Ultimate Guide to Test Automation for IT decision-makers if you want to understand how to map your goals to an efficient automated software testing program that actually delivers results. The Ultimate Guide also includes a great comparison of the various automation engines and popular software testing tools.

However, it is never enough just to have your heart in the right place. Nice guys do win, but not because they're nice! The answer to "why are you automating your tests" may have been your original goal when your team began on its test automation journey. This goal, however, may have changed based on situational factors. Ask yourself, are your tests failing because the goalposts have changed, ie. am I trying to achieve something that I didn't design my test automation program to facilitate?

If your answer is no, then you will likely identify with the benefits of automated software testing that others in your shoes are trying to achieve.

Want to see how our testing tools & services can cut your testing time from weeks to hours?

Schedule Demo Now

The expected benefits of automated software testing...

...Are fairly consistent across most industries. The World Quality Report collects them into 6 distinct buckets and we find that our customers generally aim for all 6 of these benefits when partnering with us for test automation services:

We like graphs and numbers so we came up with our own comparison of the benefits of test automation over manual software testing. We've tried this on some of our customers and they agree that the "Green Zone" encapsulates all of the six benefits above:

The Green Zone also attempts to convey that reaping the benefits of test automation is not just about tools or infrastructure or processes or people / service providers. It's about ALL of these things! This is a whole new topic and if you're interested, check out my LinkedIn article where I discuss the confluence of these factors in relation to using test automation to battle the spiralling cost of software testing.

Identification of these benefits is important, because the most common failing of test automation programs is usually a victim of trying to achieve these outcomes. If your anticipated benefits are different and your test automation is not delivering, then talk to us for personalised help.

The essential components of automated software testing

Remember, any of these components in and of themselves are almost worthless. They must work together to achieve the final outcome that is important to you.

None is more important than the other and each element must be carefully calibrated to enable collective successes. The most common mistake that test automation programs make is to assume that automation will reduce their reliance on people. Test automation does not reduce your reliance on people. It increases your reliance on good team members, but it should reduce the overall size of your testing team.

The reliance on smart team members is brought about due to a couple of significant factors:

• The robots that will perform your test automation need smart people to tell them what they do in the first place; and
• The time savings through automation should be reemployed to helping you innovate further. For this you need very smart people.

The invaluable outcome of upskilling and redeploying your team to higher value tasks is often a missed benefit of test automation. The benefit is real irrespective of whether you outsource your software testing practice or you keep it in-house.

The interplay between the 3 P's is best explained in our guide to 10x Your Testing Effectiveness for Digital Applications. Download the 10x guide for the latest best practices for software testing. Explaining all that here will turn this blog post into a thesis (and I'll also need a pulpit!).

Need some personalised help about how you can fix your test automation or get it off the ground?

Get Free Strategy Session

So which component are you getting wrong?

Probably a combination of the three P's. But for a clue, I looked at the following graph from the World Quality Report and it left me highly perplexed:

If things like test data automation, robotics automation and test environment virtualisation are "emerging automation techniques" then our Qsome Technology Platform should have scooped every innovation award in the software quality space for the last few years!

The other USB plug that stood out in that graph (for the right reasons!) was "test design automation." At Qsome when we are automating tests for our customers, this activity incorporates a broad range of activities including, test design, test scope selection and scripting.

However, the one area that we find makes the biggest difference for customers who came to us after failing with their own test automation efforts is in test design. Why? Good question! The most common flaw in test automation efforts is the the practice of writing automated scripts for manual test cases.

What's wrong with that you ask? Another good question. There are two problems with this approach:

• Manual test cases usually cover well defined functions within your applications. When they cover inter-functional user journeys, the logic of the test becomes cumbersome and unwieldy, causing the automated test to fail at multiple points.
• Manual test cases are designed to be run once. Maybe 10 times. They are simply not engineered to be run hundreds, or even thousands, of times, causing them to fail often when automated as is.

If you are undertaking quality for a large application then you may literally be shooting yourself in the foot, to the tune of:

How do you fix your test automation program?

First ask yourself, are you automating the right tests? If you are satisfied that you have that right, then look closely at your test design processes, specifically the person undertaking automated test design.

I'm not suggesting you fire them, but inquire as to whether they have the know-how necessary to convert manual test cases into automated user journeys that can be executed hundreds or thousands of times by robots. If your answer is no, then you might need to find someone who can help you this crucial function.

Test automation is not just about speed. It requires intelligence, adaptability and much gumption. When you have those three characteristics in order then you can focus on getting these 10 things right!

If you need help in cutting testing time and finding more bugs before your customers find them or simply to fix your test automation program, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.

Get Free Strategy Session

Download Ultimate Guide

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com

Publisher: Ayush Trivedi - 11:04

How powerful Salesforce test automation can help you win

Most customers who have engaged us for Salesforce test automation can identify with at least one of these issues with manual Salesforce testing:

• A bad release that has cost lots of revenue, profits or reputations;
• Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;
• Apps built by different developers are causing conflicts in production; or
• Your or your team are copping heat because of poor performance.

In fact most of our Salesforce automated testing customers can also identify with this situation:

This situation is usually the result of stressed testers who are inundated with testing requests that they simply cannot complete with any reasonable level of accuracy or integrity. One consolation point is that this symptom is not unique to your Salesforce testing program alone.

The problem with manual-only Salesforce testing...

...comes down to three primary factors that are common across most manual-only Salesforce testing programs that we have analysed:

1. Time - comprehensive tests take days if not weeks. Who has that much time?
2. Coverage - even the most thorough manual testers can't test every nook and cranny in the time available.
3. Accuracy - with so many moving parts to every customised system, not even the most thorough manual testing program can adequately check for every possible regression that may creep back as the code base grows.

The folks that test the core Salesforce platform internally before it is released to their customers have this to say about their testing practices:

Here at salesforce.com, we use the open-source Selenium project to execute over 40,000 test cases against the features of the Salesforce application.

Salesforce Engineering Team

What's good for the goose has to be good for the gander, right? But what that statement above does not tell you is that Salesforce has built its own testing tool for which it uses Selenium as an automation engine. This allows it to remedy the three problems I described above when running its automated regression tests. I'll tell you how you can get access to a similar tool a bit later.

No matter how much you try a manual-only testing program for Salesforce will result in errors, inefficiencies, bugs creeping through to production and ultimately loss of reputations, for the entire Salesforce delivery team. A steady stream of such occurrences cause end-users to lose faith in the systems, leading to a vicious cycle of inefficiencies, ineffectiveness and the departures of otherwise good and talented professionals.

Size and complexity

Although they are often related, the reason manual-only Salesforce testing programs often end in tears is because even a stock-standard Salesforce implementation is a complex beast. However, most organisations customise their Salesforce environment with third-party apps. While this extended ecosystem helps to deliver a more workable solution for the organisation, this situation becomes a headache for Salesforce testers because the number and interdependence of elements that they have to potentially test goes well beyond the small matter of the 40,000 that the Salesforce engineering team described above.

The interdependence problem is a significant one. It is also one for which manual Salesforce testing techniques have no answer. A study of Salesforce administrators came up with a ranked list of their Salesforce headaches:

You will notice that points 3 to 11 in the graphic above lead to point 12. The complexity of third party integrations, the need for data-driven quality assurance, managing user roles all combine to make Salesforce one incredibly difficult application to test manually.

If you haven't yet understood my point, this additional graphic from the same study should help hammer it home:

Now you might look at this and be forgiven for thinking that testing Salesforce on mobile devices isn't really a problem here because it's such a small fraction of the overall access points. If you came to this conclusion then you couldn't be further from the truth!

Why? Because the people who access your Salesforce environment on mobile devices are usually your colleagues in the field. Salespeople, field workers, managers approving orders, etc. These people are the ones who rely on your Salesforce system to stay to connected with each other and more importantly, stay connected with their stakeholders, who are most likely to be your customers.

Also, that particular study was conducted in 2015. Current access rates of Salesforce systems from mobile devices will likely be many multiples of the figure above. Now tell me that it's ok to not test your Salesforce environment for mobile devices and tablets!

Another report from 2016 found this:

If you belong to the 36% above, then the complexity of your Salesforce system just skyrocketed. Just think about the interdependence of data and processes that now needs to be tested. If you continue to do this manually then two things WILL happen:

• The costs of testing and time it takes to test your Salesforce ecosystem will blow out as more manual testers will need to be hired; and
• You will not be able to keep track of all the bugs that need to be fixed or have been fixed or have resurfaced after a fix, leading to very frustrated and angry users who just can't get Salesforce to work for them.

I could go on, as I'm sure could you, but I am confident that you get the drift of why a manual-only Salesforce testing program is ineffective and inadequate.

Want to see how Salesforce test automation could save you weeks every time you test?

Schedule Demo Now

So if not manual testing, then what?

Let's be clear, I'm not suggesting that you should cease and desist from manual testing of your Salesforce environment entirely. In fact anyone who tells you to do this is either a pretender or high on something. Manual testing must be used smartly and in concert with a robust and flexible test automation program to really improve the quality of your Salesforce environment.

But be careful, because many of the products that purport to offer Salesforce test automation are really only good at cycling through screens. They don't actually do real regression testing. Many of these products are often considered "industry-standard", which is a reason why many IT decision makers harbour the view that Salesforce test automation is an expensive and ineffective investment. In fact, start to run in the opposite direction if you are told that your Salesforce test automation solution will utilise "record-n-play" technology to build your automated regression suite.

The guide to 10x your testing effectiveness makes this very astute observation, among other very on point best-practice tips:

Record and play does not cut it.

10x Your Testing Effectiveness

So what does smart Salesforce test automation need?

Any effective test automation program is a combination of three very important factors:

• People - use SMEs judiciously and at the right times to help your test designers and automation engineers
• Processes - don't spend time creating process maps, focus on creating robust regression tests
• Products - select a Salesforce test automation tool that is most suited to what you actually need, not what others think is "the best"

Read this guide to 10x your testing effectiveness to understand how these three factors can be combined to deliver a truly best-practice Salesforce test automation program.

Salesforce, like other complex ERP and CRM systems, has nuances that make test automation difficult. This is why you need to be sure that the people or organisations you select to build and execute your Sales automated regression testing suite have experience in overcoming these nuances. While none of the challenges posed by Salesforce are show-stoppers for test automation, only the best test designers and automation engineers have the know-how to build Salesforce test automation that will not be bogged down by an automated mess.

At Qsome we have built a specific framework that enables faster builds of automated regression tests for Salesforce. It was while building this Salesforce-specific framework that we created functionality to take care of asynchronously loaded elements, particularly within Salesforce's console app. This intimate know-how allows us to deliver value fast and often during our engagements.

Want to see how a framework built for Salesforce test automation could save your bacon?

Schedule Demo Now

OK, but how can Salesforce test automation "help me win?"

Winning is a very relative term. The context that helps you decide you need or want to win is just as important as the destination. For some people it's a job well done. For others it is the accomplishment of goals that were previously thought to be impossible within their organisation or among their peers. For other still, it might save their job or win them awards.

Whatever your motivation for implementing Salesforce test automation there are three outcomes that you must demand of your team or your service providers:

• Cut testing time from days or weeks to mere hours
• Increase test coverage and the robustness of each test, particularly with negative testing
• Provide test-related analytics that determines when you should release to production

While most Salesforce users are satisfied with even just one of the above outcomes, you will be settling for a sub-standard outcome and literally leaving money on the table if your Salesforce test automation program does not deliver all three of these outcomes. Without achieving all three of these outcomes you are not maximising the Green Zone:

Read page 4 of our Ultimate Guide to Test Automation for a brief business case of the ROI that's possible from an investment in test automation. The example used in the Ultimate Guide is based on a sizeable, but simple application. As we learned above, the size and complexity of a customised Salesforce environment makes it an application ripe for reaping huge dividends from test automation.

Implementing Salesforce test automation is by no means an easy task. If you need help in figuring out the right plan of action or how to go about implementing your plan, speak to us understand how we might be able to help you:

Salesforce Testing Services

Request Demo Now

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com

Publisher: Ayush Trivedi - 17:51

Powerful automated software testing metrics & why you need them

Automated software testing metrics by themselves are not powerful. It is the smart use of these metrics that makes them powerful and gives you the ability to transform your software quality program. Unlike Qsome's test automation tools, most testing tools do not offer test-related analytics. The analytics that are built into our automated testing tools are one of the differentiators that sets us apart from our competitors.

Elfriede Dustin's book, Implementing Automated Software Testing, is considered gospel for those wishing to put in place a set of metrics to assess the effectiveness of their automated testing program. However, the pace of change in software quality requirements and new development methodologies have necessitated a need to prioritise some automated software testing metrics over others.

At Qsome, there are two types of metrics that help convey very different messages:

• Metrics to use to assess automated test preparation; and
• Metrics to use during test execution to help with delivery-related decision making.

It's important to be SMART

Whatever metrics you choose to follow as part of your automated testing program, apply the SMART principle to make sure that you don't waste your time. Applying this elegant principle helps to reduce errors and forces your team to focus on only those metrics that add real value to the analysis of your testing program.

I find that the part of this acronym that people most struggle with is "time-bound". In the metrics that we track with our testing tools, time-bound takes two forms:

1. The time period over which that activity is being tracked; and (more importantly)
2. A comparison of the result over different time periods.

Applying point 2 to all analyses of your metrics will help you quickly understand where your automated testing program is ripe for optimisation. This is where automated software testing statistics go from mere metrics to those of the powerful variety.

Metrics to assess test preparation & test worthiness

As the guide to 10x testing effectiveness emphasises, it is very ineffective if manual test cases are simply converted to automated test cases. To work out Percent Automatable make sure you convert your test cases into user journeys before calculating how many can be automated.

While it may be possible to automate every journey, it may not be the most effective or efficient use of resources to do this. My teams aim for a percent automatable metric of over 90%, especially for web-based and mobile app testing. Depending on your application architecture, it may be excessively cumbersome and unnecessary to automate the testing of setup or initialisation journeys.

Automation Progress refers to the number of user journeys that have been automated, from those that were deemed automatable at the outset. This is an important number for customers who want to understand how their automated testing service provider is progressing with building the automated regression testing suite.

Great decisions are made with good data. Track your own software testing metrics with our dashboard template.

Get Dashboard Now

Metrics to assess test progress and success

The 10x testing effectiveness guide focuses a lot on how to make your software testing program faster and more effective at catching bugs before they get to production. Increasing the effectiveness of anything, particularly software testing programs, requires participants to conduct a series of small experiments, track key metrics and iterate after changing one or two elements to assess the impact of the said changes.

Now that you know what to do with these test automation metrics, the first one to track is very simple, but very few teams actually track it: number of defects in production. This overarching statistic should be tracked for every version that is released to production. Plot it on a graph and if you see a downward trend, then you know that your software testing program is actually working. If your graph resembles more of the latter parts of a J-curve or has flatlined, then you need help.

The percentage of defects resolved is a very simple statistic that tells you the workload remaining in a sprint before the application needs to be tested again prior to release. Ideally, that number should be zero before a version is released to production. It's not rocket science!

An advanced version of these two metrics is defect density. This is the percentage of all defects contributed by each module that is being tested. Because we create automated tests based on user stories, this can be a little more difficult to measure, but it will provide great insights into which parts of your development process may require greater emphasis on quality.

Two metrics that we ask our customers to focus on prior to any release are percentage of tests passed and test coverage. These two are quite self-explanatory but must form the basis on which all application delivery decisions are made. If both are not above a pre-agreed threshold, that release does not make it into production. Quite simple really!

For large applications like ERP and CRM systems, test coverage can be split into the coverage of high and medium-risk user journeys or coverage of only those user-journeys that that are affected by any code changes made in the current sprint. To accomplish the latter you require your automated software testing tool to be able to automate test scope selection.

The list of metrics that can be tracked is literally endless. The question you have to ask yourself is whether it is necessary for other metrics to be tracked within your software testing program.

How will you track these automated software testing metrics?

Knowing what to track is one thing, but executing the measurement process without burdening your software testing program with loads of extra resources, inefficiencies and costs are also important. Ask your testing tool vendor what metric tracking capability is already built into the tool.

You will likely find that very few automated software testing tools have metric tracking included out-of-the-box. Qsome's testing tools do include critical test analytics to drive the software delivery process. This is one of the reasons that our customers have switched to us.

Download the Definitive Guide to Powerful Software Testing to understand how you can slash testing time and improve your software quality. If you prefer talking to reading, speak to us about your goals and challenges:

Get Testing Dashboard

Get Best Practice Guide

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com

Publisher: Ayush Trivedi - 18:56

Potent test automation services for web & mobile applications

So you're on the hunt for test automation services for your web or mobile app, right? But what is it that you need, exactly? Have you defined your requirements or are you willing to be persuaded by the glossiest marketing brochure?

There are thousands of test automation service providers willing to grab every last dollar off you. But there are a few things that you need to be sure about, before signing long term (or even short term) agreements with these service providers that put your professional reputation on the line. In fact, there are 5 key decision points you must consider before you make this decision.

Understand the context

What is it exactly that you're putting at stake with this decision? Is it the quality of applications that allow your customers to spend more money on your goods or services? Is this an application that your team members or vendors use to make sure that you are actually able to service your customers? Essentially, you need to understand the size and ramifications of the opportunity before you can effectively decide on the most suitable test automation services provider.

Most people or organisations in your shoes look for external help with test automation because they identify with at least one of these factors:

• A bad release that has cost lots of revenue, profits or reputations;
• Poor quality practices aren't an issue right now, but will become a huge issue if the status quo isn't changed;
• Upgrades are long delayed because quality can't be tamed; or
• Your customers are switching to the competition because your application has too many bugs in production.

We know this because of two reasons, a) our customers told us about these problems when they started using our test automation services, and b) all of my colleagues in our leadership team have faced these issues in our previous lives. Boy, do we have some embarrassing stories on this front!

You are definitely not alone in facing such issues. You're also a step ahead of the competition because you are doing your own research about the best solution for you, rather than jumping into the arms of the first vendor who tells you what you want to hear.

Your decision should be influenced by 6 key factors that go to the heart of every good relationship between an organisation and its test automation services provider.

Testing tools

Do you have existing testing tools that you are mandated to use? Are you prepared for your testing services provider to recommend and use tools that may be a better fit for the web and mobile applications that need to be tested?

The issue of automated software testing tools is central to any such engagement. Choose the wrong tool and you are effectively building many future inefficiencies and frustrations into the project. By selecting the automated testing tool that is best for your needs you will lay the groundwork for all parties to be more effective in performing their role and you will should also be able to reduce costs during the engagement.

Remember, previously popular tools provide the greatest incentives to service providers, but they haven't kept pace with testing needs of fast-paced digital environments that are designed to meet very exacting customer standards. To test digital web and mobile applications you require functional testing and load testing tools that are built for this purpose and will scale with your development needs, without requiring you to blow exorbitant amounts of money just to keep their infrastructure alive and up to date.

Our Ultimate Guide to Test Automation has a comparison of testing tools that you will find helpful. The guide to 10x Your Testing Effectiveness has best-practice tips that you must insist on using during your engagement. Here's a comparison of the most popular testing tools from the Ultimate Guide:

Objectivity in decision making

Presumably you want test automation because you're looking to cut testing time. This is very achievable and indeed something that any self-respecting test automation services company should be able to give you. However, the additional benefit of test automation is that it throws up a number of informative and unimpeachable test-related data that you or your service provider should use when making quality and delivery-related decisions.

For example, how do you currently decide when a version of your application goes into production? Most testing programs usually require a test or delivery manager to make this decision based on their assessment of quality readiness.

However, test automation activities, when performed with the right test automation tools, enables you to build quality-related analytics into your delivery decision making processes. These anayltics make possible the transformation of an inherently subjective, role-oriented decision to an objective, data-driven decision where an individual is never a bottleneck.

Want to see how our test automation tools could save you weeks every time you test?

Schedule Demo Now

Engagement models

Engagements with test automation service providers have not always delivered the value that customers wanted and expected. The reasons for this vary, but usually revolve around misaligned priorities, capability gaps, technology gaps and poor communication by all parties. In fact this could be true for all aspects of IT outsourcing engagements!

Time and materials was a great model for the service provider of eras past. It provided an endless flow of cash and constantly shifting goal posts meant that accountability could never really be demanded by the customer. You should avoid getting stuck in this vicious cycle.

We almost only engage with our customers on fixed-fee and fixed-scope engagements, where everyone has skin in the game and the customer's downside risk is contained while the upside can be maximised. Most capable test automation service providers will also offer fixed-fee engagements. If they don't then you need to seriously think about why this may be the case.

Adaptability and integrity

Let's face it: even with the right testing tools and a great engagement model, things will go wrong in a project. Your applications could throw up nuances that will stump even the best automation engineers; user journeys might change after feedback from end-users; there may even be some bugs that are so difficult to fix that timelines suffer blowouts.

The key to overcoming these obstacles is communication and transparency. To inculcate these principles into our dealings with customers, we don't provide weekly or monthly status reports in PDF or Excel. Instead we use the real-time reporting tools in the Qsome Technology Platform to demonstrate and analyse project progress. The fact that this information is shared in real-time and that our test engineers see exactly what our customers see, we are able to build a greater degree of trust with out customers earlier in every project.

However, not everything can be communicated through a dashboard. Sometimes we have to put up our hand and admit to something that we were trying not resulting in the desired outcome. I find that this transparency with the customer encourages greater sharing of ideas and solutions, which in-turn delivers greater value to the customer.

You must assess whether your test automation services provider is willing to be honest about the project's progress, even when the going gets tough. This is the difference between a truly successful customer-vendor partnership and one that produces mediocre, or even sub-standard, results.

So what do you actually need?

This is the most important question after all, but you cannot answer it without completely responding to the 6 decision points above. There are really only three types of services that any test automation services provider can offer you. The exact service you need right now depends on how you have responded to the decision points above.

1. Advice or consulting: a root and branch review of your process from development to delivery. This exercise is often useful prior to engaging a test automation services provider because it allows you to better understand the other elements of the puzzle that can feed into making your test automation efforts a success.
2. Test practice development: this is where your service provider builds you a fully functioning software quality program right from processes, to people, to technology, to actually designing, automating, executing and analysing tests. These services can be a combination of in-house and external elements depending on your goals and priorities.
3. Test automation services: I call this a "build and let us run" approach where the test automation service provider develops the automated regression suite and leaves it to the customer to execute as and when required. There may be some ongoing commitment from the services provider for maintenance of the regression suite, with a very limited scope.

Download the Ultimate Guide to Test Automation to better understand how you can implement a test automation program that slashes testing time and improves software quality. If you prefer talking to reading, speak to us about your goals and challenges:

Contact Us Now

Get Services Delivery Guide

Or call us on +61 3 7001 1430 or +44 20 3411 4974 if you're in the UK, or email solutions[at]qsometech.com

Publisher: Ayush Trivedi - 15:00

5 unexpected reasons why outsourced test automation fails

Failure isn't fatal, but failure to change might be.

John Wooden

And therein lies the crux of the problem with many outsourced test automation services providers. The ability and willingness to acknowledge, analyse and learn from mistakes, both your own and those of others, is a very noble and productive human characteristic, but sadly and uncommon one.

Far too often we receive mayday calls from prospects who have invested a lot of money, but more importantly time, into selecting and working with a outsourced test automation services provider, only to be rewarded with a never-ending tug of war that inflicts much damage to their professional reputations. At the root of these frustrating and poor outcomes is a test automation service provider who was either out of their depth, unwilling to own up to failure and probably did not fully understand the customer's desired future state.

In the case of outsourced test automation services, most customers are trying to prevent losses of the magnitude reported by Computer Weekly magazine from the UK:

One analyst, who follows the outsourced software testing space closely, believes:

Cost savings are a major driver and also the need to do testing in a more professional manner.

Dominique Raviart, analyst at Nelson Hall

While he is definitely on the mark, it is obvious to us that the first desired outcome often drives a disproportionate share of decision-making when selecting outsourced test automation serviced providers. This only serves to diminish the importance of and attention to actually conducting "testing in a more professional manner." In fact, as far back as 2007, Gartner predicted:

Although the statistic above related to the BPO industry but our experience tells us that this would also hold true for outsourced test automation engagements. Gartner research director Alexa Bona put that statistic down to high staff attrition rates suffered by many outsourced service providers. If you think about it, there are no free lunches, right? If you pay peanuts, you get...

While some test automation services outsourcing engagements fail because of technical reasons, it is our analysis and that of other storied industry veterans, that most fail because of misaligned priorities, inadequate discovery and the IT services industry's propensity to try and fit square blocks in round holes.

At Qsome, we ask a lot of questions up front before agreeing to work with a customer. This process of understanding one another is an important step in building confidence and assessing whether the relationship will be one of mutual wins. We have seen that the following five questions are often not asked or not understood during the "courtship" phase between a customer and the test automation services provider. Its almost never as simple as just these questions alone, but they are a great starting point if you are trying to figure out why your test automation services provider just isn't delivering.

The 5 often forgotten reasons outsourced test automation engagements fail are a set of questions that you should ask yourself before your outsource your next engagement. Answering these questions for yourself even if the outsourcing service provider does not ask you these questions will help give any engagement a greater chance of success.

1. What are you trying to achieve?

As an outsourced software testing service provider, this helps us quickly prioritise wins from a potential engagement, as the customer sees them. To the earlier link between the predominance of cost saving motives in failed engagements, answers to this question also help us understand if the customer is looking for fast and effective test automation services that really kick goals, or just an improvement to their cost control target for KPI assessment.

By no means is attempting to achieve cost savings an unworthy goal. However, if the customer is only interested in cost savings then there is little on which to form a lasting and mutually fruitful relationship.

If your software testing outsourcing partner has not asked the overarching goals for engaging, then you should explain these goals and tie progress evaluation to the achievement of these goals. Mutual and detailed understanding of the desired future state is the bedrock on which your software quality will flourish or flounder.

The future state should define quantifiable and measurable metrics covering software quality, internal and external resourcing, timeliness and investment targets.

Want to see how our testing tools & services can cut your testing time from weeks to hours?

Schedule Demo Now

2. What are your expectations of us?

The desired future state you described above is only going to be achieved if the combined labour of the customer and the outsourcing service provider produces the desired results. Just because you outsource something, does not mean you are absolved of all responsibility and accountability.

We ask this question to understand two things:

• How hands-on the customer wants to be throughout the project; and
• How the customer views their involvement in the project changing over time.

The answers to this question also help to clear many of the unknowns and misconceptions that are very common in outsourcing engagements and lead to increasing distance between both parties. You can minimise, if not eliminate, the "crossed wires" and "we thought you were taking care of that" moments by clearly understanding the roles each partner will play throughout the engagement.

This clarity can be a big boon in engagements where customers want us to develop a test practice for them and leave them to run it. It goes without communication that overall outcomes are significantly improved through this greater degree of communication.

3. How important is this goal in the big scheme of things?

This answer really tells a story about customers' priorities. Are your non-financial priorities at least on par with your cost saving goals? Are you really serious about finding bugs before they get into production or are you more concerned about shifting risk?

Prioritising these goals will help you make a more informed decision about your preferred outsourced testing provider. Some outsourcing partners will be happy to simply help you achieve your cost targets, whereas others may decide that they would rather concentrate on projects where cost reduction is one element of a good outcome.

It is a horses for courses world, so make sure you communicate your goals in prioritised order so that you can attract the outsourced test automation partner that is best aligned to what you are really trying to achieve.

How do you ACTUALLY know if your software testing is working? You must have a dashboard to easily track key testing metrics.

Get Dashboard Now

4. How will we talk to each other?

Phone, email and Whatsapp are all relevant and useful modes of communication. Many stalwarts of this industry also produce very fancy looking PDF or Excel reports prior to every progress meeting. But, how do you really assess project progress?

To remove this information asymmetry we developed a real-time dashboard as part of Qsome's set of functional test automation tools and load testing tools. This dashboard is not only the central source of truth for project progress, because our test engineers see exactly what our customers see, but it also helps our customers make highly objective decisions about if and when they release to production. This removal of subjectivity between both parties and within both parties is a critical trust-building exercise.

By eliminating any doubts about the veracity of the information being discussed, we enable deeper, more frequent and more productive discussions about how we can achieve the customer's desired future state even faster.

Ask yourself, how could objective and accurate real-time data exchange between you and your outsourced test automation partners improve the results that are delivered by your team for the organisation? Think about it.

5. Have we understood you properly?

The clincher! All the thinking and explaining and strategising you have done to date are of no value if you and your outsourced test automation partner are not on the same page.

A simple technique that we ask our team to practice during sales meetings and progress review meetings is to explain what they think they have understood in their own words back to the customer. Nothing promotes clarity and consistency like repetition.

Any idea, plan, or purpose may be placed in the mind through repetition of thought.

Napoleon Hill

If this question does one thing, it is that it quickly brings unrealistic expectations to the fore. By repeating your goals and expectations aloud back to you, your potential outsourcing partners will be able to quickly decide if they are really attracted to you or if they just want the money. If it is the former, then you may just be on to a good thing.

If you picked up a strong whiff of communication being the central theme behind these questions, then you are on point. Good communication can be an abstract concept to those who miss the forest for the trees. In fact, you know communication is non-existent if someone believes they have communicated well with their outsourced test automation partner because the "legal agreement is watertight!"

Let's be honest, if any commercial relationship devolves into both parties quoting from the contract, you may as well close the door on achieving any goals. While this is an unfortunate fact of life, you can minimise the chances of suffering this fate if you and your test automation outsourcing service provider understand your answers to these key questions.

If you need help in cutting testing time and finding more bugs before your customers find them, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.

Schedule Qsome Demo

Download Ultimate Guide

Or call us on +61 3 7001 1430 or +44 20 3411 4974 if you're in the UK, or email solutions[at]qsometech.com

Publisher: Anonymous - 15:24

1 powerful idea that made our software testing services awesome

Our customers turn to our software testing services to help them transform their software testing programs from cost centres to value generators. If we followed the same principles that the industry has been using for decades, we would never deliver the lofty outcomes that our customers seek. Instead we turned our practices upside down and created some overarching principles that are non-negotiable in every project. One such principle that helps us deliver this value consistently and across projects is the concept of team-owned tests.

Don't worry if you read that and thought, "WTF!" That is the most common reaction from test design and test automation candidates who attend our interviews. It is our version of XP's collective ownership principle.

What is the problem we’re solving?

Many symptoms contribute to the problem of lengthy, onerous and ineffective software testing cycles that seem more like a burden or an annoying cousin to be locked in the spare room, than the good looking new boyfriend who is to be shown off to friends. In fact, I think the waterfall concept gets unfairly castigated for giving software testing the bad rep it currently seems unable to escape from. It's a combination of a number factors that are having a devastating impact on an economy-wide level:

The activities that have got the software testing profession and function to this point stem from hardware testing. Developing, testing and maintaining hardware is intrinsically a more static exercise compared to testing software. More so now that cloud, agile, devops, etc are hellbent on redefining the concept of speed and efficiency in the software development pipeline.

The very nature of modern software design and development renders traditional software testing methodologies somewhat akin to what propeller-driven planes are to the airline industry. User journeys in software development change often in response to evolving user demands, adding to software testing costs and pressure. The technologies and methodologies (think DevOps and CI) used to build these newfangled applications add an extra layer of complexity to a tester’s task and the costs of testing an application, to the extent that:

Also, and very unlike software development, software testing is undertaken by many smaller teams where the test and test script is often designed, scripted and executed by different people. Conversely, in most development teams a single developer is responsible for creating a function or even a module within an application. Therefore, traditional human models about “ownership” and “only I have to know what’s going on under the hood” are about as relevant as a keyboard is to Siri or Alexa.

Essentially, the problem we are solving with the concept of team-owned tests is that traditional methods of executing software testing only result in wasted time, lots of bugs in production and shredded reputations in this day and age. The traditional methods of software testing do not allow easy modification, maintenance or, often, even execution.

If your software testing (particularly automated regression testing) takes an age and still leaks bugs to production, it is probably suffering from this problem.

The difficulties in changing mindset & culture are huge

You will agree that asking a proud and talented professional to unlearn a very natural human trait such as “this test is mine” is a very difficult task. In fact, the problem of culture was referenced in the latest World Quality Report via quote from this very astute professional

The number one challenge is finding the skilled people.

IT Director, Pharmaceuticals Company in Ireland

To overcome this cultural issue that permeates the entire software testing industry the world over, we had to create some very specific structures within our teams to enable the implementation of team-owned software testing. So we now test the understanding this principle as early as the first interview. Some people understand the concept before the explanation has left our lips. Most take a little bit of prodding and require examples to grasp it. About 20% of our interview candidates find within the first few minutes of the technical interview that they are better off looking for jobs in a different company.

You will find that your software testing teams probably display a similar distribution. However, you will find a similar problem to the one GE encountered when implementing Jack Welch’s philosophy of bringing the middle band up to the “stars” band. This is not just a change of personal habit, but a cultural shift that must be espoused by every team member, else it makes little difference in transforming software testing from a cost centre to a value creator. Download our best practice guide for digital application testing to understand how your team can also transform software testing into a value creator for your organisation.

We have found that peer reviews of test design, test scripting (where automation is concerned) and test maintenance activities help to propagate this new mindset. Peer reviews are not the magic bullet to overcoming this challenge, but every law needs a sheriff to enforce it, and our take on extreme development (for software testing) is delivering the desired shift.

Want to see how our software testing tools & services can cut your testing time from weeks to hours?

Schedule Demo Now

But, what is team-owned software testing in practice?

Great question. To help our folks deduce if they are implementing this principle in their work, we recommend that they ask themselves one big question:

Can the person next to you understand your test / script / idea without you explaining it to them?

Qsome principle #1 for team-owned tests

One of the ways we build this self-assessment into our test creation pipeline is by splitting the test design and test scripting processes for automated software testing. If the person scripting the test cannot understand the designed test then that question is automatically answered in the negative.

This question also assumes that the person coming after you has no knowledge of the system under test. For obvious reasons, this assumption is not feasible for totally out-of-the-box applications under test, but we find that it remains relevant for most applications that we test for our customers.

The achievement of this goal in software development often results in us users triumphantly labelling an application “intuitive.” Similarly at Qsome, we aim to build intuitive manual and automated software tests that can not only transcend our internal team boundaries, but also the silos in which most of our customers' teams operate.

Then what?

As a company, we are never short of questions. So, after the Big Bertha of a question that gets asked up front, we add a left and a right jab to really start probing:

Have you placed only logic and the right logic in your scripts?

What are you having for dinner tonight?

This is not a joke. The first question is unquestionably a very serious and pertinent one, given that we believe in thorough and robust software testing that actually uncovers bugs instead of acting like a revolving door that allows anything to be released to production.

At Qsome, we are VERY big on test automation. We automate early and often, to the extent that we our default thinking is that the testing of most digital applications can be significantly automated, until a posse of insurmountable obstacles reveal themselves from the shadows. To make this philosophy work, we prefer to use decision trees to support our data-driven testing framework. Our test automation engineers know that the stress must be placed on the data set, rather than the automation script. This is not an easy task because it is generally not the way that their minds have been wired to work throughout their education.

The odd thing about dinner is that you can only have it if you have the time. If your answer to questions 1 and 2 is not “yes,” then you may just have to skip dinner and head straight to a hearty breakfast, because you will be using that time to fix your work.

My thoughts above about dinner and breakfast are only half in jest.

Our customers have reduced bugs in production by over 70%? Want to see how?

Request Demo Now

Why are you so demanding of your testers?

Wow you really do like speaking truth to power, huh? That is probably also the question that abounds in the minds of the 20% of interview candidates I introduced earlier. The thing is, we really don’t see our approach in that light.

We started out with one very important and achievable goal: to transform software testing from a cost centre to a value generator. There are few ways to achieve this goal unless our teams change the way that software testing is performed.

A crucial obstacle / reality in achieving this goal is the fact that software tests must be as robust and dynamic as the applications that they are built to test. Additionally, modern IT functions are increasingly distributed, outsourced, remote or many other buzzwords that simply mean that your software tests are definitely going to be handled by many different people.

What is the worth of a software test if it requires a dinner date over fine wine to be understood by a person other than she who wrote it?

In football (of the European, not Australian or American variety) the players practice for weeks on end to develop a telepathic understanding of their positioning on the pitch or decision making in pressure situations. For reasons good or bad, modern IT teams, especially in large organisations, have no such luxury. They must get it right the first time or sh*t hits the fan and there’s never enough disinfectant to clean up the mess it makes of some reputations.

How do you know that this has made your software testing services better?

At work, I don’t believe in making a rule that does not have a firm and achievable goal in mind. But the real results that this principle helps achieves will vary depending on who you ask. For two very recent projects, customers reported the following:

• One customer reduced testing time from about 4 weeks for every major release to 36 hours after working with us.
• Another customer has seen a 58% fall in support tickets about bugs in production, after working with us for just 3 months.

I’ll leave you with a simple question, what could such results do for you?

If you need help in cutting testing time and finding more bugs before your application's users find them, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.

Get Free Strategy Session

Download Ultimate Guide

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com

Publisher: Ayush Trivedi - 13:52

How to select the best outsourced software testing services company

The selection of the best and most compatible provider of outsourced software testing services could be the difference between your applications delivering real business value or ending up on the scrapheap of those poorly executed and costly great ideas that damaged the reputations of everyone involved. The positive or negative results of this decision will be magnified because of the added complexity of the relationship between your teams and those of your outsourced software testing services partner.

I strongly recommend a thorough perusal of best practice guide to software testing to enhance your understanding of what you should reasonably expect from your outsourced testing services partner. Once this preparatory work has been done, there are six crunch questions that you must ask prospective outsourcing partners before making your decision.

First, a word of caution

Price and affordability has its place when selecting a test automation service provider, but the underlying value that is delivered should be equally, if not more important. When assessing value, it's always worth remembering:

By no means am I advocating that your organisation pay through the nose for software testing tools and services that you do not need. Nor should you even consider contract with service providers that you genuinely believe have a sub-standard offer.

However, you should stop yourself now and enquire within about your primary motive for outsourcing your software testing program. If it is for reasons of pure cost minimisation, then you may have to incorporate specific provisions in your agreement with your outsourcing partner to ensure that the non-financial outcomes of a good software testing program are delivered throughout the engagement. If you don't, then the statistic above would imply that your outsourcing project is already doomed to fail.

Why do organisations outsource software testing?

A recent KPMG survey of senior IT executives found that testing for mobile applications was a strong driver behind looking for outsourced application testing services:

Having been in this space for some time now, we have heard a lot of reasons the other 73% of customers request our outsourced software testing or test practice development services. We categorise these into three main buckets:

• Symptoms of poor execution
• Inadequate software testing tools
• The by-product of human ambition

By far the most prevalent bucket is the one defined by symptoms of poorly executed software testing program. There are many examples of such symptoms and we colloquially refer to them as sh*t hitting the fan moments. You may have heard, or even experienced some of these in your career:

Our software testing takes too long and our competitors are releasing their products much faster than us.

IT Director of a mid-sized company

Others are more concerned about the rising cost of supporting bug-infested applications, particularly in the digital age of inter-dependent applications environments:

All of our test cases pass, but the support tickets keep piling up every time we release to production.

CIO of a financial services company

There is a group of IT professionals that is trying to make their technology teams more responsive to business needs and commercial realities, only to suffer increasing amounts of reputation damage with each failed project:

We have great ideas but our execution, particularly when it comes to quality, is letting us down in a big way.

Senior Applications Manager of a mining and resources company

There is another group of IT decision makers that are driving or are key members of expansive digital transformation programs, but they simply have not had the time to keep abreast of the latest developments in software testing tools. In fact, we realised that this problem was more common than we had assumed after coming across this statistic:

If you fall into this bucket then I recommend you read our Ultimate Guide to Test Automation for IT Decision-Makers. While it will give you some important tips about how implement test automation, it will also provide a comparison of enterprise-grade software testing tools.

The final bucket is also a very real commercial reality. Most (not all) good software testers are also qualified software engineers. They started their professional lives with dreams of writing code and creating amazing software. Somewhere along the way, they fell into a software testing role that provided them almost all of their wants, other than that job satisfaction that comes with being a badass developer.

Finding good software testers is a hard task. Retaining them for a decent length of time is even more difficult. The churn that comes with frequent turnover in your team will cripple any otherwise effective application delivery program if you are not able to give your testers an engaging work environment where they feel valued.

There are many more reasons we could add to this list. The point is, if you can identify with any of the above or variations on these common issues then you are absolutely correct to seek the expert assistance of an outsourced software testing services company. While there is no one "best" outsourced software testing company, there will be one or two that are best suited to your project. Ask the following questions to find your best fit.

Does the provider use testing tools that suit your needs?

Modern software testing tools can do a lot more, a lot faster than those tools that have been preferred by large enterprises for the last decade. If your organisation has already gone digital or is undergoing a digital transformation, then you need a fit-for-purpose software testing tool that centralises the software testing activities that will need to be executed with each code change. Where it was commonly accepted that test design, test management, test automation and test execution would be done by clumsily throwing together a bunch of different testing tools, the new breed of testing tools allow users to perform all these tasks in place.

At Qsome, we always assert that the best automated software testing tools include the following features, without exception:

All good software testing tools include test management, planning and execution. Only the best software testing tools also include automated test execution, automated scope selection and comprehensive debugging tools and analytics too.

Never accept a testing tool just because it is presented as the "industry-standard" or most popular. Not all tools are created equal and you will should take the time to understand which testing tool is right for your needs. Our guide to 10x Your Testing Effectiveness offers a list of verifiable factors that will help you understand what type of testing tool your organisation actually needs.

Before you complete this process, don't forget about load testing, especially for digital!

Want to see how our test automation tools use analytics to cut testing time to hours or minutes?

Schedule Demo Now

How will you objectively assess progress during the project?

We live in a world where all relevant information should be at your fingertips. The minutes of your weekly progress meetings should contain agenda items, not a bunch of statistics that attempt to show you progress. Your software testing service partner must provide you real-time information about the status of your tests. Information in "real-time" is key here, because without this immediacy you will still be playing catch up.

The data points that you need visibility of during a project can be varied like those given below, but should also be able to be customised to your needs:

• Test script design and creation, including editable data sets
• Visibility of tests passed, failed and unexecuted
• Defect reports, particularly those that are overdue
• Verifiable results of test execution, including video logs, screenshots and technical results

Does the provider commit to measurable and achievable project outcomes?

Insist that your software testing service provider agree to a fixed-fee and fixed-scope engagement. Such engagements compel both parties to work to reasonable timelines, concrete deliverables and measurable results. For the occasional project where a time and materials model is required, be certain to ensure that the value you desired from test outsourcing agreement are still being delivered.

In addition to the project-specific outcomes that you require from an outsourced software testing engagement, there are three key non-negotiable outcomes that your software testing services provider should deliver:

• Reduce testing time from days / weeks to hours, without compromising application quality; and
• Increased depth and coverage of tests to minimise the number of bugs that leak into production; and
• An objective, analytics-based methodology to verify if an application is ready for release.

Can the service provider deal with failure?

Let's be honest, things will go wrong in every project. So will your test outsourcing partner cover up their failures or will they communicate with you about the best path forward? Will they be honest with you in a timely fashion that allows you to pivot your development and delivery plans?

Assuming that you outsourcing your software testing for more than just cost management reasons, you will want to see thought leadership from your outsourcing partner. You will want them to experiment with new ideas that will add future value to your partnership. However, you must be ready to manage less than optimal results, because we all know that failure is a necessary outcome of innovation. Your partnership will flourish if both parties can manage sub-optimal results in a responsible and constructive manner.

The ability to deal with failure is a very difficult characteristic to assess about a third party, but it may just be the most important of the six decision points on which you will make your decision. Churchill's line about failure has proven very instructive for when we have encountered it in our projects:

Success is not final, failure is not fatal: it is the courage to continue that counts.

Winston Churchill

A customer of ours from the United Kingdom is a very innovative company that placed a great deal of faith in our technology, but most importantly in our ideas. After a few weeks of work it became clear to our project leads that the path we had adopted was not going to deliver on our promises to the customer. Our plan to tackle the project wasn't wrong, but our execution was needed a tweak.

When communicating this to the customer, we explained the reasons behind our actions and conclusions. Not only was a viable and effective remedy presented, but we also communicated how this would affect the project's timelines. Most importantly, we delivered on this new plan. A few months after this experience the same customer signed a second agreement to assist them with the remainder of their applications ecosystem.

Most organisations, even the ones outsourcing the work, do not handle failure well. The loss of time, money and more importantly, face, is an unpleasant exercise. Despite this the fruits of encouraging thought leadership and innovation can outweigh short term losses if managed well.

Our customers have reduced bugs in production by over 70%? What could such results do for you?

Get Services Delivery Guide

Can the outsourced testing provider demonstrate a testing framework that is better than yours?

So you've got your testing tool, engagement model and end-goals sorted. But then what? What is the outsourced software testing partner going to do that is a) different from your current approach, and b) how do they know that there approach will deliver the results you seek?

The single biggest factor that kills enthusiasm and future funding for test automation is a regression suite that merely cycles through a bunch of screens without actually testing for bugs. This often happens when you or your outsourced testing provider relies on generic testing framework, with no regard for how that may fit with your applications, appropriate automated test design and the value of model-based testing.

It is of vital importance to test the solution provider's technical expertise and their ability to marry technical skill with the core requirements of the user journeys that your applications enable. Check out our guide to 10x Your Testing Effectiveness to understand the technical best-practices that your software testing services partner should adhere to.

Can the provider back their claims with customer references?

This may seem obvious, but you will be surprised by how often reference checks are not performed. Even if you are 100% satisfied by what your provider of choice has demonstrated to you, it doesn't hurt to seek the views of others who have travelled the outsourced testing journey.

When speaking to referees, focus on the key topics of communications, timeliness, strengths and weaknesses in delivery. Then use this knowledge during your outsourced project to improve communication with your chosen partner and ensure that the intended benefits of the outsourcing project are delivered.

If you need help in cutting software testing time and finding more bugs before your application's users find them, speak to us understand how we might be able to help you. Right now, we're offering a free strategy session to help set you on the right path to achieve your goals.

Book Free Strategy Session

Get Services Delivery Guide

Or contact us on +61 8 8312 1287 or solutions[at]qsometech.com

Publisher: Anonymous - 02:30