The Comprehensive Guide to Rerunning Failed Tests with TestNG

As a seasoned quality assurance expert with over 10 years of experience in test automation, I‘ve executed test suites across thousands of real mobile devices and browsers. One inevitable challenge in any testing project is the occurrence of unexpected test case failures. However, with the right tools and strategies, these failures can be efficiently diagnosed, fixed and re-run.

This comprehensive technical guide will teach you how to reliably detect and rerun failed test cases using TestNG integrated with Selenium WebDriver.

Chapter 1 – Introduction

Before jumping into the details, let‘s first broadly understand:

• Why tests fail – Common root causes
• Significance of TestNG – Benefits over other frameworks
• Top strategies to rerun and fix failures

Why Do Tests Fail?

In my decade long testing journey, I‘ve encountered countless test failures ranging from simple script bugs to complex infrastructure issues.

Here are the most common categories:

Application Defects

• Functional bugs
• Regression bugs
• Error handling issues

Test Suite Defects

• Locator changes
• Assertion failures
• Race conditions
• Dependency conflicts

Environment Issues

• Network glitches
• Hardware failures
• Database outages

Configuration Changes

• Testing framework upgrades
• New libraries/dependencies
• Browser version changes

By anticipating these failure categories, we can strategically establish mechanisms to surface, diagnose and rectify test failures early.

Why Use TestNG to Run Failed Tests?

While debugging failures, speed is everything. The faster we re-execute failing tests, the quicker we validate fixes. This accelerates release cycles and improves product quality.

From my experience, TestNG stands out from other frameworks with these advantages:

1. Detailed Failure Reporting

• Precise test metrics on pass %, fails, skips
• Drill-down on root cause via stack traces
• Custom logging for test diagnostics

2. Native Support for Re-Runs

• Auto-generated testng-failed.xml file
• Contains only failed test cases
• Re-execute failures in isolation

3. Annotations and Grouping

• Logical classification of test cases
• Set priorities and dependencies
• Helps structure tests and identify areas impacted

4. Multi-threading and Parallelization

• Simultaneous test execution
• Significantly reduces test cycles

5. Customization and Extensibility

• Adapt to specific process needs
• Integrate with test management tools

The bottom line is that TestNG, together with Selenium, provides a comprehensive test harness to surface, rerun and resolve test failures rapidly.

Top Strategies for Rerunning Failures

Through trial-and-error over thousands of test cycles, I‘ve found these techniques to be highly effective:

• Analyze test reports to pinpoint root cause
• Isolate failures and re-execute only failing tests
• Debug systematically when cause is unclear
• Document errors to identify patterns
• Refactor complex test cases
• Cross-verify related functionality has not broken

Now let‘s explore step-by-step how to apply these strategies using TestNG to efficiently rerun failed tests.

Chapter 2 – Step-by-Step Process

The end-to-end process involves:

1. Writing test cases
2. Executing full test suite
3. Analyzing failures
4. Leveraging testng-failed.xml file
5. Rerunning just failed tests
6. Fixing errors
7. Confirming tests now pass

1. Writing Test Cases

I begin with authoring TestNG test classes with Selenium WebDriver bindings in Java:

//Test1.java

import org.testng.Assert;

public class Test1 {

  @Test
  public void test1(){  
    Assert.assertTrue(true); //Pass
  } 
}

//Test2.java  

import org.testng.Assert;

public class Test2 {   

  @Test
  public void test2(){

    Assert.assertTrue(false); //Fail 
  }
}

//Test3.java

import org.testng.Assert;

public class Test3 {

  @Test
  public void test3(){

    Assert.assertTrue(true); //Pass
  }  
}

To demonstrate failure handling, I intentionally fail one test – Test2.

The source code is available on GitHub to clone as a sample project.

2. Executing Full Test Suite

Now I execute all the test cases together via:

Command Line

mvn test

CI/CD Pipelines

On completion, I inspect the test-output results:

Emailable Report

Index HTML Report

We clearly see one test has failed while others passed.

3. Analyzing Failures

The next step is to analyze why Test2 failed.

By expanding the test row, I view the failure stack trace:

The error java.lang.AssertionError indicates assertion failure on line X. I also debug in my IDE to troubleshoot further.

4. Leveraging testng-failed.xml

A key artifact automatically generated by TestNG is testng-failed.xml file.

It selectively includes only failed tests from the last run.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Default suite]" verbose="0">

  <test name="Default test(failed)">
    <classes>
      <class name="test.Test2">  
        <methods>
          <include name="test2"/>
        </methods>
      </class>
    </classes>
  </test>

</suite>

This helps efficiently execute just the failed test cases.

5. Rerunning Failed Tests

Now I trigger a new test run using testng-failed.xml:

mvn test -Dsurefire.suiteXmlFiles=testng-failed.xml 

This reruns only Test2 which fails again:

6. Fixing Root Cause

Next step is to fix the intentional failure I introduced:

//Updated Test2

import org.testng.Assert;

public class Test2 {

  @Test
  public void test2(){

    Assert.assertTrue(true); //Fixed
  }

}

I save this change and re-execute testng-failed.xml.

7. Confirming Tests Now Pass

On running testng-failed.xml once again:

I see Test2 now PASSING. The failure is fixed!

I can optionally do a full test suite run to confirm all test cases are passing consistently.

This completes the end-to-end workflow.

Chapter 3 – Best Practices

From my years of test automation experience, here are 8 best practices I highly recommend when rerunning failed tests:

1. Deep Dive into Root Cause

• Inspect stack trace thoroughly
• Debug failed tests to understand issue
• Don‘t just re-execute without diagnosis

2. Document Errors Meticulously

• Note environments, inputs, dates
• Identify patterns in issues
• Help address underlying triggers

3. Standardize Failure Handling

• Create listeners/hooks to catch failures
• Take screenshots or system logs
• Ensure consistent response

4. Refactor Complex Test Code

• Break convoluted test cases into smaller cases
• Decompose test classes into logical units
• Improves structure and isolates problems

5. Validate Related Functionality

• Check areas that could be impacted
• Confirm no regression bugs introduced
• Include related test cases in testng-failed.xml

6. Track Recurring Test Failures

• Filter highly failing test cases
• Failures beyond threshold need deeper fixes
• Address reliability issues proactively

7. Set Up Appropriate Wait Timeouts

• Dynamic wait conditions for response variations
• Avoid premature pass/failures

8. Re-run Test Suite Completely

• Once fixes are verified, do a full test run
• Validate holistic stability and completeness

Adopting these best practices is guaranteed to help test teams achieve rapid, reliable and thorough reruns of failed test cases.

Chapter 4 – Key Takeaways

Let me recap the major highlights:

✔️ Understand why tests fail – Categorize common failure root causes like application defects, test suite issues, environment problems etc.

✔️ Leverage TestNG capabilities – Features like descriptive reports, annotations, custom logs help debug failures faster.

✔️ Utilize testng-failed.xml – Auto-generated file to easily re-execute just failed test cases.

✔️ Analyze->Fix->Retest – Methodically diagnose root cause, address errors, confirm fixes work.

✔️ Adopt best practices – Standardize responses, validate comprehensively, track recurring issues etc.

✔️ Result = Faster release cycles -Accelerates confirming fixes and improves software quality.

I hope this guide gives you a firm understanding of how to efficiently leverage TestNG with Selenium WebDriver to rerun failed tests cases. Do share your feedback or questions in the comments section below.

Author: Mike Smith  
QA Architect & Test Automation Expert
10+ years experience executing test automation frameworks for Fortune 500 companies.