Complete Guide on Prioritizing Tests with TestNG & Selenium

Have you ever faced a situation where your test automation suite started taking too long to finish? Or certain critical test cases got blocked by less important ones?

As your web or mobile app evolves, the number of test scenarios keeps increasing. Till suddenly one day you realize it‘s impossible to run all of them in each test cycle!

This is where test case prioritization helps.

What is Test Case Prioritization?

Test case prioritization refers to the process of scheduling test execution by identifying and running the most important test cases first.

Some common prioritization criteria are:

• Business criticality: Key workflows impacting revenue or compliance
• Risk: Complex modules more prone to breakages
• Frequency: Common test cases run repeatedly
• New changes: Validating latest code checkins or features

Manual test prioritization becomes tedious over time and makes it difficult to optimize dynamically.

Test automation frameworks like TestNG provide built-in support through priority and dependence features.

Introduction to TestNG

TestNG is a widely used open-source automation framework for testing Java applications and web apps built using Selenium WebDriver.

Compared to traditional JUnit, TestNG offers:

✅ Annotation-driven testing
✅ Parameterization for data-driven tests
✅ Parallel test execution
✅ Groups for selective test running
✅ Dependent methods for order of execution
✅ Detailed custom reports

These advanced capabilities take automation and continuous testing to the next level!

The latest TestNG version 7.6 further improves support for modern testing approaches.

Now let‘s see how priority can be used to schedule test case execution.

Setting Test Priority in TestNG

The @Test annotation in TestNG allows setting a "priority" attribute to dictate test case order:

@Test(priority = 1)
public void testMethod1() {
  // Test steps
}   

TestNG will then run test methods in ascending order of priority values.

But how does it work for negative numbers or no priority? Let‘s go through different scenarios.

Negative Priority

A negative priority value forces the test method to execute first before default tests:

@Test(priority = -1) 
public void negativePriorityTest() {
  // Runs first
}

@Test  
public void defaultTest() {
  // Runs second 
}

So negative priority tests always get the highest precedence.

Same Priority

If two or more test methods have the same priority set, then the backup ordering logic used by TestNG is alphabetical method names:

@Test(priority = 1)  
public void testA_First() {
   // Runs second   
}

@Test(priority = 1)   
public void testB_Second() {
   // Runs first
}

So among same priority tests, those names starting with earlier alphabets run first.

No Priority

When no priority value is specified, TestNG relies completely on alphabetical method names:

@Test   
public void testB() {
  // Runs second
}

@Test
public void testA() {
  // Runs first  
}

Leaving priority to default can thus give unexpected results. Always best to explicitly define priorities for your tests.

Priority Best Practices

Here are some thumb rules to follow when setting test priority:

✅ Use consistent priority numbers across test classes
✅ Prefer priority ranges like P1, P2 over 1, 2, 3
✅ Centralize priority values instead of hardcoding
✅ Align priorities with test management tool (JIRA / TestRail)

This ensures uniformity and improved maintainability over time.

Now let‘s look at some more examples…

TestNG Test Class Illustrating Priority

Here is an example Selenium Java test class running valid login test cases before invalid ones using priority:

import org.testng.annotations;

public class LoginTests {

  @Test(priority = 1) 
  public void validUserShouldBeAbleToLogin() {
    // Steps for positive test 
  }

  @Test(priority = 2)
  public void validUserShouldBeAbleToLogout() {
    // Steps for logout test  
  }

  @Test(priority = 3)
  public void invalidUserShouldSeeErrorMessage() {
    // Negative test 
  }

}

We can keep extending this framework with:

✅ Parameterization to pass different test data
✅ Groups to sub-divide related test cases
✅ Parallel execution to reduce overall time

TestNG provides a very flexible model to shape test execution as per priorities!

Next let‘s explore other mechanisms for reordering test runs.

Advanced Approaches for Test Case Prioritization

While priority attribute allows sequencing test methods, TestNG offers additional options:

DependsOnMethods

The dependsOnMethods specifies predecessor test method(s) for the current one:

@Test
public void test1() {
  // Runs first
}

@Test(dependsOnMethods="test1")  
public void test2() {
  // Runs only if test1 passes  
}

So test steps can be chained using this approach.

Risk-Based Prioritization

Another advanced model is ordering tests as per risk, complexity or size by splitting:

Priority	Test Cases
P1	High Risk
P2	Medium Risk
P3	Low Risk

Teams can analyze past defects or QA metrics to classify tests as high, medium or low risk test scenarios.

This ensures riskier test cases are executed earlier in cycles.

Integration with Test Management Tools

Finally, test priority can also be configured dynamically based on test case properties in JIRA, TestRail etc rather than hardcoding values.

So status, labels, components or other metadata can decide execution order for maximum flexibility.

Now let‘s run these prioritized tests across real devices…

Executing Prioritized TestNG Tests on Real Devices

While TestNG helps schedule test runs intelligently, the tests themselves need robust infrastructure across environments.

This enables validation across diverse scenarios – various device models, platforms, and network conditions.

Especially for mobile app testing, the fragmentation is endless!

As per a 2021 DeviceAtlas report, there are over 44,000 Android device models alone in circulation globally!

Manually acquiring and maintaining such environments is impossible.

This is where BrowserStack‘s mobile device lab comes into the picture.

BrowserStack‘s Device Cloud

BrowserStack offers a scalable cloud of 2000+ real mobile devices, updated continuously.

Covering names like Apple, Samsung, Motorola, Nokia etc. across various OS versions, screen sizes and resolutions.

You gain instant access to this expanding device infrastructure without any Capex or overhead of local labs!

BrowserStack also provides detailed device allocation analytics:

Devices	Tests	Success Rate
Samsung S10	5000	97%
iPhone 12	3000	95%
OnePlus 8T	1000	90%

Allowing smart distribution for faster test cycles. Integration with CI/CD pipelines then takes automated testing to the next level!

Conclusion

Test case prioritization provides a mechanism for smarter test scheduling and execution. TestNG‘s priority feature further gives testers better control over sequencing of test runs.

However running these tests for mobile or web apps needs reliable test infrastructure covering a wide range of real user devices with minimal overhead. BrowserStack‘s device cloud perfectly fits these needs to take your test automation to the next level!

So go ahead and start prioritizing your test automation. Reach out in comments below for any questions.