Test Runners

What it is

A test runner discovers test files, executes the cases, collects results and reports pass/fail. It is the engine underneath the assertions: it decides what runs, in what order, how failures are surfaced, and how exit codes drive CI.

Runners split into two broad styles:

1
  Code-driven runners              Keyword/table-driven runners
2
  ┌───────────────────────┐        ┌───────────────────────────┐
3
  │ test files → functions│        │ keyword libraries         │
4
  │   ↓ run in-process    │        │   ↓ data tables .robot    │
5
  │ assertions + fixtures │        │ steps read like prose     │
6
  └──────────┬────────────┘        └────────────┬──────────────┘
7
             ▼                                  ▼
8
      exit code → CI                   exit code + HTML report → CI

Robot Framework is the flagship of the second style: tests are written as human-readable keywords in tabular .robot files, so testers and domain experts can read and author suites without programming. It is widely used for system and acceptance testing and ships its own runner command, robot.

When to use it / when to avoid it

Use a runner when

You need consistent discovery, isolation and reporting for a suite of tests
You want a single exit code that gates your CI pipeline
You need parallelism, retries, tagging or sharding at scale
Non-programmers must read or write the tests (→ Robot Framework, keyword-driven)

Avoid or reduce it when

The project has a handful of one-off checks you run by hand
The runner would become a thick framework layer over simple scripts
You only need a quick assertion and already have a shell/script workflow

Frameworks & tools

Tool	Ecosystem	Notes
Vitest	JS/TS	Vite-native, Jest-compatible API, fast watch mode
Jest	JS/TS	Batteries included: runner, assertions, mocks, coverage
Mocha	JS/TS	Flexible runner; pair with Chai/Sinon
pytest	Python	Fixtures, parametrisation, rich plugin ecosystem
go test	Go	Built in; table-driven tests are idiomatic
JUnit 5	Java/JVM	The JVM standard, with a platform/engine split
TestNG	Java/JVM	Suites, groups, parallel and data-driven execution
xUnit	.NET	Modern, `[Fact]`/`[Theory]`
NUnit	.NET	Mature, data-driven `[TestCase]`
Robot Framework	Python/JVM	Keyword-driven runner; tabular `.robot` suites, `robot` CLI

Quick setup

1
npm install -D vitest
2
npx vitest            # watch mode; `npx vitest run` for CI

1
pip install pytest
2
pytest -q

1
go test ./...
2
go test -run TestCheckout ./...

1
# Via Maven Surefire
2
mvn -q test

1
dotnet new xunit -o MyApp.Tests
2
dotnet test

Example

1
import { describe, it, expect } from 'vitest';
2

3
describe('parser', () => {
4
  it('reads a key=value line', () => {
5
    expect(parse('a=1')).toEqual({ a: '1' });
6
  });
7
});

1
def test_reads_key_value():
2
    assert parse('a=1') == {'a': '1'}

1
func TestParse(t *testing.T) {
2
  if got := Parse("a=1"); got["a"] != "1" {
3
    t.Fatalf("got %v", got)
4
  }
5
}

1
import static org.junit.jupiter.api.Assertions.assertEquals;
2
import org.junit.jupiter.api.Test;
3

4
class ParserTest {
5
  @Test
6
  void readsKeyValue() {
7
    assertEquals("1", Parser.parse("a=1").get("a"));
8
  }
9
}

1
using Xunit;
2

3
public class ParserTests
4
{
5
    [Fact]
6
    public void ReadsKeyValue()
7
    {
8
        Assert.Equal("1", Parser.Parse("a=1")["a"]);
9
    }
10
}

1
*** Settings ***
2
Library    Collections
3

4
*** Test Cases ***
5
Reads Key Value
6
    ${result}=    Parse    a=1
7
    Should Be Equal    ${result}[a]    1

Anti-patterns & best practices

Keep the runner thin. A runner should orchestrate, not contain business setup; push reusable logic into libraries and fixtures.
Pin the runner and its plugins so a CI upgrade does not change behaviour silently.
Fail fast, but report everything. Use --bail only when flakiness is under control.
Use tags/groups for selective runs instead of commenting tests in and out.
For keyword-driven suites, keep keywords reusable and avoid putting assertion logic in ad-hoc strings scattered across test cases.
Do not mix styles in one suite without reason: pick code-driven or keyword-driven per layer of the pyramid.