Regression Testing

What it is

A regression test verifies that behaviour which used to work still works after a change. Every git push, dependency bump, refactor or config tweak risks breaking something unrelated — regression testing is the net that catches it before users do.

1
  before ──► change ──► regression suite ──► behaved as before?
2
                          │ pass                │
3
                          └ fail → a regression  └ ship with confidence

Regression testing is not a test level (unit, integration, E2E) but a purpose: any test at any level can be a regression test. The practice is usually automated and lives in CI, growing over time as each fixed bug earns a permanent test.

When to use it / when to avoid it

Use it for

After every code change, before merge or release
After upgrading a dependency, framework or runtime
Whenever a bug is fixed — add a test so it can never regress silently
Refactors that should preserve observable behaviour

Avoid it when

You are verifying one specific fix in isolation → a sanity test is cheaper and sharper
You need to confirm the build even boots → start with a smoke test
The “test” only re-asserts the implementation details of the code you just wrote

Frameworks & tools

Tool	Ecosystem	Notes
Vitest / Jest	JS/TS	Snapshot tests catch unintended output drift
Playwright	Multi	Replay critical journeys on every build
pytest	Python	Parametrised regression cases; `pytest-regressions`
Go testing	Go	`go test ./...`; golden files for output comparison
JUnit 5	Java/JVM	Suites + tags; run the full suite on every PR
xUnit / NUnit	.NET	`dotnet test` across the whole solution
Robot Framework	Python	Keyword suites run on every release candidate
BackstopJS / Percy	Multi	Visual regression (screenshot diffing)
ApprovalTests	Multi	Golden-master / approval testing
k6	Multi	Performance regression baselines

Quick setup

1
npm install -D vitest @vitest/coverage-v8
2
npx vitest run --coverage

1
pip install pytest pytest-regressions
2
pytest -q

1
go test ./... -count=1
2
go test ./... -update   # regenerate golden files after intended change

1
<!-- Maven: fail the build if any test regresses -->
2
<plugin>
3
  <groupId>org.apache.maven.plugins</groupId>
4
  <artifactId>maven-surefire-plugin</artifactId>
5
  <configuration>
6
    <testFailureIgnore>false</testFailureIgnore>
7
  </configuration>
8
</plugin>

1
dotnet test --no-build --verbosity normal

Example

1
import { describe, it, expect } from 'vitest';
2
import { formatInvoice } from './invoice';
3

4
describe('formatInvoice (regression)', () => {
5
  it('keeps the established output format', () => {
6
    expect(formatInvoice({ total: 1200, currency: 'USD' }))
7
      .toMatchInlineSnapshot('"$12.00"');
8
  });
9
});

1
def test_invoice_format_stable(file_regression):
2
    result = format_invoice(total=1200, currency="USD")
3
    file_regression.check(result)

1
func TestInvoiceFormatRegression(t *testing.T) {
2
  got := FormatInvoice(1200, "USD")
3
  golden, _ := os.ReadFile("testdata/invoice.golden")
4
  if got != string(golden) {
5
    t.Fatalf("regression: output changed\nwant %q\ngot  %q", golden, got)
6
  }
7
}

1
import static org.junit.jupiter.api.Assertions.assertEquals;
2
import org.junit.jupiter.api.Test;
3

4
class InvoiceRegressionTest {
5
  @Test
6
  void formatStaysStable() {
7
    assertEquals("$12.00", Invoice.format(1200, "USD"));
8
  }
9
}

1
using Xunit;
2

3
public class InvoiceRegressionTests
4
{
5
    [Fact]
6
    public void FormatStaysStable()
7
    {
8
        Assert.Equal("$12.00", Invoice.Format(1200, "USD"));
9
    }
10
}

Anti-patterns & best practices

Add a test with every bug fix. A fixed bug without a regression test will come back.
Prefer behaviour over snapshots alone. Snapshots catch drift but say nothing about intent; pair them with explicit asserts.
Keep the suite fast enough to run every time. If it is too slow to run, it will not run.
Do not assert implementation details. Tests bound to private structure fail on harmless refactors.
Update goldens deliberately. A golden change should be a reviewed, intended change, never a blind regenerate.
Quarantine, do not ignore, flaky tests. A red suite people ignore is not a safety net.