Compatibility Testing

What it is

Compatibility testing checks that the software behaves correctly across the environments users actually have: operating systems, browsers, devices, screen sizes, locales and versions. The matrix is the whole game.

1
                       browser
2
                 Chrome  Firefox  Safari  WebView
3
   Windows   ───   ✓       ✓       –       –
4
   macOS     ───   ✓       ✓       ✓       –
5
   iOS       ───   –       –       ✓       ✓
6
   Android   ───   ✓       ✓       –       ✓
7

8
   Plus: screen sizes, input methods, locales, app versions.

Combinatorial growth is real, so you pick a covering set of combinations rather than every cell.

When to use it / when to avoid it

Use it for

Public web apps where any browser or device may arrive
Native and mobile apps that target several OS versions
Regulated or enterprise products with a fixed, documented support matrix
CSS, fonts, video, WebRTC, file handling — the areas browsers disagree on most

Avoid or reduce it when

You control the runtime (a kiosk, an embedded device) — test only that target
You test every cell manually instead of pairing a matrix with automation
The matrix is undocumented, so nobody knows what “supported” means
You chase the newest browser while ignoring a large share of real traffic

Frameworks & tools

Tool	Ecosystem	Notes
BrowserStack	Cloud devices	Real browsers and devices on demand
Sauce Labs	Cloud devices	Cross-browser and mobile testing cloud
Playwright	JS/TS, Python, Java, .NET	Chromium, Firefox and WebKit engines locally
Selenium Grid	Multi	Run tests across remote browsers and nodes
LambdaTest	Cloud devices	Selenium/Playwright grid in the cloud
Appium	Mobile	Native, hybrid and web apps on iOS/Android
responsive design mode	Browsers	Quick viewport and touch emulation
Appium Inspector	Mobile	Inspect and drive native mobile app elements

Quick setup

1
npm install -D @playwright/test
2
npx playwright install --with-deps
3
npx playwright test --project=chromium --project=firefox --project=webkit

1
pip install playwright
2
playwright install
3
playwright test

1
go get github.com/tebeka/selenium

1
<dependency>
2
  <groupId>com.saucelabs</groupId>
3
  <artifactId>sauce-java</artifactId>
4
  <version>1.2.3</version>
5
  <scope>test</scope>
6
</dependency>

1
dotnet add package Microsoft.Playwright.NUnit
2
pwsh bin/Debug/net8.0/playwright.ps1 install

Example

1
import { defineConfig, devices } from '@playwright/test';
2

3
export default defineConfig({
4
  projects: [
5
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
6
    { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
7
    { name: 'webkit', use: { ...devices['Desktop Safari'] } },
8
    { name: 'pixel', use: { ...devices['Pixel 7'] } },
9
    { name: 'iphone', use: { ...devices['iPhone 15'] } },
10
  ],
11
});

1
import pytest
2
from playwright.sync_api import sync_playwright
3

4
@pytest.mark.parametrize("browser", ["chromium", "firefox", "webkit"])
5
def test_home_renders(browser):
6
    with sync_playwright() as p:
7
        page = getattr(p, browser).launch().new_page()
8
        page.goto("http://localhost:4321/")
9
        assert page.title()

1
caps := map[string]interface{}{
2
  "browserName":    "chrome",
3
  "bstack:options": map[string]string{"os": "Windows", "osVersion": "11"},
4
}
5
wd, err := selenium.NewRemote(caps, "https://hub-cloud.browserstack.com/wd/hub")
6
_ = wd
7
_ = err

1
MutableCapabilities caps = new MutableCapabilities();
2
caps.setCapability("browserName", "safari");
3
caps.setCapability("platformName", "macOS 14");
4
WebDriver driver = new RemoteWebDriver(new URL(SauceURL), caps);

1
using Microsoft.Playwright;
2
using Microsoft.Playwright.NUnit;
3

4
[TestFixture]
5
public class HomePageTests : PageTest
6
{
7
    [Test]
8
    public async Task RendersOnSafari()
9
    {
10
        await Page.GotoAsync("http://localhost:4321/");
11
        await Expect(Page).ToHaveTitleAsync(new Regex(".+"));
12
    }
13
}

Anti-patterns & best practices

Define the support matrix explicitly. Publish which OS/browser/device versions you actually support, then test those.
Automate the matrix, manual-test the edges. Playwright and a cloud grid cover breadth; humans check visual and input nuances.
Weight by real analytics. Fix the browsers your users have, not the one you use.
Cover input and locale, not just rendering. Touch, keyboard, RTL and translated strings break differently.
Test older versions deliberately. The failure is often on the oldest supported version, not the newest.
Keep device lab images fresh. Stale emulators lie about modern behaviour.