Software Testing¶

Testing produces evidence about software quality and risk. It can reveal failures; it cannot prove that non-trivial software has no defects. A useful strategy tests important behavior at the cheapest boundary that gives trustworthy feedback.

Start with Risk and an Oracle¶

For each behavior ask:

What could fail, and who or what would be harmed?
How likely is it and how detectable would it be?
What result would distinguish correct from incorrect behavior?
At which boundary can that result be observed reliably?
Which cases are too expensive or unsafe to test manually?

A test oracle is the source used to decide the expected result: a requirement, invariant, independent implementation, model, standard, or trusted observation. Weak oracles create tests that merely repeat implementation mistakes.

Core Principles¶

Testing shows the presence of failures, not their absence.
Exhaustive testing is usually impossible; select cases by risk.
Test important behavior early and continuously.
Defects cluster around complexity, change, and weak boundaries.
Repeating the same tests eventually stops finding new information.
Testing depends on context; a game and a medical device need different evidence.
A system can satisfy its specification and still fail users.

These align with the maintained ISTQB Foundation syllabus, but a team does not need certification terminology to apply them.

Test Boundaries¶

Unit Tests¶

Exercise a small unit of behavior in memory. They should be fast, deterministic, and focused on public behavior rather than private implementation details.

def allocate(total: int, recipients: int) -> tuple[int, int]:
    if recipients <= 0:
        raise ValueError("recipients must be positive")
    return divmod(total, recipients)


def test_allocate_preserves_total():
    each, remainder = allocate(10, 3)
    assert each * 3 + remainder == 10


def test_allocate_rejects_no_recipients():
    try:
        allocate(10, 0)
    except ValueError:
        return
    raise AssertionError("expected ValueError")

Mock only boundaries that are slow, non-deterministic, destructive, or outside the test's scope. Excessive mocking couples tests to call sequences and can pass while components fail together.

Integration Tests¶

Verify components with real boundary behavior: database constraints and transactions, serialization, filesystem rules, broker semantics, or an external service sandbox. Use the real dependency when its behavior is what matters.

Contract Tests¶

Verify that a provider and consumer agree on request, response, event, and compatibility semantics. Schema validation alone does not test status codes, authorization, defaults, side effects, ordering, or timing assumptions.

End-to-End Tests¶

Exercise a critical workflow through the deployed system. They catch wiring and configuration failures but are slower and harder to diagnose. Keep a small set for high-value journeys rather than reproducing every lower-level case.

Acceptance and Exploratory Testing¶

Acceptance tests provide evidence that agreed user or business behavior works. Exploratory testing combines learning, test design, and execution to discover risks scripted checks miss. Give it a mission, record observations, and turn recurring valuable findings into durable checks.

A Practical Test Portfolio¶

Prefer many fast, focused tests; enough integration and contract tests to cover real boundaries; and few broad end-to-end tests. This is an economic shape, not a required geometric ratio.

Place a regression test at the lowest boundary that reproduces the escaped defect faithfully. A bug caused by a database isolation rule needs a real-database test, not a mocked unit test.

Designing Test Cases¶

Equivalence Partitions¶

Group inputs expected to behave alike and select representatives from valid and invalid groups. For an allowed age of 18–120, useful partitions include below 18, 18–120, and above 120.

Boundary Values¶

Failures cluster at edges. Test values immediately below, at, and above important boundaries: 17, 18, 120, and 121.

Decision Tables¶

List combinations of conditions and expected actions when business rules interact. This exposes missing and contradictory rules better than prose.

State Transitions¶

Test valid and invalid events from each important state, repeated events, and recovery after interruption. State bugs often hide in sequences rather than single inputs.

Pairwise and Combinatorial Selection¶

When every combination is infeasible, cover interactions among a chosen number of parameters. Use this only when interaction strength assumptions are reasonable; safety-critical combinations may require more.

Error Guessing¶

Use experience and incident history: empty values, duplicates, clock boundaries, Unicode, retries, reordered events, full storage, expired credentials, and interrupted migrations.

Properties and Examples¶

Example tests communicate concrete business behavior. Property-based tests generate many inputs against invariants such as:

sorting preserves the multiset and produces ordered output;
encoding then decoding returns the original supported value;
allocating parts preserves the total;
an idempotent operation repeated with the same key has one effect.

Generators must respect the real input domain, and failures should shrink to a reproducible example. Properties complement, rather than replace, meaningful examples.

Test-Driven Development¶

TDD is a short feedback cycle:

write a small failing test for desired behavior;
make it pass with the minimum implementation;
improve the design while tests stay green.

It can clarify APIs and provide regression coverage. It is not mandatory for every change, and it does not replace integration, security, usability, or exploratory testing. Avoid tests that freeze private structure before behavior is understood.

Coverage and Mutation Testing¶

Statement and branch coverage reveal code that did not execute; they do not show whether assertions were meaningful or requirements were covered. Use coverage to find gaps, not as a quality score or universal target.

Mutation testing makes small code changes and checks whether tests fail. Surviving meaningful mutants can expose weak assertions. Exclude equivalent or irrelevant mutants and use the technique selectively where correctness risk justifies its cost.

Non-Functional Testing¶

Performance¶

Define workload, data shape, environment, and success thresholds before running a test.

load tests assess expected and peak demand;
stress tests find saturation and failure behavior;
soak tests reveal leaks and degradation over time;
spike tests assess sudden changes;
capacity tests estimate safe operating limits.

Measure latency distributions, throughput, errors, saturation, and recovery. A benchmark without reproducible conditions is an anecdote.

Reliability and Resilience¶

Test timeouts, retries, dependency failure, overload, failover, restoration, and partial recovery. Verify safety invariants during the failure, not only that the service eventually returns.

Chaos experiments require a hypothesis, production-representative conditions, observable impact, bounded blast radius, and an abort mechanism.

Security¶

Derive tests from threats and trust boundaries. Cover authentication, object- and action-level authorization, input handling, session lifecycle, secret exposure, dependency risk, logging, and abuse limits. Automated scanners miss business-logic flaws and need human review.

Use the living OWASP Web Security Testing Guide for web-specific depth.

Accessibility and Compatibility¶

Combine automated checks with keyboard use, screen-reader evaluation, zoom and contrast review, and representative users. Test supported browsers, devices, operating systems, locales, time zones, and upgrade paths according to actual usage and policy.

Distributed and Asynchronous Systems¶

Test duplicate, delayed, reordered, and missing messages; consumer restarts; poison messages; replay; partition reassignment; and version skew. Assert eventual outcomes with bounded polling rather than fixed sleeps.

Verify idempotency at the durable side-effect boundary. A mocked broker cannot prove database and message-publication atomicity.

Test Data and Environments¶

Generate the smallest data that communicates intent.
Keep tests independent and clean up reliably.
Never copy sensitive production data without authorization and protection.
Make clocks, randomness, identifiers, and external responses controllable where needed.
Keep test infrastructure close enough to production semantics for the claim being tested.
Version schemas, fixtures, and environment configuration with the code.

Shared environments create hidden coupling. Prefer isolated resources or unique namespaces when feasible.

Determinism and Flaky Tests¶

A flaky test sometimes passes and fails without a relevant product change. Treat it as a defect because it trains teams to ignore evidence.

Common causes:

fixed sleeps and timing races;
leaked global or external state;
order dependence;
unseeded randomness;
locale, time-zone, or clock dependence;
unstable external services;
resource exhaustion.

Diagnose and fix the cause. Quarantine only when necessary to restore signal, with an owner and deadline. Blind retries hide instability and inflate pipeline time.

Testing in Delivery Pipelines¶

Order checks for useful feedback:

formatting, static analysis, and focused unit tests;
build and component integration tests;
contract, migration, security, and broader integration checks;
deploy to a production-like environment and run critical workflows;
release progressively and verify production signals.

Parallelize independent checks, cache safely, and run expensive suites according to risk. One immutable artifact should move through environments.

Testing in production can include synthetic checks, canaries, shadow traffic, and feature experiments. It complements pre-production testing; it does not justify exposing users to uncontrolled risk.

Useful Metrics¶

Track outcomes rather than test counts:

escaped defects and affected users;
time to detect and correct failures;
flaky-test rate and quarantine age;
feedback time and queue time;
failures by boundary and cause;
coverage of critical risks and workflows;
mutation results for selected critical code.

Metrics become harmful when turned into individual targets. A team can raise coverage or test count without increasing confidence.

Review Checklist¶

Which risk or behavior does each test cover?
Is the oracle independent and meaningful?
Is this the cheapest faithful test boundary?
Are failures deterministic and easy to diagnose?
Are real integration semantics tested where they matter?
Are negative, boundary, concurrency, and recovery cases covered?
Do security and authorization tests follow the threat model?
Are critical user workflows represented without duplicating all lower layers?
Does production monitoring cover failures tests cannot predict?
Which tests no longer provide enough value to keep?

A maintainable suite maximizes trustworthy information per unit of execution and maintenance cost.