Chapter 7: Integration Testing

Unit tests verify individual components in isolation. Integration tests verify that components work together correctly. While unit tests use mocks to isolate dependencies, integration tests use real databases, APIs, and services to test actual system behavior. This chapter teaches you when and how to write effective integration tests.

What is Integration Testing?

Integration testing validates interactions between components. Instead of mocking the database, you use a real test database. Instead of mocking the payment API, you hit a sandbox endpoint. Integration tests catch problems that unit tests miss—configuration errors, network issues, database constraints, API contract violations.

Unit Test Example:

Integration Test Example:

The integration test uses a real database and verifies data actually persists correctly.

When to Write Integration Tests

Write Integration Tests For:

• Database operations and transactions
• External API interactions
• File system operations
• Multi-component workflows
• Authentication and authorization
• Data validation across system boundaries

Skip Integration Tests For:

• Pure business logic (use unit tests)
• Simple calculations
• String formatting
• Trivial getters/setters

The Testing Pyramid: 70% unit tests (fast, isolated), 20% integration tests (slower, realistic), 10% end-to-end tests (slowest, full system). Integration tests sit in the middle—more realistic than unit tests but faster than full E2E tests.

Database Testing Strategies

Testing database code requires careful setup and teardown. You don't want tests interfering with each other or polluting production data.

Strategy 1: Test Database

Use a separate database for tests. Never test against production data.

# conftest.py
import pytest
from sqlalchemy import create_engine
from myapp.models import Base

@pytest.fixture(scope='session')
def test_db():
    # Create test database
    engine = create_engine('sqlite:///:memory:')
    Base.metadata.create_all(engine)
    yield engine
    Base.metadata.drop_all(engine)

Strategy 2: Transaction Rollback

Each test runs in a transaction that rolls back after the test. The database stays clean.

@pytest.fixture
def db_session(test_db):
    connection = test_db.connect()
    transaction = connection.begin()
    session = Session(bind=connection)

    yield session

    session.close()
    transaction.rollback()
    connection.close()

Now every test gets a fresh database state. Changes never persist between tests.

Strategy 3: Fixtures for Test Data

Create reusable test data:

@pytest.fixture
def sample_user(db_session):
    user = User(email="test@example.com", name="Test User")
    db_session.add(user)
    db_session.commit()
    return user

def test_update_user(db_session, sample_user):
    sample_user.name = "Updated Name"
    db_session.commit()

    # Verify update
    updated = db_session.query(User).filter_by(id=sample_user.id).first()
    assert updated.name == "Updated Name"

API Integration Testing

Testing APIs requires hitting actual endpoints (or sandbox endpoints) rather than mocking responses.

Testing Your API:

from fastapi.testclient import TestClient
from myapp import app

client = TestClient(app)

def test_create_user_endpoint():
    response = client.post("/users", json={
        "email": "test@example.com",
        "name": "Test User"
    })

    assert response.status_code == 201
    data = response.json()
    assert data['email'] == "test@example.com"
    assert 'id' in data

Testing External APIs:

For external APIs, use sandbox environments when available:

import requests

def test_payment_processing():
    # Use sandbox endpoint
    response = requests.post(
        "https://api.stripe.com/v1/charges",
        auth=("sk_test_...", ""),  # Test API key
        data={"amount": 1000, "currency": "usd", "source": "tok_visa"}
    )

    assert response.status_code == 200
    charge = response.json()
    assert charge['paid'] == True

Avoid Testing Third-Party APIs Directly: Don't make real API calls in regular test runs—they're slow, unreliable, and might cost money. Instead, test your wrapper code with integration tests that hit your code, not theirs.

Test Isolation and Cleanup

Integration tests must be isolated. One test's data shouldn't affect another.

Problem: Test Pollution:

def test_create_user():
    user = create_user("alice@example.com")  # Creates in DB
    assert user.email == "alice@example.com"

def test_user_uniqueness():
    # Fails if previous test ran first!
    user = create_user("alice@example.com")

Solution: Cleanup After Each Test:

@pytest.fixture(autouse=True)
def cleanup_database(db_session):
    yield
    # Cleanup after test
    db_session.query(User).delete()
    db_session.commit()

Or use transaction rollback as shown earlier.

Docker for Integration Tests

Docker provides isolated, reproducible test environments. Start a database in Docker for tests, run tests, then destroy the container.

docker-compose.test.yml:

version: '3.8'
services:
  test_db:
    image: postgres:14
    environment:
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_password
    ports:
      - "5433:5432"

Run tests with Docker:

docker-compose -f docker-compose.test.yml up -d
pytest tests/integration/
docker-compose -f docker-compose.test.yml down

This ensures every developer and CI server uses identical database versions.

Integration Test Best Practices

Keep Integration Tests Focused: Test one integration path at a time. Don't test the entire application in one test.

Use Test Databases: Never test against production databases. Use SQLite in-memory for speed or Docker containers for realism.

Clean Up After Tests: Reset database state between tests. Use transaction rollback or explicit cleanup.

Make Tests Independent: Tests should run in any order. Use fixtures to set up required state.

Balance Speed and Realism: In-memory databases are fast but unrealistic. PostgreSQL in Docker is realistic but slower. Choose based on what you're testing.

Mock External Services: Don't hit real third-party APIs in tests. Use sandbox endpoints, VCR cassettes, or mocks for external services.

Common Integration Testing Mistakes

Mistake 1: Testing Too Much: Integration tests that exercise the entire application are slow and fragile. Keep them focused.

Mistake 2: No Cleanup: Tests that don't clean up leave data in the database, causing future tests to fail mysteriously.

Mistake 3: Shared State: Tests that depend on each other or shared global state break when run in parallel or in different orders.

Mistake 4: Testing Third-Party Libraries: Don't test that SQLAlchemy or requests works. Test that your code uses them correctly.

Mistake 5: Slow Tests: Integration tests are slower than unit tests, but shouldn't take minutes. Optimize with in-memory databases and parallel execution.

Integration vs Unit vs E2E

Unit Tests: Fast, isolated, test single components. Use mocks. 70% of test suite.

Integration Tests: Medium speed, test component interactions. Use test databases/APIs. 20% of test suite.

E2E Tests: Slow, test complete user workflows through UI or API. Use staging environment. 10% of test suite.

Use the right test type for the right purpose. Unit tests catch logic bugs. Integration tests catch configuration and contract issues. E2E tests catch user-facing bugs.

Real-World Integration Testing Scenarios

Scenario 1: Testing Database Transactions

Transactions must be atomic—either all operations succeed or all fail. Integration tests verify this:

def test_transfer_money_transaction(db_session):
    # Setup
    account1 = Account(balance=1000)
    account2 = Account(balance=500)
    db_session.add_all([account1, account2])
    db_session.commit()

    # Test transfer
    try:
        account1.balance -= 200
        account2.balance += 200
        db_session.commit()
    except:
        db_session.rollback()
        raise

    # Verify both accounts updated
    assert account1.balance == 800
    assert account2.balance == 700

Scenario 2: Testing API Rate Limiting

Integration tests can verify rate limiting works:

def test_api_rate_limiting():
    client = TestClient(app)

    # Make requests up to limit
    for i in range(100):
        response = client.get("/api/data")
        assert response.status_code == 200

    # Next request should be rate limited
    response = client.get("/api/data")
    assert response.status_code == 429  # Too Many Requests

Scenario 3: Testing Authentication Flow

def test_login_flow(client, db_session):
    # Create user
    user = User(email="test@example.com")
    user.set_password("secretpassword")
    db_session.add(user)
    db_session.commit()

    # Test login
    response = client.post("/login", data={
        "email": "test@example.com",
        "password": "secretpassword"
    })

    assert response.status_code == 200
    assert "session_token" in response.cookies

Debugging Integration Test Failures

Integration tests fail for different reasons than unit tests. Here's how to debug them:

Check Database State: When a database test fails, inspect the actual data. Use database tools to query tables directly and see what state tests left behind.

Verify Test Isolation: Run the failing test alone. If it passes alone but fails in the suite, you have test pollution. Tests are sharing state or depending on execution order.

Check Connection Configuration: Integration tests often fail due to misconfigured connections. Verify database URLs, API endpoints, and credentials in test configuration.

Look at Test Data: Failed assertions often indicate test data issues. Verify fixtures set up the expected state before the test runs.

Enable Logging: Integration tests interact with real systems. Enable debug logging to see actual SQL queries, HTTP requests, and responses.

Continuous Integration for Integration Tests

Integration tests in CI require more setup than unit tests but provide critical coverage:

Setup Test Infrastructure: CI needs access to test databases and services. Use Docker Compose to spin up dependencies:

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_PASSWORD: test
        ports:
          - 5432:5432

    steps:
      - uses: actions/checkout@v2
      - name: Run integration tests
        run: pytest tests/integration/
        env:
          DATABASE_URL: postgresql://postgres:test@localhost:5432/test

Separate Integration and Unit Tests: Run unit tests first (fast feedback), then integration tests (thorough coverage). This speeds up CI:

pytest tests/unit/  # Fast, runs first
pytest tests/integration/  # Slower, runs after unit tests pass

Parallel Execution: Run integration tests in parallel when possible:

pytest -n auto tests/integration/  # Uses pytest-xdist for parallel execution

Cache Dependencies: Cache Docker images and database schemas to speed up CI runs.

Integration Testing Quick Reference

When to Write:

• Database CRUD operations
• API endpoints
• File operations
• Authentication flows
• Multi-step workflows

When to Skip:

• Pure calculations
• Simple string operations
• Already well-tested libraries
• Trivial getters/setters

Key Principles:

1. Isolation: Each test runs independently
2. Cleanup: Reset state after every test
3. Focus: Test one integration at a time
4. Speed: Use in-memory databases when possible
5. Realism: Use Docker for production-like environments

Common Tools for Integration Testing:

• pytest: Primary test framework with powerful fixtures
• SQLAlchemy: ORM with excellent test support
• TestClient: FastAPI/Flask test client for API testing
• Docker Compose: Isolated test services and databases
• pytest-xdist: Parallel test execution for speed

Remember: Integration tests are more expensive to run than unit tests but catch real-world configuration and integration issues that mocks miss completely. Balance comprehensive coverage with reasonable execution time. Use them strategically to test critical system integrations while relying on faster unit tests for most coverage.

Course Recommendations

Advanced Python Testing

• Integration testing strategies
• Database testing patterns
• API testing techniques
• Enroll at paiml.com

Microservices Testing

• Service integration testing
• Contract testing
• Distributed system testing
• Enroll at paiml.com

DevOps for Python Developers

• Docker for testing
• CI/CD integration test pipelines
• Test environment management
• Enroll at paiml.com

Quiz