Chapter 6: Code Coverage and Test Quality

Code coverage measures what percentage of your code is executed during tests. It's a useful metric but often misunderstood. High coverage doesn't guarantee good tests, and chasing 100% coverage can waste time. This chapter teaches you to use coverage effectively—as a tool for finding untested code, not as a quality goal itself.

What is Code Coverage?

Code coverage is the percentage of your source code that runs during your test suite. If you have 100 lines of code and tests execute 80 of them, you have 80% coverage.

Why Coverage Matters: Untested code is likely to have bugs. Coverage helps identify code that has zero tests. It answers: "What did I forget to test?"

Why Coverage Alone Isn't Enough: A test that executes code but doesn't assert anything has perfect coverage but zero value. Coverage measures execution, not verification. You can have 100% coverage with terrible tests.

Types of Coverage:

Line Coverage: Percentage of lines executed. Most common and easiest to understand.

Branch Coverage: Percentage of code branches (if/else paths) tested. More thorough than line coverage.

Function Coverage: Percentage of functions called during tests.

Statement Coverage: Percentage of statements executed, similar to line coverage.

Branch coverage is generally more useful than line coverage because it ensures you test both true and false conditions.

Using pytest-cov

pytest-cov integrates coverage.py with pytest, making coverage analysis simple.

Installation:

pip install pytest-cov

Basic Usage:

# Run tests with coverage
pytest --cov=mypackage tests/

# Generate HTML report
pytest --cov=mypackage --cov-report=html tests/

# Show missing lines
pytest --cov=mypackage --cov-report=term-missing tests/

Understanding the Output:
```
Name Stmts Miss Cover Missing

mypackage/utils.py 45 5 89% 23-27
mypackage/core.py 89 0 100%

TOTAL 134 5 96%

"Stmts" is total statements. "Miss" is untested statements. "Cover" is percentage. "Missing" shows which line numbers weren't executed.

<!-- 

  
    
  
 -->

## Interpreting Coverage Reports

**Line vs Branch Coverage**:

```python
def calculate_discount(price, is_member):
    if is_member:
        return price * 0.9
    return price

A test calling calculate_discount(100, True) gives 100% line coverage but only 50% branch coverage—you didn't test the is_member=False path.

The Missing Lines Column: This is the most valuable part of coverage reports. It tells you exactly which lines never executed. Review these lines and decide if they need tests.

Coverage by File: Look at coverage per file. Files with low coverage (<70%) likely need more tests. Files with 100% coverage might be over-tested.

Setting Realistic Coverage Goals

The 80/20 Rule: Aim for 80-85% overall coverage. The last 20% often isn't worth the effort. Some code is hard to test (error handling, edge cases, integration points) and provides diminishing returns.

What to Test First: Test core business logic thoroughly (90%+ coverage). Integration code and utilities can have lower coverage (60-70%). Configuration files and trivial getters don't need tests.

Coverage Targets by Code Type:

• Business logic: 85-95%
• Utilities: 75-85%
• Integration code: 60-75%
• UI/presentation: 40-60%
• Configuration: 0-20%

Coverage as a Floor, Not a Ceiling: Set minimum coverage (e.g., 80%) and fail builds below that. Don't set maximum or reward highest coverage—that encourages gaming the metric.

Coverage Anti-Patterns

Pattern 1: Testing for Coverage, Not Behavior

Bad:

def test_just_for_coverage():
    result = some_function(5)
    # No assertions! Just executing code for coverage.

This achieves coverage without testing anything. Always assert expected behavior.

Pattern 2: Testing Implementation Details

Bad:

def test_internal_method():
    obj = MyClass()
    obj._private_method()  # Testing internals

Testing private methods inflates coverage but makes tests brittle. Test public interfaces.

Pattern 3: 100% Coverage Obsession

Chasing 100% coverage wastes time on trivial code. Test important paths thoroughly instead of every line superficially.

Pattern 4: No Assertions

def test_code_runs():
    process_data([1, 2, 3])
    # Test passes if no exception, but doesn't verify correctness

Coverage counts this as tested, but it's not. Add assertions.

Using Coverage to Find Gaps

Step 1: Run coverage and identify untested files or functions

pytest --cov=src --cov-report=term-missing

Step 2: Review missing lines

Are they:

• Critical business logic? → Write tests
• Error handling? → Write tests for error cases
• Dead code? → Delete it
• Trivial getters? → Maybe skip

Step 3: Write tests for important gaps

Focus on code that has consequences: calculations, validations, data transformations.

Step 4: Ignore unimportant gaps

Trivial code like return self.name doesn't need tests. Logging statements don't need coverage.

Branch Coverage vs Line Coverage

Line coverage can hide untested paths:

def validate_age(age):
    return age >= 18 and age <= 120

# Test
def test_valid_age():
    assert validate_age(25) == True

This has 100% line coverage (all lines executed) but 50% branch coverage (never tested False case).

Enable Branch Coverage:

pytest --cov=src --cov-branch --cov-report=term-missing

Branch coverage catches these gaps. It ensures you test both True and False for each condition.

Coverage in CI/CD

Enforce Minimum Coverage:

# Fail if coverage below 80%
pytest --cov=src --cov-fail-under=80

Add this to CI pipeline to prevent coverage regression.

Coverage Badges: Display coverage percentage in README using services like Codecov or Coveralls. This makes coverage visible to the team.

Track Coverage Trends: More important than absolute coverage is the trend. Is coverage improving or declining? Declining coverage suggests insufficient testing of new code.

When High Coverage Doesn't Help

Example: High Coverage, Low Quality:

def transfer_money(from_account, to_account, amount):
    from_account.balance -= amount
    to_account.balance += amount

def test_transfer():
    acc1 = Account(balance=100)
    acc2 = Account(balance=50)
    transfer_money(acc1, acc2, 30)
    # No assertions! Just executes code.

This test has 100% coverage but verifies nothing. Coverage is meaningless without assertions.

Coverage Can't Test:

• Logic correctness (only assertions do that)
• Performance or efficiency
• Security vulnerabilities
• User experience

Quality Over Quantity

Good Test (even if low coverage):

def test_payment_processing_end_to_end():
    # Tests critical path thoroughly
    payment = create_payment(100)
    result = process_payment(payment)
    assert result.status == "success"
    assert result.transaction_id is not None
    assert payment.amount_charged == 100

Bad Tests (even with high coverage):

def test_all_functions():
    function1(); function2(); function3()
    # Executes everything but tests nothing

Focus on: Critical paths, edge cases, business logic, error handling

Don't worry about: Getters/setters, trivial utilities, generated code

Practical Coverage Workflow

1. Write tests first (TDD ensures coverage naturally)
2. Run coverage periodically to find gaps
3. Review uncovered code and prioritize
4. Write tests for important gaps
5. Ignore trivial uncovered code
6. Maintain minimum threshold in CI (e.g., 80%)
7. Focus on assertions, not just execution

Coverage is a diagnostic tool, not a goal. Use it to find weak spots, then write meaningful tests.

Coverage Configuration Best Practices

Create a .coveragerc file to configure coverage analysis:

[run]
source = src
omit =
    */tests/*
    */venv/*
    */__pycache__/*
    */migrations/*

[report]
precision = 2
show_missing = True
skip_covered = False

[html]
directory = htmlcov

This configuration tells coverage to measure your src directory while excluding tests, virtual environments, and generated code from analysis.

Exclude Generated Code: Don't measure coverage on auto-generated files, migrations, or vendor code. These inflate coverage numbers without adding value.

pytest.ini Integration:

[tool:pytest]
addopts = --cov=src --cov-report=term-missing --cov-report=html --cov-fail-under=80

Now every pytest run includes coverage automatically. The build fails if coverage drops below 80%.

Improving Low Coverage Areas

When you find code with low coverage, follow this process:

Step 1: Identify the Gap. Run coverage and find files below your threshold. Focus on files with business logic, not utilities.

Step 2: Understand Why. Is the code untested because it's:

• New and tests haven't been written yet?
• Hard to test due to external dependencies?
• Legacy code that works but lacks tests?
• Dead code that should be deleted?

Step 3: Write Tests or Refactor. For hard-to-test code, refactor for testability. Extract dependencies, use dependency injection, or split complex functions.

Step 4: Add Tests. Focus on the happy path first, then edge cases, then error conditions. Each test should improve coverage and verify behavior.

Example - Improving Coverage:

Before (60% coverage):

def send_email_report(user_email, report_data):
    """Send report via email."""
    import smtplib  # Hard to test!

    server = smtplib.SMTP('smtp.gmail.com', 587)
    message = format_report(report_data)
    server.sendmail('noreply@example.com', user_email, message)
    server.quit()

After refactoring for testability (100% coverage possible):

def send_email_report(user_email, report_data, email_service=None):
    """Send report via email."""
    if email_service is None:
        email_service = EmailService()  # Default, but injectable

    message = format_report(report_data)
    email_service.send(user_email, message)

# Now easy to test with mock email_service
def test_send_email_report():
    mock_email = Mock()
    send_email_report('user@example.com', {'total': 100}, mock_email)
    mock_email.send.assert_called_once()

Dependency injection makes the code testable without hitting real SMTP servers.

Coverage Pitfalls to Avoid

Pitfall 1: Deleting Tests to Reduce Coverage Variance. Never delete working tests. If coverage fluctuates, investigate why—don't game the numbers.

Pitfall 2: Writing Tests Just for Coverage. Tests should verify behavior. If you're writing tests solely to increase coverage, you're doing it wrong.

Pitfall 3: Ignoring Branch Coverage. Line coverage alone misses untested paths. Always use branch coverage for accurate analysis.

Pitfall 4: Not Measuring Coverage Regularly. Run coverage in CI on every commit. Don't wait for the quarterly review to discover coverage dropped.

Pitfall 5: Setting Coverage Targets Too High or Too Low. Too high (95%+) wastes time on trivial code. Too low (60%) leaves important code untested. 80-85% is the sweet spot for most projects.

Coverage Doesn't Replace Code Review

Coverage tells you what code ran, not whether it works correctly. Even with 100% coverage, you still need:

Code Review: Human review catches logic errors, security issues, and design problems that tests miss.

Manual Testing: Some bugs only appear through user interaction. Coverage doesn't test usability or user experience.

Static Analysis: Tools like mypy, pylint, and bandit find issues coverage can't—type errors, code smells, security vulnerabilities.

Performance Testing: Coverage says nothing about speed or efficiency. Fast tests with good coverage don't guarantee fast production code.

Use coverage as one tool among many. Combine it with code review, static analysis, and other quality practices for robust software.

Course Recommendations

Advanced Python Testing

• Coverage analysis strategies
• Effective test design
• Quality metrics beyond coverage
• Enroll at paiml.com

Test-Driven Development Mastery

• TDD naturally achieves high coverage
• Writing tests that matter
• Coverage in TDD workflows
• Enroll at paiml.com

Software Quality Engineering

• Metrics and measurement
• Quality vs quantity in testing
• Building quality into processes
• Enroll at paiml.com

Quiz