Skip to main content

Test Execution

This guide covers how to effectively execute tests with Dochia, from planning and discovery to running tests and analyzing results. Learn how to use Dochia's systematic testing and fuzzing capabilities to thoroughly validate your APIs.

Overview​

Dochia provides two main testing approaches:

  • Systematic Testing (dochia test) - Comprehensive testing using predefined playbooks
  • Continuous Fuzzing (dochia fuzz) - Randomized testing for vulnerability discovery

Both approaches can be combined with result analysis and replay capabilities for complete API validation workflows.

Test Planning and Discovery​

Exploring Your API​

Before running tests, understand what you're testing:

# Discover available endpoints
dochia list --paths -c openapi.yml

# See available testing strategies
dochia list --playbooks

# Understand what formats are supported
dochia list --formats

Understanding Your OpenAPI Specification​

# Get detailed information about a specific endpoint
dochia list --paths -c openapi.yml --path "/api/users"

Planning Your Test Strategy​

Consider these factors when planning tests:

  1. API Maturity - New APIs benefit from comprehensive systematic testing
  2. Security Requirements - Critical APIs need both systematic and fuzz testing
  3. Performance Constraints - Use rate limiting and timeouts appropriately
  4. Environment - Different strategies for dev, staging, and production

Systematic Testing with dochia test​

Basic Test Execution​

Start with a simple test run to validate your setup:

# Basic test run
dochia test -c openapi.yml -s http://localhost:8080

# Test with authentication
dochia test -c openapi.yml -s http://localhost:8080 \
  -H "Authorization=Bearer $API_TOKEN"

Focused Testing​

Target specific areas of your API:

# Test specific endpoints
dochia test -c openapi.yml -s http://localhost:8080 \
  --paths "/users,/orders,/products"

# Test specific HTTP methods
dochia test -c openapi.yml -s http://localhost:8080 \
  --http-methods "POST,PUT,PATCH"

# Test endpoints with specific tags
dochia test -c openapi.yml -s http://localhost:8080 \
  --tags "user-management,billing"

Playbook Selection​

Choose appropriate testing strategies:

# Use specific playbooks for targeted testing
dochia test -c openapi.yml -s http://localhost:8080 \
  --playbooks "EmptyStringsInFields,NullValuesInFields"

# Skip problematic playbooks
dochia test -c openapi.yml -s http://localhost:8080 \
  --skip-playbooks "VeryLargeStrings"

# Test only field-related issues
dochia test -c openapi.yml -s http://localhost:8080 --fields-only

Advanced Configuration​

Configure tests for your specific environment:

# Production-safe testing with rate limiting
dochia test -c openapi.yml -s https://api.production.com \
  --max-requests-per-minute 60 \
  --max-response-time 10000 \
  --reference-data ./prod-reference.yml \
  --headers ./prod-headers.yml \
  --skip-deprecated-operations \
  -o ./prod-test-results

# Development testing with comprehensive coverage
dochia test -c openapi.yml -s http://localhost:8080 \
  --max-tested-fields 100 \
  --use-examples \
  --detailed-execution-stats \
  --execution-stats \
  -v

Continuous Fuzzing with dochia fuzz​

Basic Fuzzing​

Start fuzzing critical endpoints:

# Basic fuzzing for 5 minutes
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/users -X POST --st 300

# Stop after finding 5 errors
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/users -X POST --se 5

Targeted Fuzzing​

Focus on specific vulnerability types:

# Look for server errors
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/users -X POST \
  --mc "500,502,503" \
  --st 600

# Test for input reflection (potential XSS/injection)
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/search -X GET \
  --mi \
  --mr "script|javascript|<|>" \
  --se 3

Security-Focused Fuzzing​

Test for security vulnerabilities:

# Authentication bypass testing
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/admin/users -X POST \
  --mc "200,201" \
  --se 1 \
  -H "Authorization=Bearer invalid-token"

# Rate limiting testing
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/upload -X POST \
  --mc "429,503" \
  --max-requests-per-minute 1000 \
  --sm 100

Understanding Test Results​

Result Interpretation​

Dochia categorizes results into three types:

  • ✔ Success - API handled invalid input correctly
  • ⚠ Warning - Unexpected but not necessarily wrong response
  • ⨯ Error - API failed to handle invalid input properly

Common Result Scenarios​

Success Examples:

  • API returns 400 for invalid input
  • API returns 401 for missing authentication
  • API returns 422 for validation errors

Warning Examples:

  • Undocumented response codes that might be valid
  • Unexpected content types that don't break functionality
  • Performance issues that don't affect correctness

Error Examples:

  • 500 errors from invalid input (should return 4XX)
  • 2XX errors from bad input
  • Information leaks in error messages
  • Authentication bypass scenarios
  • Schema validation failures

Analyzing Results​

# View results in browser
open dochia-report/index.html

# Check specific test details
cat dochia-report/Test1.json | jq '.result'

# Count results by type
grep -r '"result"' dochia-report/ | cut -d'"' -f4 | sort | uniq -c

Result Analysis and Debugging​

Using the Explain Command​

When you encounter unfamiliar error reasons or response codes, use dochia explain to understand them:

# Explain error reasons you don't recognize
dochia explain --type error_reason "Not matching response schema"

# Explain custom Dochia response codes
dochia explain --type response_code 953

Debugging Failed Tests​

When tests fail, follow this debugging process:

# 1. Identify the failure
grep -r '"result":"ERROR"' dochia-report/ | head -5

# 2. Examine specific test details
cat dochia-report/Test5.json | jq '.'

# 3. Explain the error reason
ERROR_REASON=$(cat dochia-report/Test5.json | jq -r '.errorReason')
dochia explain --type error_reason "$ERROR_REASON"

# 4. Replay the test for debugging
dochia replay Test5 -v

Common Debugging Scenarios​

Schema Validation Failures:

# Replay with verbose output to see the mismatch
dochia replay Test5 -v

# Check if API contract is up to date
dochia list --paths -c openapi.yml --path "/api/users"

Authentication Issues:

# Test with fresh authentication
dochia replay Test5 -H "Authorization=Bearer $NEW_TOKEN"

Connection Problems:

# Retry with increased timeouts
dochia replay Test5 -s http://localhost:8080 --connection-timeout 30

Test Replay and Validation​

Replaying Tests​

Re-run specific tests for debugging or validation:

# Replay failed tests
dochia replay Test5,Test12,Test18

# Replay against different environment
dochia replay Test5 -s https://staging-api.example.com

# Replay with updated authentication
dochia replay Test5 -H "Authorization=Bearer $FRESH_TOKEN"

Environment Validation​

Test the same scenarios across environments:

# Test against multiple environments
for env in dev staging prod; do
  echo "Testing $env environment..."
  dochia replay Test1,Test2,Test3 \
    -s https://$env-api.example.com \
    -H "Authorization=Bearer ${env^^}_TOKEN" \
    --output ./results-$env
done

Regression Testing​

Validate fixes after development:

# Identify previously failed tests
FAILED_TESTS=$(grep -l '"result":"ERROR"' dochia-report/Test*.json | \
               sed 's/.*Test\([0-9]*\)\.json/Test\1/' | \
               tr '\n' ',' | sed 's/,$//')

# Replay failed tests after fixes
if [ -n "$FAILED_TESTS" ]; then
  dochia replay $FAILED_TESTS \
    --output ./regression-results \
    -v
fi

Complete Testing Workflows​

Development Workflow​

#!/bin/bash
# Complete development testing workflow

echo "=== API Discovery ==="
dochia list --paths -c openapi.yml

echo "=== Basic Validation ==="
dochia test -c openapi.yml -s http://localhost:8080 \
  --dry-run

echo "=== Systematic Testing ==="
dochia test -c openapi.yml -s http://localhost:8080 \
  --execution-stats \
  -o ./dev-results

echo "=== Security Fuzzing ==="
dochia fuzz -c openapi.yml -s http://localhost:8080 \
  -p /api/users -X POST \
  --mc "500" --st 300 \
  -o ./fuzz-results

echo "=== Result Analysis ==="
if grep -q '"result":"ERROR"' dev-results/*.json; then
  echo "Errors found, analyzing..."
  grep -h '"errorReason"' dev-results/*.json | \
    cut -d'"' -f4 | sort -u | \
    while read reason; do
      echo "Explaining: $reason"
      dochia explain --type error_reason "$reason"
    done
fi

CI/CD Integration Workflow​

#!/bin/bash
# CI/CD testing workflow

set -e

echo "=== Environment Setup ==="
export API_URL=${API_URL:-http://localhost:8080}
export API_TOKEN=${API_TOKEN:-""}

echo "=== Pre-flight Checks ==="
dochia info
dochia list --paths -c openapi.yml --json | jq '.numberOfPaths'

echo "=== Core Functionality Tests ==="
dochia test -c openapi.yml -s $API_URL \
  -H "Authorization=Bearer $API_TOKEN" \
  --tags "core,critical" \
  --max-requests-per-minute 120 \
  --max-response-time 5000 \
  --hide-success \
  -o ./ci-results

echo "=== Security Validation ==="
dochia fuzz -c openapi.yml -s $API_URL \
  -p /api/auth/login -X POST \
  --mc "200,201" --se 1 \
  -o ./security-results

echo "=== Result Validation ==="
ERROR_COUNT=$(grep -r '"result":"ERROR"' ci-results/ | wc -l)
if [ $ERROR_COUNT -gt 0 ]; then
  echo "❌ Found $ERROR_COUNT errors"
  exit 1
else
  echo "✅ All tests passed"
fi

Production Monitoring Workflow​

#!/bin/bash
# Production API monitoring workflow

echo "=== Production Health Check ==="
dochia test -c openapi.yml -s https://api.production.com \
  --paths "/health,/status" \
  --max-requests-per-minute 10 \
  --connection-timeout 30 \
  --max-response-time 10000 \
  -H "Authorization=Bearer $PROD_MONITOR_TOKEN" \
  -o ./prod-health

echo "=== Critical Path Validation ==="
dochia test -c openapi.yml -s https://api.production.com \
  --tags "critical" \
  --playbooks "EmptyStringsInFields,NullValuesInFields" \
  --max-requests-per-minute 30 \
  --reference-data ./prod-reference.yml \
  -o ./prod-critical

echo "=== Monitoring Report ==="
TOTAL_TESTS=$(find prod-* -name "Test*.json" | wc -l)
ERROR_COUNT=$(grep -r '"result":"ERROR"' prod-*/ 2>/dev/null | wc -l)
WARNING_COUNT=$(grep -r '"result":"WARN"' prod-*/ 2>/dev/null | wc -l)

echo "Total Tests: $TOTAL_TESTS"
echo "Errors: $ERROR_COUNT"
echo "Warnings: $WARNING_COUNT"

if [ $ERROR_COUNT -gt 0 ]; then
  echo "⚠️  Production issues detected"
  # Send alerts or notifications
fi

Best Practices​

Test Planning​

  1. Start Small - Begin with basic tests before comprehensive testing
  2. Know Your API - Use dochia list to understand available endpoints
  3. Environment Appropriate - Different strategies for different environments
  4. Incremental Testing - Build up test coverage gradually

Execution Strategy​

  1. Systematic First - Run dochia test before dochia fuzz
  2. Focus Areas - Use tags and path filtering for targeted testing
  3. Rate Limiting - Respect API limits, especially in production
  4. Authentication - Keep tokens fresh and environment-appropriate

Result Analysis​

  1. Understand Categories - Learn the difference between errors, warnings, and success
  2. Use Explain - Leverage dochia explain to understand results
  3. Replay for Debugging - Use dochia replay to investigate issues
  4. Document Findings - Keep track of recurring issues and patterns

Automation​

  1. Script Workflows - Automate common testing patterns
  2. CI/CD Integration - Include API testing in deployment pipelines
  3. Monitoring - Set up regular production health checks
  4. Alerting - Notify teams when critical issues are found

Troubleshooting Common Issues​

Connection Problems​

# Increase timeouts for slow networks
dochia test -c openapi.yml -s http://localhost:8080 \
  --connection-timeout 30 \
  --read-timeout 30 \
  --write-timeout 30

Authentication Failures​

# Verify token is valid
dochia replay Test5 -H "Authorization=Bearer $NEW_TOKEN"

Schema Validation Issues​

# Check if API contract is current
dochia list --paths -c openapi.yml --path "/problematic/endpoint"

Performance Issues​

# Adjust performance settings
dochia test -c openapi.yml -s http://localhost:8080 \
  --max-response-time 10000 \
  --max-requests-per-minute 60

Next Steps​

After mastering test execution, explore: