Reporting and Result Analysis
This guide covers how to understand, analyze, and customize Dochia's test reports. Learn how to interpret test results, work with different output formats, and integrate reporting into your development workflows.
Overview
Dochia generates comprehensive reports in multiple formats:
- HTML Reports - Interactive web-based reports for detailed analysis
- JSON Results - Machine-readable data for automation and integration
- Console Output - Real-time feedback during test execution
- Custom Formats - Configurable output for specific needs
Report Structure and Organization
Default Output Structure
When you run tests, Dochia creates organized output:
dochia-report/
├── index.html # Main HTML report
├── Test1.json # Individual test results
├── Test2.json
├── Test3.json
├── ...
└── assets/ # CSS, JS, images for HTML report
Custom Output Directories
Control where reports are generated:
# Specify custom output directory
dochia test -c openapi.yml -s http://localhost:8080 \
-o ./custom-results
# Organize by date and environment
dochia test -c openapi.yml -s http://localhost:8080 \
-o "./reports/$(date +%Y-%m-%d)/staging"
# Separate systematic and fuzz results
dochia test -c openapi.yml -s http://localhost:8080 \
-o ./systematic-results
dochia fuzz -c openapi.yml -s http://localhost:8080 \
-p /api/users -X POST \
-o ./fuzz-results
Understanding Test Results
Result Categories
Dochia categorizes each test into three result types:
✅ SUCCESS - API handled invalid input correctly
- Returns appropriate 4XX status codes for bad input
- Validates input and rejects malformed requests
- Maintains security boundaries
⚠️ WARNING - Unexpected but potentially valid behavior
- Undocumented response codes that might be legitimate
- Performance issues that don't affect functionality
- Minor schema deviations
❌ ERROR - API failed to handle invalid input properly
- Returns 5XX errors for invalid input (should be 4XX)
- Leaks sensitive information in error messages
- Allows unauthorized access or operations
- Violates OpenAPI specification
Result Details
Each test result contains comprehensive information:
{
"testId": "Test 1",
"traceId": "33d8094c-f052-40c3-b858-6d1ebf627ddf",
"scenario": "Send [abugidas chars] in headers",
"expectedResult": "Should return [4XX]",
"result": "error",
"resultReason": "Unexpected response code: 406",
"resultDetails": "Response does NOT match expected result...",
"request": {
"httpMethod": "GET",
"url": "http://localhost:8080/payments/events",
"headers": [...],
"payload": "{}",
"timestamp": "Wed, 13 Aug 2025 23:47:24 +0300"
},
"response": {
"responseCode": 406,
"responseTimeInMs": "5",
"headers": [...],
"jsonBody": {...}
},
"path": "/payments/events",
"playbook": "AbugidasInHeaders",
"server": "http://localhost:8080"
}
HTML Report Analysis
Interactive Web Reports
The HTML report provides rich interactive analysis:
# Generate and open HTML report
dochia test -c openapi.yml -s http://localhost:8080
open dochia-report/index.html
Key Report Sections
Executive Summary
- Total tests run, success/warning/error counts
- Overall API health score
- Test execution time and coverage
Results by Category
- Filter by SUCCESS, WARNING, ERROR
- Search for specific error patterns
Endpoint Analysis
- Results grouped by API endpoint
- Identify problematic paths
- See which playbooks found issues
Filtering and Navigation
Use the HTML report's interactive features:
# The HTML report includes:
# - Filter buttons for result types
# - Search functionality for specific errors
# - Expandable details for each test
# - Links to replay specific tests
JSON Data Analysis
Working with Individual Results
Analyze specific test results:
# View a specific test result
cat dochia-report/Test5.json | jq '.'
# Extract key information
cat dochia-report/Test5.json | jq '{
testId: .testId,
result: .result,
resultReason: .resultReason,
endpoint: .path,
method: .request.httpMethod
}'
# Check response details
cat dochia-report/Test5.json | jq '.response | {
responseCode: .responseCode,
responseTime: .responseTimeInMs,
contentType: .responseContentType,
contentLength: .contentLengthInBytes
}'
Bulk Analysis with Command Line Tools
Analyze patterns across all results:
# Count results by type
grep -r '"result"' dochia-report/ | cut -d'"' -f4 | sort | uniq -c
# Find all errors
find dochia-report/ -name "Test*.json" -exec grep -l '"result":"error"' {} \;
# List unique error reasons
grep -r '"resultReason"' dochia-report/ | cut -d'"' -f4 | sort -u
# Count errors by endpoint
grep -r '"result":"error"' dochia-report/ | \
xargs -I {} sh -c 'cat {} | jq -r .path' | \
sort | uniq -c | sort -nr
# Find tests that took longest
find dochia-report/ -name "Test*.json" -exec jq -r '
"\(.testId): \(.response.responseTimeInMs // "N/A")ms"
' {} \; | sort -k2 -nr | head -10
Advanced JSON Analysis
Use jq for complex analysis:
# Create summary by endpoint
find dochia-report/ -name "Test*.json" | \
xargs jq -s 'group_by(.path) | map({
endpoint: .[0].path,
total: length,
errors: map(select(.result == "error")) | length,
warnings: map(select(.result == "warn")) | length,
success: map(select(.result == "success")) | length
})'
# Find patterns in error messages
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "error") | .resultReason' | \
sort | uniq -c | sort -nr
# Analyze response times by result type
find dochia-report/ -name "Test*.json" | \
xargs jq -r '"\(.result) \(.response.responseTimeInMs // 0)"' | \
awk '{
sum[$1] += $2; count[$1]++
} END {
for (type in sum) print type ": " sum[type]/count[type] "ms avg"
}'
Custom Report Generation
Creating Custom Summaries
Generate tailored reports for different audiences:
#!/bin/bash
# Executive summary script
REPORT_DIR="dochia-report"
TOTAL_TESTS=$(find $REPORT_DIR -name "Test*.json" | wc -l)
ERRORS=$(grep -r '"result":"error"' $REPORT_DIR/ | wc -l)
WARNINGS=$(grep -r '"result":"warn"' $REPORT_DIR/ | wc -l)
SUCCESS=$(grep -r '"result":"success"' $REPORT_DIR/ | wc -l)
echo "=== API Testing Executive Summary ==="
echo "Total Tests: $TOTAL_TESTS"
echo "✅ Success: $SUCCESS ($(( SUCCESS * 100 / TOTAL_TESTS ))%)"
echo "⚠️ Warnings: $WARNINGS ($(( WARNINGS * 100 / TOTAL_TESTS ))%)"
echo "❌ Errors: $ERRORS ($(( ERRORS * 100 / TOTAL_TESTS ))%)"
echo ""
if [ $ERRORS -gt 0 ]; then
echo "=== Top Error Reasons ==="
grep -r '"resultReason"' $REPORT_DIR/ | cut -d'"' -f4 | \
sort | uniq -c | sort -nr | head -5
echo ""
fi
echo "=== Most Problematic Endpoints ==="
grep -r '"result":"error"' $REPORT_DIR/ | \
xargs -I {} sh -c 'cat {} | jq -r .path' | \
sort | uniq -c | sort -nr | head -5
Security-Focused Reports
Create reports highlighting security issues:
#!/bin/bash
# Security analysis script
REPORT_DIR="dochia-report"
echo "=== Security Analysis Report ==="
# Check for information leaks
echo "🔍 Information Leak Analysis:"
LEAK_COUNT=$(grep -r '"resultReason":"Error details leak"' $REPORT_DIR/ | wc -l)
echo " - Error detail leaks: $LEAK_COUNT"
# Check for authentication bypasses
echo "🔐 Authentication Analysis:"
AUTH_BYPASSES=$(find $REPORT_DIR -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and (.response.responseCode == 200 or .response.responseCode == 201)) | .testId' | wc -l)
echo " - Potential auth bypasses: $AUTH_BYPASSES"
# Check for server errors from bad input
echo "💥 Error Handling Analysis:"
SERVER_ERRORS=$(find $REPORT_DIR -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and (.response.responseCode >= 500)) | .testId' | wc -l)
echo " - Server errors from bad input: $SERVER_ERRORS"
# List specific security issues
if [ $LEAK_COUNT -gt 0 ] || [ $AUTH_BYPASSES -gt 0 ] || [ $SERVER_ERRORS -gt 0 ]; then
echo ""
echo "🚨 Security Issues Found:"
if [ $LEAK_COUNT -gt 0 ]; then
echo " Information Leaks:"
grep -r '"resultReason":"Error details leak"' $REPORT_DIR/ | \
xargs -I {} sh -c 'cat {} | jq -r " - \(.request.httpMethod) \(.path)"' | head -5
fi
if [ $AUTH_BYPASSES -gt 0 ]; then
echo " Potential Authentication Bypasses:"
find $REPORT_DIR -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and (.response.responseCode == 200 or .response.responseCode == 201)) |
" - \(.request.httpMethod) \(.path) (Status: \(.response.responseCode))"' | head -5
fi
fi
Performance Analysis Reports
Analyze API performance characteristics:
#!/bin/bash
# Performance analysis script
REPORT_DIR="dochia-report"
echo "=== Performance Analysis Report ==="
# Response time analysis
echo "⏱️ Response Time Analysis:"
find $REPORT_DIR -name "Test*.json" | \
xargs jq -r 'select(.response.responseTimeInMs) | .response.responseTimeInMs' | \
awk '{
sum += $1; count++;
if ($1 > max) max = $1;
if (min == 0 || $1 < min) min = $1
} END {
if (count > 0) {
print " - Average: " sum/count "ms"
print " - Min: " min "ms"
print " - Max: " max "ms"
}
}'
# Slow endpoints
echo ""
echo "🐌 Slowest Endpoints:"
find $REPORT_DIR -name "Test*.json" | \
xargs jq -r 'select(.response.responseTimeInMs) |
"\(.response.responseTimeInMs) \(.request.httpMethod) \(.path)"' | \
sort -nr | head -5 | \
awk '{print " - " $2 " " $3 ": " $1 "ms"}'
# Timeout analysis
echo ""
echo "⏰ Timeout Analysis:"
TIMEOUT_COUNT=$(grep -r '"resultReason":"Response time exceeds max"' $REPORT_DIR/ | wc -l)
echo " - Timeouts: $TIMEOUT_COUNT"
if [ $TIMEOUT_COUNT -gt 0 ]; then
echo " Endpoints with timeouts:"
grep -r '"resultReason":"Response time exceeds max"' $REPORT_DIR/ | \
xargs -I {} sh -c 'cat {} | jq -r " - \(.request.httpMethod) \(.path)"' | \
sort -u | head -5
fi
Integration with Development Workflows
CI/CD Pipeline Integration
Generate reports suitable for automation:
#!/bin/bash
# CI/CD reporting script
REPORT_DIR="dochia-report"
OUTPUT_FILE="test-summary.json"
# Generate machine-readable summary
TOTAL_TESTS=$(find $REPORT_DIR -name "Test*.json" | wc -l)
ERRORS=$(grep -r '"result":"error"' $REPORT_DIR/ | wc -l)
WARNINGS=$(grep -r '"result":"warn"' $REPORT_DIR/ | wc -l)
SUCCESS=$(grep -r '"result":"success"' $REPORT_DIR/ | wc -l)
# Create JSON summary for CI/CD
cat > $OUTPUT_FILE << EOF
{
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"summary": {
"total": $TOTAL_TESTS,
"success": $SUCCESS,
"warnings": $WARNINGS,
"errors": $ERRORS,
"successRate": $(echo "scale=2; $SUCCESS * 100 / $TOTAL_TESTS" | bc)
},
"status": "$([ $ERRORS -eq 0 ] && echo "PASS" || echo "FAIL")"
}
EOF
echo "Summary written to $OUTPUT_FILE"
# Set exit code for CI/CD
[ $ERRORS -eq 0 ] && exit 0 || exit 1
Slack/Teams Integration
Send reports to team channels:
#!/bin/bash
# Slack notification script
REPORT_DIR="dochia-report"
WEBHOOK_URL="$SLACK_WEBHOOK_URL"
TOTAL_TESTS=$(find $REPORT_DIR -name "Test*.json" | wc -l)
ERRORS=$(grep -r '"result":"error"' $REPORT_DIR/ | wc -l)
WARNINGS=$(grep -r '"result":"warn"' $REPORT_DIR/ | wc -l)
SUCCESS=$(grep -r '"result":"success"' $REPORT_DIR/ | wc -l)
# Choose emoji and color based on results
if [ $ERRORS -eq 0 ]; then
EMOJI="✅"
COLOR="good"
STATUS="PASSED"
else
EMOJI="❌"
COLOR="danger"
STATUS="FAILED"
fi
# Send Slack notification
curl -X POST -H 'Content-type: application/json' \
--data "{
\"attachments\": [{
\"color\": \"$COLOR\",
\"title\": \"$EMOJI API Test Results - $STATUS\",
\"fields\": [
{\"title\": \"Total Tests\", \"value\": \"$TOTAL_TESTS\", \"short\": true},
{\"title\": \"Success\", \"value\": \"$SUCCESS\", \"short\": true},
{\"title\": \"Warnings\", \"value\": \"$WARNINGS\", \"short\": true},
{\"title\": \"Errors\", \"value\": \"$ERRORS\", \"short\": true}
],
\"footer\": \"Dochia API Testing\",
\"ts\": $(date +%s)
}]
}" \
$WEBHOOK_URL
Git Integration
Track test results in version control:
#!/bin/bash
# Git integration script
REPORT_DIR="dochia-report"
RESULTS_DIR="test-results"
# Create results archive
mkdir -p $RESULTS_DIR
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
COMMIT_HASH=$(git rev-parse --short HEAD)
ARCHIVE_NAME="results-$COMMIT_HASH-$TIMESTAMP"
# Copy essential results
mkdir -p "$RESULTS_DIR/$ARCHIVE_NAME"
# Create lightweight summary
TOTAL_TESTS=$(find $REPORT_DIR -name "Test*.json" | wc -l)
ERRORS=$(grep -r '"result":"error"' $REPORT_DIR/ | wc -l)
cat > "$RESULTS_DIR/$ARCHIVE_NAME/README.md" << EOF
# Test Results for Commit $COMMIT_HASH
- **Date**: $(date)
- **Commit**: $COMMIT_HASH
- **Total Tests**: $TOTAL_TESTS
- **Errors**: $ERRORS
- **Status**: $([ $ERRORS -eq 0 ] && echo "✅ PASS" || echo "❌ FAIL")
## Error Summary
$(if [ $ERRORS -gt 0 ]; then
echo "### Top Error Reasons"
grep -r '"resultReason"' $REPORT_DIR/ | cut -d'"' -f4 | sort | uniq -c | sort -nr | head -5
fi)
EOF
# Add to git (optional)
# git add "$RESULTS_DIR/$ARCHIVE_NAME"
# git commit -m "Add test results for $COMMIT_HASH"
Report Customization and Filtering
Filtering Results by Criteria
Focus on specific types of issues:
# Show only errors
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "error") |
"\(.testId): \(.request.httpMethod) \(.path) - \(.resultReason)"'
# Show results for specific endpoints
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.path | startswith("/api/users")) |
"\(.result): \(.request.httpMethod) \(.path)"'
# Show results from specific playbooks
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.playbook == "EmptyStringsInFieldsPlaybook") |
"\(.result): \(.path) - \(.resultReason // "Success")"'
# Show only high-severity issues
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and
(.resultReason | contains("leak") or contains("bypass") or contains("500"))) |
"\(.testId): \(.resultReason)"'
Creating Custom Views
Generate focused reports for different stakeholders:
#!/bin/bash
# Developer-focused report
echo "=== Developer Report ==="
echo "Focus: Schema validation and API contract compliance"
echo ""
# Schema validation issues
echo "📋 Schema Validation Issues:"
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.resultReason | contains("schema")) |
" - \(.request.httpMethod) \(.path): \(.resultReason)"' | \
head -10
echo ""
# Contract violations
echo "📄 API Contract Violations:"
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and
(.resultReason | contains("Undocumented") or contains("specification"))) |
" - \(.request.httpMethod) \(.path): \(.resultReason)"' | \
head -10
#!/bin/bash
# Security team report
echo "=== Security Team Report ==="
echo "Focus: Security vulnerabilities and information leaks"
echo ""
# Security issues
echo "🔒 Security Issues:"
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and
(.resultReason | contains("leak") or contains("bypass") or
(.response.responseCode == 200 and .path | contains("admin")))) |
" - \(.request.httpMethod) \(.path): \(.resultReason)"'
echo ""
# Authentication issues
echo "🔐 Authentication Issues:"
find dochia-report/ -name "Test*.json" | \
xargs jq -r 'select(.result == "ERROR" and
(.response.responseCode == 200 or .response.responseCode == 201) and
(.request.headers.Authorization // "" | contains("invalid"))) |
" - \(.request.httpMethod) \(.path): Potential auth bypass"'
Best Practices for Reporting
Report Organization
- Consistent Naming - Use descriptive output directory names
- Time-based Organization - Include timestamps in report paths
- Environment Separation - Keep dev/staging/prod results separate
- Archive Management - Regularly clean up old reports
Analysis Workflow
- Start with Summary - Review overall success/failure rates
- Focus on Errors - Prioritize fixing ERROR results over warnings
- Group by Pattern - Look for common error reasons or endpoints
- Track Progress - Compare results across test runs
Automation Integration
- Machine-Readable Output - Generate JSON summaries for automation
- Threshold-Based Alerts - Set up notifications for error counts
- Trend Analysis - Track metrics over time
- Integration Points - Connect with existing monitoring systems
Team Communication
- Stakeholder-Specific Reports - Tailor content for different audiences
- Actionable Insights - Focus on what needs to be fixed
- Context Preservation - Include enough detail for debugging
- Regular Updates - Share progress and improvements
Advanced Reporting Techniques
Trend Analysis
Track changes over time:
#!/bin/bash
# Trend analysis script
RESULTS_BASE="./test-results"
CURRENT_REPORT="dochia-report"
echo "=== API Testing Trend Analysis ==="
# Compare with previous runs
if [ -d "$RESULTS_BASE" ]; then
LATEST_ARCHIVE=$(ls -1t $RESULTS_BASE/ | head -1)
if [ -n "$LATEST_ARCHIVE" ] && [ -f "$RESULTS_BASE/$LATEST_ARCHIVE/README.md" ]; then
echo "Comparing with previous run: $LATEST_ARCHIVE"
# Current stats
CURRENT_ERRORS=$(grep -r '"result":"error"' $CURRENT_REPORT/ | wc -l)
CURRENT_TOTAL=$(find $CURRENT_REPORT -name "Test*.json" | wc -l)
PREV_ERRORS=$(grep "Errors:" "$RESULTS_BASE/$LATEST_ARCHIVE/README.md" | grep -o '[0-9]*' || echo "0")
PREV_TOTAL=$(grep "Total Tests:" "$RESULTS_BASE/$LATEST_ARCHIVE/README.md" | grep -o '[0-9]*' || echo "0")
# Calculate changes
ERROR_CHANGE=$((CURRENT_ERRORS - PREV_ERRORS))
TOTAL_CHANGE=$((CURRENT_TOTAL - PREV_TOTAL))
echo "📊 Changes since last run:"
echo " - Total tests: $CURRENT_TOTAL ($([ $TOTAL_CHANGE -ge 0 ] && echo "+")$TOTAL_CHANGE)"
echo " - Errors: $CURRENT_ERRORS ($([ $ERROR_CHANGE -ge 0 ] && echo "+")$ERROR_CHANGE)"
if [ $ERROR_CHANGE -lt 0 ]; then
echo " ✅ Error count decreased!"
elif [ $ERROR_CHANGE -gt 0 ]; then
echo " ⚠️ Error count increased"
else
echo " ➡️ Error count unchanged"
fi
fi
fi
Multi-Environment Comparison
Compare results across environments:
#!/bin/bash
# Multi-environment comparison
echo "=== Multi-Environment Comparison ==="
ENVIRONMENTS=("dev" "staging" "prod")
for env in "${ENVIRONMENTS[@]}"; do
REPORT_DIR="./reports/$env/dochia-report"
if [ -d "$REPORT_DIR" ]; then
TOTAL=$(find $REPORT_DIR -name "Test*.json" | wc -l)
ERRORS=$(grep -r '"result":"error"' $REPORT_DIR/ | wc -l)
SUCCESS_RATE=$(echo "scale=1; ($TOTAL - $ERRORS) * 100 / $TOTAL" | bc)
echo "🌍 $env environment:"
echo " - Tests: $TOTAL"
echo " - Errors: $ERRORS"
echo " - Success Rate: $SUCCESS_RATE%"
echo ""
fi
done
Report Archival and Cleanup
Manage report storage:
#!/bin/bash
# Report archival script
ARCHIVE_DIR="./archived-reports"
CURRENT_REPORT="dochia-report"
RETENTION_DAYS=30
# Create archive
mkdir -p $ARCHIVE_DIR
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
COMMIT_HASH=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
# Archive current report
if [ -d "$CURRENT_REPORT" ]; then
tar -czf "$ARCHIVE_DIR/report-$COMMIT_HASH-$TIMESTAMP.tar.gz" $CURRENT_REPORT
echo "Archived report to: report-$COMMIT_HASH-$TIMESTAMP.tar.gz"
fi
# Cleanup old archives
find $ARCHIVE_DIR -name "report-*.tar.gz" -mtime +$RETENTION_DAYS -delete
echo "Cleaned up archives older than $RETENTION_DAYS days"
Troubleshooting Report Issues
Common Report Problems
Missing or Empty Reports
# Check if tests actually ran
ls -la dochia-report/
echo "Test files found: $(find dochia-report/ -name "Test*.json" | wc -l)"
# Verify output directory permissions
ls -ld dochia-report/
Incomplete JSON Data
# Check for malformed JSON files
find dochia-report/ -name "Test*.json" | while read file; do
if ! jq . "$file" >/dev/null 2>&1; then
echo "Malformed JSON: $file"
fi
done
HTML Report Not Loading
# Check for missing assets
ls -la dochia-report/assets/
# Verify HTML file exists and is readable
ls -la dochia-report/index.html
Performance Optimization
For large test suites:
# Use parallel processing for analysis
find dochia-report/ -name "Test*.json" | \
xargs -P 4 -I {} jq -r '.result' {} | \
sort | uniq -c
# Create indexed summaries for faster access
find dochia-report/ -name "Test*.json" | \
xargs jq -c '{testId, result, endpoint: .path}' > results-index.jsonl
Next Steps
After mastering reporting and analysis, explore:
- Configuration - Advanced configuration options
- Test Execution - Comprehensive testing workflows
- CI/CD Integration - Automating reports in pipelines
- Best Practices - Advanced analysis strategies