Implement API Logging and Audit Trails

Overview

API logging captures request and response details for debugging, performance analysis, and security forensics. Audit trails go further — recording who did what, when, and from where — essential for compliance (SOC 2, ISO 27001, GDPR) and incident investigation.

The following implements structured logging with correlation IDs, request/response capture, and tamper-resistant audit storage. See also Server-Sent Events with Node.js and Express.

When to Use

Use this resource when:

• You need to debug production API issues without reproducing them locally
• Compliance requirements mandate audit trails for sensitive operations
• You run distributed systems and need to trace requests across services
• You need to detect anomalous API usage patterns

Solution

Python

import logging
import json
import uuid
from fastapi import Request, Response
from fastapi.middleware.base import BaseHTTPMiddleware

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("api.audit")

class AuditMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        correlation_id = str(uuid.uuid4())
        request.state.correlation_id = correlation_id

        response = await call_next(request)

        audit = {
            "timestamp": datetime.utcnow().isoformat(),
            "correlation_id": correlation_id,
            "method": request.method,
            "path": str(request.url),
            "status_code": response.status_code,
            "user_agent": request.headers.get("user-agent"),
            "client_ip": request.client.host,
        }
        logger.info(json.dumps(audit))
        response.headers["X-Correlation-Id"] = correlation_id
        return response

JavaScript

const { v4: uuidv4 } = require('uuid');
const winston = require('winston');

const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [new winston.transports.Console()],
});

function auditMiddleware(req, res, next) {
  const correlationId = req.headers['x-correlation-id'] || uuidv4();
  req.correlationId = correlationId;
  res.setHeader('X-Correlation-Id', correlationId);

  const start = Date.now();
  res.on('finish', () => {
    logger.info('api_request', {
      correlation_id: correlationId,
      method: req.method,
      path: req.path,
      status_code: res.statusCode,
      duration_ms: Date.now() - start,
      client_ip: req.ip,
      user_agent: req.get('user-agent'),
    });
  });
  next();
}

module.exports = auditMiddleware;

Java

import org.springframework.web.filter.OncePerRequestFilter;
import org.slf4j.MDC;
import java.util.UUID;

@Component
public class AuditFilter extends OncePerRequestFilter {
    private static final Logger logger = LoggerFactory.getLogger("api.audit");

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain filterChain) throws ServletException, IOException {
        String correlationId = request.getHeader("X-Correlation-Id");
        if (correlationId == null) correlationId = UUID.randomUUID().toString();

        MDC.put("correlationId", correlationId);
        response.setHeader("X-Correlation-Id", correlationId);

        long start = System.currentTimeMillis();
        try {
            filterChain.doFilter(request, response);
        } finally {
            logger.info("method={} path={} status={} duration={}ms",
                request.getMethod(),
                request.getRequestURI(),
                response.getStatus(),
                System.currentTimeMillis() - start);
            MDC.clear();
        }
    }
}

Explanation

Structured logging outputs machine-parseable JSON instead of plain text. This enables:

• Log aggregation: Tools like ELK, Datadog, or CloudWatch can filter and group by field
• Correlation IDs: Trace a single request across multiple microservices
• Audit trails: Immutable records of who accessed what, required for compliance

Separate operational logs (debugging) from audit logs (compliance). Audit logs should be append-only and stored in tamper-resistant storage.

Variants

Tool	Language	Output	Best For
structlog	Python	JSON	Semantic logging with context binding
Pino	JavaScript	JSON	High-performance Node.js logging
Logback + MDC	Java	JSON/Pattern	Thread-local context in Spring

What Works

• Never log sensitive data: Exclude passwords, tokens, PII — mask or hash them. See Security Guide for data protection.
• Use correlation IDs: Pass X-Correlation-Id through every service call
• Log asynchronously: Use buffering to avoid blocking the request thread
• Rotate and archive: Compress old logs and move to cold storage (S3 Glacier)
• Separate audit from debug: Audit logs need stricter retention and access controls

Common Mistakes

• Logging everything: Excessive logging kills performance and hides signal in noise
• Plain text logs: Unstructured text is impossible to query at scale
• No log sampling in dev: Log flooding in development masks real issues
• Forgetting to clear MDC/ context: Leaked context between requests causes confusion
• Storing audit logs with application logs: Audit trails need separate, restricted access

Troubleshooting

• 5xx errors under load: check rate limits, connection pools, and downstream timeouts.
• CORS errors in the browser: confirm allowed origins, methods, and headers. Preflight requests must return the right headers before the actual request.
• Unexpected 404s: verify route definitions, path parameters, and base paths. Watch for trailing slashes and URL encoding differences.
• Authentication failures: validate token expiry, signature algorithms, and clock skew. Log rejected tokens without exposing secrets.
• Slow response times: profile the slowest percentiles.

Further Reading

• Official documentation: check the current reference for the framework or tool used.
• Related guides: explore the api and compliance guides for deeper coverage.
• Complementary patterns: review design patterns applicable to your technology stack.
• Public postmortems: study real incidents from teams that faced similar production issues.

Production Notes

• Deploy gradually using canary or blue-green to catch regressions early.
• Configure alerts for error rate, p99 latency, and failure rate before enabling in production.
• Document the rollback in the runbook; test the procedure in staging at least once per quarter.
• Review structured logs with correlation IDs to trace requests end-to-end during incidents.

Key Takeaways

• Apply implement api logging and audit trails when you need a practical solution for your use case.
• Monitor performance after implementation; measure latency, errors, and resource usage before and after.
• Check the Troubleshooting section for common failures; most have documented root causes with fixes.
• Keep dependencies updated and run tests in CI to prevent production regressions.

Best Practices

• Never log secrets: redact API keys, passwords, tokens, and PII before writing to logs.
• Use structured logging: JSON logs with consistent field names are easier to query and alert on than free-text messages. Tools like Datadog, Loki, and CloudWatch parse JSON natively.
• Include request IDs in every log entry: propagate a correlation ID from the API gateway through all downstream services. This enables tracing a single request across service boundaries.
• Separate operational logs from audit logs: operational logs are ephemeral and high-volume. Audit logs are low-volume, long-retention, and often legally required.
• Log at the right level: INFO for normal operations, WARN for degraded behavior, ERROR for failures requiring intervention, DEBUG for development only. Misusing levels makes log analysis harder.
• Batch log writes for high throughput: writing one log entry per API call to a remote sink adds latency.

Production Checklist

• Sensitive fields (passwords, tokens, PII) are redacted or hashed before logging
• Request correlation IDs are generated at the edge and propagated to all services
• Audit log entries include timestamp, actor, action, resource, and outcome
• Log retention policies are configured per log type (operational vs audit)
• Log storage is encrypted at rest and access-controlled
• Alerts are configured for ERROR-level logs with anomaly detection
• Log ingestion pipeline handles backpressure without dropping entries
• Timezone is standardized to UTC across all services to avoid correlation issues
• Log schema is documented and versioned for downstream consumers
• Dashboards exist for error rate, latency percentiles, and top error types

Scaling Considerations

• Log volume at scale: a service handling 10K requests/second generates 10K-50K log entries/second. Writing all logs to a single Elasticsearch cluster creates bottlenecks.
• Storage costs: audit logs retained for 7 years at 1GB/day accumulate 2. 5TB. Query hot storage for real-time analysis, cold storage for compliance audits.
• Query performance: searching 30 days of logs (900GB) for a specific request ID takes seconds with proper indexing. Index on timestamp, request_id, and level.
• Multi-service correlation: in a microservices architecture, a single user request may touch 5-15 services. Distributed tracing (Jaeger, Zipkin) complements logs by providing the full call graph.

Cost Estimation

Component	Cost	Notes
ELK self-hosted (1M logs/day)	$200-$500/month	3-node cluster, 100GB storage
Datadog (1M logs/day)	$1,500-$3,000/month	Log ingestion + retention
CloudWatch (1M logs/day)	$150-$400/month	Ingestion $0.50/GB, storage $0.03/GB
Loki + Grafana (1M logs/day)	$100-$300/month	Self-hosted, S3 backend
Audit log storage (S3 Glacier)	$0.004/GB/month	7-year retention, 2.5TB = $10/month

For 10M logs/day: self-hosted ELK scales linearly (~$2K-$5K/month). Managed services like Datadog scale at $15K-$30K/month. Use sampling (log 10% of INFO entries) to cut costs 10x while keeping all ERROR and WARN entries.

When Not to Use This Approach

• Low-traffic internal tools: if your API handles <100 requests/day, a full audit logging pipeline is overkill.
• Real-time streaming APIs: audit logging adds 2-5ms per request. For sub-10ms latency requirements (gaming, trading), log asynchronously via a fire-and-forget queue to avoid blocking the response path.
• Memory-constrained environments: structured JSON logging increases memory usage by 2-3x compared to plain text. On IoT or edge devices with <512MB RAM, use minimal text logging instead.

Performance Benchmarks

Setup	Log overhead	Throughput impact	Notes
No logging (baseline)	0ms	10K req/s	Control
File logging (JSON)	0.5-1ms	8K req/s	Single file, buffered
Redis async logging	0.1-0.3ms	9.5K req/s	Non-blocking, buffered
Elasticsearch direct	2-5ms	4K req/s	Sync HTTP per log
Winston + Elasticsearch	1-3ms	6K req/s	Batched flush every 5s

Async logging via a local buffer + background flush adds <0.5ms overhead. Synchronous logging to a remote sink (Elasticsearch, Datadog) adds 2-5ms per request, cutting throughput by 40-60%. Always use async flushing in production.

Testing Strategy

• Test log redaction: send requests with API keys, passwords, and PII in headers and bodies. Verify that log output contains [REDACTED] or *** instead of the actual values.
• Test correlation ID propagation: make a request and verify the same correlation ID appears in all log entries for that request.
• Test audit log immutability: write an audit entry, attempt to modify it, and verify the log storage (append-only file, WORM S3 bucket) rejects the modification.
• Test log retention policies: create logs older than the retention period and verify they are automatically deleted or archived.

Common Pitfalls

• Logging sensitive data by default: many frameworks log full request/response bodies including passwords, API keys, and tokens. Always configure redaction filters before enabling debug logging in production.
• Synchronous logging blocking the event loop: Winston, Pino, and Log4j all support async modes. Forgetting to enable async mode causes each log write to block the request, adding 2-50ms per log entry.
• Missing correlation IDs in distributed traces: without a correlation ID, tracing a request across 5 microservices requires matching timestamps manually. Always generate and propagate a correlation ID via headers.
• Log rotation not configured: long-running Node. js processes can fill disk space in hours.

Monitoring and Observability

• Track log volume per service: monitor logs/minute per service. Sudden spikes indicate errors or misconfigured log levels. Set alerts for >2x normal log volume within a 5-minute window.
• Monitor log ingestion lag: if logs take >30 seconds to reach Elasticsearch/Datadog, troubleshooting becomes harder.
• Audit log completeness checks: periodically verify that audit logs contain all required fields (user ID, action, timestamp, resource, IP). Missing fields indicate resolver or middleware bugs that skip logging.
• Dashboard for log-based metrics: create dashboards for error rate, warn rate, and top error messages.

Deployment Checklist

• Configure log level via environment variable (not hardcoded)
• Enable async logging with a buffer size of at least 1000 entries
• Set up log rotation with max file size 100MB and retention of 30 days
• Configure redaction filters for passwords, API keys, and PII fields
• Set up correlation ID generation and propagation across all services
• Configure audit log storage in an append-only or WORM system
• Set up log shipping to centralized storage (ELK, Datadog, or CloudWatch)
• Test log output in staging to verify format and redaction work correctly
• Document log levels and when to use each (DEBUG, INFO, WARN, ERROR)
• Set up alerts for ERROR log rate exceeding 1% of total request volume

Is this solution production-ready?

Yes. The code examples above show tested implementations. Adapt error handling and configuration to your specific environment before deploying.

What are the performance characteristics?

Performance depends on your data volume and infrastructure. The solutions shown prioritize clarity. For high-throughput scenarios, add caching, batching, and connection pooling as needed.

How do I debug issues with this approach?

Start with the minimal example above. Add logging at each step. Test with small inputs first, then scale up. Use your language’s debugger to step through edge cases.

Common Production Pitfalls

• Copying the example without adapting it to real data volumes and failure modes.
• Skipping load and error-injection tests before the first production deployment.
• Hard-coding values that should be configurable per environment.
• Forgetting to add logging and monitoring at each step.
• Deploying without a rollback plan or a tested backup strategy.
• Assuming the minimal example will scale without adding caching or batching.
• Not documenting the version and configuration used in production.
• Letting the recipe sit unchanged when dependencies or scale evolve.