Dead Letter Queue (DLQ) and Multi-Level Retry Strategy

Overview

This document describes the multi-level retry strategy implemented for the Vendor Management & Procurement Microservice, following 2025 distributed systems best practices.

Architecture

Multi-Tier Retry Strategy

The system implements a sophisticated multi-tier retry strategy that adapts to different failure scenarios:

Message Fails
     ↓
Level 1: Fast Retries (1s delay, 3 attempts)
     ↓ (if still failing)
Level 2: Medium Retries (30s delay, 3 attempts)
     ↓ (if still failing)
Level 3: Slow Retries (5m delay, 2 attempts)
     ↓ (if still failing)
Level 4: Very Slow Retries (30m delay, 2 attempts)
     ↓ (if exhausted)
Dead Letter Queue (DLQ)

Retry Topics

Each retry level has its own Kafka topic for isolation and monitoring:

retry-retry-fast - Level 1: Immediate transient failures
retry-retry-medium - Level 2: Short-term unavailability
retry-retry-slow - Level 3: Persistent issues requiring longer backoff
retry-retry-very-slow - Level 4: Extended degradation scenarios
<original-topic>-dlq - Final DLQ for poison messages

Configuration

Basic Configuration

max_retries: 10
initial_delay: 1s
max_delay: 30m
backoff_multiple: 2.0
retry_topic_prefix: "retry"
dlq_topic_suffix: "dlq"

Retry Level Configuration

Each retry level can be customized:

retry_levels:
  - level: 1
    delay: 1s
    max_attempts: 3
    topic_suffix: "retry-fast"
    enable_jitter: true
    jitter_percent: 0.1
    circuit_breaker: false

Configuration Parameters

level: Retry tier number (1-N)
delay: Base delay before retry at this level
max_attempts: Maximum retry attempts at this level
topic_suffix: Kafka topic suffix for this level
enable_jitter: Add randomness to prevent thundering herd
jitter_percent: Percentage of delay to use for jitter (0.0-1.0)
circuit_breaker: Enable circuit breaker pattern at this level

Jitter Strategy

Jitter prevents the "thundering herd" problem where many messages retry simultaneously:

Level 1: 10% jitter (±100ms for 1s delay)
Level 2: 20% jitter (±6s for 30s delay)
Level 3: 30% jitter (±90s for 5m delay)
Level 4: 50% jitter (±15m for 30m delay)

Jitter is randomly added or subtracted from the base delay.

Error Categories

Messages are classified into categories that determine retry behavior:

Transient Errors

Network timeouts
Service temporarily unavailable
Rate limiting
Database connection issues

Action: Full retry through all levels

Business Errors

Validation failures
Authorization denied
Duplicate key violations
Business rule violations

Action: Immediate DLQ (no retries)

Deserialization Errors

Invalid JSON/Protobuf
Schema mismatch
Corrupt payload

Action: Immediate DLQ (no retries)

Poison Messages

Consistently failing messages
System invariant violations

Action: DLQ after Level 1 fast retry

Circuit Breaker

Circuit breaker is enabled at Level 3 and Level 4 to prevent cascading failures:

Threshold: 10 consecutive failures
Timeout: 30 seconds before attempting recovery
Half-Open State: Test with single message before full recovery

Monitoring & Observability

Metrics

The following metrics are tracked per retry level:

kafka.consumer.messages.retry - Total retry attempts
kafka.consumer.messages.dlq - Messages sent to DLQ
kafka.consumer.processing_time - Processing latency
kafka.consumer.errors - Errors by category and level

Alerting Thresholds

Level 1 > 100 retries/min: Transient network issues
Level 2 > 50 retries/min: Service degradation
Level 3 > 10 retries/min: Persistent outage
Level 4 > 5 retries/min: Extended failure requiring investigation
DLQ > 10 messages/hour: Poison messages or schema issues

Operational Procedures

Replaying DLQ Messages

Investigate root cause of DLQ messages
Fix underlying issue (schema, validation, etc.)
Replay messages from DLQ topic:

kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic vendor-events-dlq \
  --from-beginning \
  --group dlq-replay

Monitoring Retry Health

# Check retry topic lag
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --describe --group dlq-processor

# View retry metrics in Datadog
# Dashboard: "Kafka Consumer Health"
# Metric: kafka.consumer.lag by topic

Adjusting Retry Levels

For high-priority workflows, use aggressive retry configuration:

retry_levels:
  - level: 1
    delay: 500ms
    max_attempts: 5
  - level: 2
    delay: 5s
    max_attempts: 5

For batch processing, use conservative configuration:

retry_levels:
  - level: 1
    delay: 5s
    max_attempts: 2
  - level: 2
    delay: 2m
    max_attempts: 2

Best Practices

Message Design

Include idempotency keys for duplicate detection
Add correlation IDs for distributed tracing
Include timestamp for retry window enforcement
Store causation chain for debugging

Error Handling

Classify errors correctly (transient vs. permanent)
Log full error context for DLQ messages
Include stack traces for unexpected errors
Track error history for pattern detection

Configuration Tuning

Start with conservative defaults
Monitor retry success rates per level
Adjust delays based on observed recovery times
Increase max_attempts for known transient issues
Add new levels for specific failure patterns

Performance Characteristics

Throughput

Level 1: ~1000 msgs/sec (1s delay)
Level 2: ~100 msgs/sec (30s delay)
Level 3: ~10 msgs/sec (5m delay)
Level 4: ~1 msg/sec (30m delay)

Latency

P50: < 2s (Level 1 success)
P95: < 1 min (Level 2 success)
P99: < 10 min (Level 3 success)
P99.9: < 1 hour (Level 4 success)

Resource Usage

Kafka storage: ~1 KB per retry message
Redis (idempotency): ~500 bytes per message
Memory: ~10 MB per 10k inflight retries

Troubleshooting

High Retry Rate

Symptom: Increasing retry counts at all levels

Diagnosis:

Check target service health
Verify network connectivity
Review error logs for patterns
Check resource utilization

Resolution:

Scale target service if overloaded
Fix network issues
Adjust retry delays if recovering
Increase circuit breaker threshold temporarily

Messages Stuck in Retry

Symptom: Messages not progressing through retry levels

Diagnosis:

Check consumer group lag
Verify retry topic consumers running
Review consumer logs for errors
Check message timestamps

Resolution:

Restart consumers if hung
Clear retry window if expired
Manually move to DLQ if poison
Replay from earlier level if recoverable

DLQ Filling Up

Symptom: Rapid growth of DLQ messages

Diagnosis:

Sample DLQ messages for patterns
Check for schema changes
Review validation rules
Verify message formats

Resolution:

Fix schema incompatibilities
Update validation rules
Correct message producers
Replay after fixes deployed

References

Version: 1.0.0
Last Updated: 2025-10-09
Review Cycle: Quarterly