Infranotes Gateway – Assessment and Recommendations (2025)

Status: implemented (advanced security features removed as of 2025-12-21)
Audience: platform/architecture, SRE, security, service owners

UPDATE (2025-12-21): The recommendation to remove advanced security features has been implemented. The gateway now follows a strict "thin gateway" architecture with advanced security features (fingerprinting, bot detection, challenges, threat intelligence) removed. See commit e611285 for details.

Context

The Go-based gateway (Gin) is a configurable L7 edge focused on routing, authN/Z, QoS, and observability. It supports two routing modes (static config and Kubernetes-native EndpointSlice discovery), JWT/JWKS verification, Redis-backed rate limiting, custom circuit breakers, multiple load-balancing algorithms, and a core security stack (CORS/headers/IP filtering/DDoS/content limits). Observability integrates DataDog, OpenTelemetry, and Prometheus counters.

Strengths

Clear separation of concerns and thin-gateway posture; no domain orchestration in-edge
JWT/JWKS implementation with key caching and algorithm allow-list
Robust rate limiting (sliding window, token bucket, fixed window) with Redis Lua, circuit breaker, and fail-open behavior
Circuit breaker and load balancer are feature-complete and instrumented; session affinity and health-aware strategies present
Kubernetes-native discovery (EndpointSlices) with cache invalidation and lock discipline
Structured logging, request IDs, distributed tracing injection, and backend spans with latency metrics

Key risks and improvement opportunities

Advanced security at edge: Fingerprinting, behavioral analysis, PoW/CAPTCHA challenges, and threat intel checks are powerful but heavy for a gateway. Default-off and path-scoped usage recommended; typically better in a dedicated edge/WAF tier [IMPLEMENTED 2025-12-21] - All advanced security features removed
Upload/content filtering: Filename-based allow-listing is weak; prefer MIME detection and ICAP/ClamAV scanning with size/time budgets
Telemetry duplication: Prometheus metrics coexist with DataDog/OTel; standardize via OpenTelemetry exporters to DataDog to reduce duplication
JWT validation policy: Ensure explicit time-based leeway, per-route audience/scope checks where backends differ; consider optional scope checks for sensitive routes
Admin/observability surface: /observability/*, /metrics/detailed, /service-discovery/* endpoints should be gated (auth/role or env guard) to avoid data exposure
Rate limiting fail-open: Preserve availability, but add an emergency in-process fallback (fixed window) if Redis/CB fail to avoid unlimited traffic during outages
Config hot-reload: Viper reloads config; document which settings are safe to hot-apply vs require restart; ensure thread-safe re-application (rate-limit rules, pools)
Discovery mode nuances: Static vs K8s routing behave differently; make migration explicit and guard K8s mode outside clusters (feature flag/env)
Header forwarding: Default deny for custom_headers; ensure no secrets/PII forwarded by accident; keep narrow allow-lists

Recommendations (actionable)

Default posture and scoping for advanced security [IMPLEMENTED 2025-12-21]

~~Keep gateway thin (routing, authN/Z, QoS, observability). Make advanced security modules (fingerprinting, bot detection, challenges, threat intel) opt-in and path-scoped via config/feature flags~~
~~Prefer a dedicated edge security tier (e.g., WAF/sidecar) for heavy features; gateway can consume decisions via headers or sidecar API~~
COMPLETED: Gateway now follows strict "thin gateway" architecture. All advanced security features removed (commit e611285). Gateway focuses on routing, authN/Z, QoS, and observability only.

Secure management/observability surfaces

Require JWT auth + role for /observability/*, /service-discovery/*, and any detailed metrics endpoints; optionally disable in production via env
Return reduced payloads (no PII/high-cardinality) and implement rate limits on these endpoints

JWT hardening and policy

Use an explicit parser with time leeway (clock_skew) and strict iss/aud checks
Add optional per-route audience/scope enforcement for backends that require different grants
Keep alg allow-list minimal (RS256, ES256) and HMAC only in development

Rate limiting resilience

Retain Redis-backed algorithms with CB and fail-open; add a lightweight in-process fixed-window fallback (global and per-IP) when Redis/CB are down
Continue to emit standard X-RateLimit-* headers in all outcomes; expose current mode (redis|fallback) in a debug header only in non-prod

Telemetry standardization

Adopt OpenTelemetry as the single instrumentation surface; export to DataDog using OTel exporters
Phase out direct Prometheus emission unless still scraped; otherwise, keep Prometheus only for exporter compatibility
Ensure end-to-end trace propagation (W3C traceparent); sample policies consolidated in one place

Upload/content security

Replace extension checks with MIME detection and integrate ICAP/ClamAV; enforce per-file and total payload budgets; apply per-path limits
Add response body size/time budgets for proxying large responses; keep current response-size middleware as a guardrail

Config reload policy and thread-safety

Document which settings are hot-reload-safe (rate-limit rules, headers, CORS) and which require restart (discovery mode, LB algorithm)
When hot-reloading rules/pools, use atomic swaps of data structures or guards to avoid race conditions

Discovery mode guardrails

Only enable K8s discovery when running in cluster or explicitly flagged; otherwise force static mode
Provide clear migration helpers between static and K8s (feature flag, readiness checks, logs)

Header forwarding defaults

Default-deny custom_headers; explicitly allowlist only the minimum required auth/trace headers per route
Log forwarded header names at debug level for audit in non-production

Load-balancer behavior convergence

Prefer K8s discovery pools for real multi-instance targeting; in static mode, clarify expectations (single backend vs pools)
Provide consistency notes in docs so operators know what to expect per mode

Phased plan and acceptance

Phase 0 — Guardrails (1 week)

Gate /observability/*, /metrics/detailed, /service-discovery/* behind JWT + role and/or env guards
Acceptance: Non-auth calls receive 401/403 in prod; minimal payloads returned; rate limits applied

Phase 1 — Telemetry unification (1–2 weeks)

Consolidate on OTel → DataDog; remove duplicate direct Prometheus emission where not required
Acceptance: Traces/metrics visible in DD; consistent trace propagation; SLO dashboards stable

Phase 2 — JWT policy (1 week)

Enforce leeway and strict iss/aud; add optional per-route audience/scope config
Acceptance: Sensitive routes reject tokens with wrong aud/scopes; regression tests pass

Phase 3 — Security posture (2 weeks)

Make advanced security modules default-off and path-scoped; improve upload security (MIME + ICAP/ClamAV)
Acceptance: No advanced modules active by default; upload scanning enforced on configured routes

Phase 4 — Resilience fallback (1 week)

Add in-process emergency rate limiter used only when Redis/CB fail; keep fail-open otherwise
Acceptance: Chaos test with Redis down limits traffic to safe caps; error budgets preserved

Phase 5 — Discovery and LB (1–2 weeks)

Harden K8s discovery gating; document migration; converge behavior notes between modes
Acceptance: Non-K8s envs never attempt K8s discovery; migration guide published

Operational notes

Continue to prefer 30s backend request timeouts; tune per-route when necessary
Keep X-RateLimit-* headers for all responses; include Retry-After on 429
Maintain structured JSON logging with request_id, correlation_id, and trace identifiers

Success metrics

Reduced exposure: management endpoints gated; no unauthenticated access in prod
Consistent telemetry: single exporter pipeline, coherent trace propagation and SLOs
Security: stronger upload protections; advanced modules only where needed; no PII or secrets forwarded unintentionally
Resilience: gateway keeps availability with bounded traffic during Redis/CB outages