Spring AI Canary

Overview

Ship it. Let the AI watch it. Let the AI decide if it lives or dies.

Progressive Canary Lab is a production-style progressive delivery system built entirely in Java 21 and Spring Boot 3 — but with a twist: the rollout controller doesn’t run on hardcoded thresholds. It’s an AI agent, powered by Spring AI and GPT-4o-mini, that reads live traffic metrics every 10 seconds and autonomously decides whether to promote the canary, hold, or roll everything back.

Problem

Traditional canary systems bake rollout logic into static if/else thresholds — “roll back if error rate > 5%” — which are brittle, need constant tuning, and can’t reason about context (a 2% error rate on 3 requests means something very different than on 3,000). Teams need a rollout mechanism that can weigh multiple signals together, explain its reasoning, and adapt its decisions the way an on-call engineer would.

What I Built

• A six-service multi-module Spring Boot system: app-service (stable + canary instances), flag-service (in-memory feature flags with a kill switch), router-service (traffic-splitting reverse proxy with sliding-window metrics), rollout-controller (the AI decision engine), and loadgen for simulating traffic
• An AI agent (CanaryAiAgent) that polls live error rate, p99 latency, and request counts for both stable and canary every 10 seconds, sends a structured prompt to GPT-4o-mini at temperature=0, and parses a deterministic ACTION / NEXT_PCT / REASON decision
• A traffic router that starts canary at 1% and lets the AI progressively promote it through 1 → 5 → 25 → 50 → 100%, with manual override endpoints for rollout and rollback
• A feature-flag service gating a hidden fraud-detection code path, with instant kill-switch and rollout-percent controls independent of traffic routing
• FNV-1a hash-based request bucketing (via a shared library) for consistent, deterministic traffic splitting
• Fully Dockerized services with Docker Compose orchestration, Kubernetes manifests under infra/k8s, and Makefile-driven demo flows for both healthy and unhealthy canary scenarios

Key Outcomes

• Replaced brittle static thresholds with an LLM that reasons over live metrics and states its logic in plain English — e.g. “Canary p99 812ms far exceeds safe threshold, rolling back immediately”
• Demonstrated a clean end-to-end healthy promotion path (1% → 100%) driven purely by AI decisions, alongside a reproducible unhealthy scenario where the agent detects degraded latency and auto-rolls back within seconds
• Reinforced the same production distinction that matters at scale: a traffic rollback fixes routing, a feature kill switch fixes behavior — and the AI agent triggers both together during an incident
• Built a fully observable system — /metrics, /flags, and /health endpoints plus live agent decision logs — so every AI action is auditable, not a black box