[{"content":"Mar 2026\nGitHub repository\nFinds the straggler in a distributed data-parallel training job and explains why it is slow. One slow rank makes every other rank wait at every collective; the symptom (step time went up) says nothing about the cause. Flight Recorder joins per-rank collective timing from the PyTorch NCCL flight recorder with per-GPU NVML telemetry, names the straggling rank, and ranks likely causes (thermal or power throttle, ECC errors, degraded PCIe/NVLink, GPU contention, host stall) with the evidence behind each.\nGo agents stream per-rank events over gRPC to a coordinator that correlates collectives by (process_group, sequence_id), so no cross-node clock synchronization is needed Status: early build. The agent, ring buffer, correlator, attribution engine and gRPC transport exist; there are no validation results yet. Results will be published, with hardware named, once a captured run exists.\n","permalink":"https://ryanjhamby.dev/projects/gpu-flight-recorder/","summary":"\u003cp\u003e\u003cstrong\u003eMar 2026\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/RyanJHamby/distributed-gpu-training-flight-recorder\"\u003eGitHub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFinds the straggler in a distributed data-parallel training job and explains why it is slow. One slow rank makes every other rank wait at every collective; the symptom (step time went up) says nothing about the cause. Flight Recorder joins per-rank collective timing from the PyTorch NCCL flight recorder with per-GPU NVML telemetry, names the straggling rank, and ranks likely causes (thermal or power throttle, ECC errors, degraded PCIe/NVLink, GPU contention, host stall) with the evidence behind each.\u003c/p\u003e","title":"GPU Flight Recorder"},{"content":"Mar 2026 - Oct 2026\nGitHub repository\nRust-accelerated temporal join engine for point-in-time ML feature pipelines: join trades, quotes, and signals without look-ahead bias, in batch or streaming.\nRust as-of join engine with O(n+m) merge-scan kernels, zero-copy Arrow via the PyCapsule interface, and lock-free UnsafeCell writes across Rayon workers Streaming incremental join with watermark-based emission, per-symbol state, and configurable lateness tolerance; streaming output verified identical to batch 132 Rust tests, including property-based checks of every kernel against reference implementations ","permalink":"https://ryanjhamby.dev/projects/flowstate/","summary":"\u003cp\u003e\u003cstrong\u003eMar 2026 - Oct 2026\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/RyanJHamby/FlowState\"\u003eGitHub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eRust-accelerated temporal join engine for point-in-time ML feature pipelines: join trades, quotes, and signals without look-ahead bias, in batch or streaming.\u003c/p\u003e\n\u003cdiv class=\"achievement\"\u003e\nRust as-of join engine with O(n+m) merge-scan kernels, zero-copy Arrow via the PyCapsule interface, and lock-free \u003ccode\u003eUnsafeCell\u003c/code\u003e writes across Rayon workers\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nStreaming incremental join with watermark-based emission, per-symbol state, and configurable lateness tolerance; streaming output verified identical to batch\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\n\u003cspan class=\"metric\"\u003e132 Rust tests\u003c/span\u003e, including property-based checks of every kernel against reference implementations\n\u003c/div\u003e","title":"FlowState"},{"content":"Sep 2025 - Sep 2026\nGitHub repository\nMinimal, high-performance order book engine in C++ designed to explore the foundations of low-latency trading systems. Includes nanosecond-level order matcher skeleton, lock-free queues, thread-local memory pools, and automated benchmarking on AWS EC2 spot instances.\nBuilt C++20 order matching engine with lock-free SPSC ingestion, thread-local slab pools, and inlined matching. On an isolated EC2 c6i.large core over 1M orders: P50 0.21 µs, P99.9 3.1–3.2 µs, 1.4–1.8M orders/sec Automated EC2 spot benchmarking with cloud-init for reproducible, low-cost latency profiling Latency and throughput figures are from the repo README (EC2 c6i.large, isolated core, 3 runs), not a pow run capture.\nArchitecture Order Matching Core:\nMinimal memory allocation design for sub-microsecond latency Inlined matching logic for CPU cache efficiency Lock-free queue skeleton for order ingestion Thread-local memory pools to reduce contention Benchmarking Pipeline:\nSimulates 1M+ orders with alternating buy/sell sides Measures average microseconds per match Reproducible results on local machine or EC2 Separation of Concerns:\nOrder ingestion (queue) → matching engine → latency measurement Decoupled architecture mirrors real exchange pipelines Local Development Build \u0026amp; Run:\ncd orderbook-engine chmod +x scripts/build_and_run_benchmark.sh ./scripts/build_and_run_benchmark.sh This script:\nCleans and creates fresh build directory Configures with CMake and builds with parallel jobs Runs validation executable Executes latency benchmark → build/benchmark_results.txt Unit Tests:\nchmod +x scripts/run_tests.sh ./scripts/run_tests.sh Test coverage:\nOrder creation, type validation, comparison operations Order addition, matching, stress testing (1000 orders) Lock-free queue push/pop operations Thread-local memory pool allocation (2000 stress allocations) Cloud Benchmarking Automated EC2 Spot Execution:\nchmod +x scripts/run_benchmark.sh ./scripts/run_benchmark.sh Process:\nLaunches EC2 spot instance with specified AMI/instance type Cloud-init installs dependencies (build-essential, cmake, git, perf) Clones repository and executes benchmark Logs results to /home/ubuntu/benchmark_results.txt Automatically shuts down instance after completion Benefits: Repeatable benchmarking at low cost with cloud infrastructure consistency.\nDesign Insights Sub-microsecond latency goal: Minimal allocations, cache-aware layouts, inlined hot paths Lock-free architecture: Eliminates mutex contention in order ingestion pipeline EC2 spot automation: Reduces operational overhead while maintaining reproducibility Optimization roadmap: SIMD vectorization for price matching NUMA-aware memory allocation for multi-threaded workloads Order cancellations and partial fill simulation Latency spike detection and metrics logging ","permalink":"https://ryanjhamby.dev/projects/order-book-engine/","summary":"\u003cp\u003e\u003cstrong\u003eSep 2025 - Sep 2026\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/RyanJHamby/order-book-engine\"\u003eGitHub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eMinimal, high-performance order book engine in C++ designed to explore the foundations of low-latency trading systems. Includes nanosecond-level order matcher skeleton, lock-free queues, thread-local memory pools, and automated benchmarking on AWS EC2 spot instances.\u003c/p\u003e\n\u003cdiv class=\"achievement\"\u003e\nBuilt C++20 order matching engine with lock-free SPSC ingestion, thread-local slab pools, and inlined matching. On an isolated EC2 \u003ccode\u003ec6i.large\u003c/code\u003e core over 1M orders: \u003cspan class=\"metric\"\u003eP50 0.21 µs, P99.9 3.1–3.2 µs, 1.4–1.8M orders/sec\u003c/span\u003e\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nAutomated EC2 spot benchmarking with cloud-init for reproducible, low-cost latency profiling\n\u003c/div\u003e\n\u003cp\u003e\u003cem\u003eLatency and throughput figures are from the repo README (EC2 \u003ccode\u003ec6i.large\u003c/code\u003e, isolated core, 3 runs), not a \u003ccode\u003epow run\u003c/code\u003e capture.\u003c/em\u003e\u003c/p\u003e","title":"Low-Latency Order Book Engine"},{"content":"Nov 2025 - May 2026\nGitHub repository\nFully automated stock screening system scanning 3,800+ US stocks daily using Mark Minervini\u0026rsquo;s Trend Template methodology. Identifies high-conviction buy/sell signals via phase-based trend classification, relative strength momentum, and fundamental quality filters.\nAutomated daily scans of 3,800+ stocks with 74% API reduction via Git-based fundamental caching and GitHub Actions automation Phase-based classification identifying Phase 2 uptrend breakouts passing 7 of 8 Minervini criteria (50\u003e150\u003e200 SMA alignment, relative strength ≥70, volume confirmation) Risk-managed position management with max 10% risk, min 2:1 R:R ratio, automated stop-loss trailing, and tax-aware filtering for long-term positions Figures on this project are from the repo README, not a captured run.\nCore Features Phase-Based Trend Classification:\nPhase 1 (Base): Consolidation after decline Phase 2 (Uptrend): Confirmed uptrend with 50\u0026gt;150\u0026gt;200 SMA alignment — BUY ZONE Phase 3 (Distribution): Topping pattern with weakening momentum Phase 4 (Downtrend): Declining trend — AVOID Volatility Contraction Pattern (VCP):\nIdentifies price consolidation before explosive breakout Formula: Range_i = ((High_i - Low_i) / Close_i) × 100 VCP_Score = (∑ Contractions / Total_Periods) × Trend_Quality Example: 13 contractions (6.5% → 5.8% → 7.6% → 6.0%) with 54/100 quality score indicates imminent breakout Intelligent Market Regime Filtering:\nOnly generates buy signals when SPY in Phase 1/2 with ≥15% of stocks in Phase 2 Avoids low-probability trades in declining markets Sell signals always generated (can exit any market) Smart Caching Strategy:\nFundamentals stored in Git repository as JSON with metadata Earnings season (6 weeks): refresh if \u0026gt;7 days old Normal periods: refresh if \u0026gt;90 days old Result: 74% fewer API calls, 15-20 min faster scans Sample Output View sample scan output (Jan 9, 2026) OPTIMIZED FULL MARKET SCAN - ALL US STOCKS Scan Date: 2026-01-09 Generated: 2026-01-09 14:04:49 SCANNING STATISTICS Total Universe: 3,819 stocks Analyzed: 1,188 stocks Buy Signals: 370 | Sell Signals: 111 Error Rate: 0.26% SPY Trend: Phase 2 - Uptrend (Bullish) • 50 SMA: $678.39 (slope: 0.0563) • 200 SMA: $626.72 (slope: 0.0982) • Confidence: 85% Market Breadth (n=1,188): • Phase 1 (Base): 560 (47.1%) • Phase 2 (Uptrend): 504 (42.4%) • Phase 3 (Distribution): 1 (0.1%) • Phase 4 (Downtrend): 123 (10.4%) Market Regime: RISK-ON (Strong) BUY #1: WWD | Score: 104.5/110 Phase: 2 | Entry Quality: Good | R:R: 3.2:1 Stop Loss: $299.25 • 11.1% above 50 SMA (strong breakout) • Revenue: +16.5% YoY | EPS: +64% YoY • Volume confirmation (1.17 ratio) ","permalink":"https://ryanjhamby.dev/projects/stock-screener/","summary":"\u003cp\u003e\u003cstrong\u003eNov 2025 - May 2026\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/RyanJHamby/stock-screener\"\u003eGitHub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eFully automated stock screening system scanning 3,800+ US stocks daily using Mark Minervini\u0026rsquo;s Trend Template methodology. Identifies high-conviction buy/sell signals via phase-based trend classification, relative strength momentum, and fundamental quality filters.\u003c/p\u003e\n\u003cdiv class=\"achievement\"\u003e\nAutomated daily scans of \u003cspan class=\"metric\"\u003e3,800+ stocks\u003c/span\u003e with \u003cspan class=\"metric\"\u003e74% API reduction\u003c/span\u003e via Git-based fundamental caching and GitHub Actions automation\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nPhase-based classification identifying Phase 2 uptrend breakouts passing 7 of 8 Minervini criteria (50\u003e150\u003e200 SMA alignment, relative strength ≥70, volume confirmation)\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nRisk-managed position management with \u003cspan class=\"metric\"\u003emax 10% risk, min 2:1 R:R ratio\u003c/span\u003e, automated stop-loss trailing, and tax-aware filtering for long-term positions\n\u003c/div\u003e\n\u003cp\u003e\u003cem\u003eFigures on this project are from the repo README, not a captured run.\u003c/em\u003e\u003c/p\u003e","title":"Intelligent Stock Screener"},{"content":"Aug 2023 - May 2026\nGitHub repository\nSystematic ES futures trading system powered by eigendecomposition of economic indicator covariance matrices. Decomposes 8×8 macro factor covariance to identify uncertainty regimes and size positions via macro surprise exposure.\nBacktested on real FRED data over 17 years (2007–2024): 0.28 Sharpe, 2.10% annualized return, 3.2% max drawdown with half-Kelly, regime-adjusted position sizing C++ factor decomposition engine (Eigen 3.4.0) processing 8 FRED indicators + VIX with AWS Lambda daily automation End-to-end pipeline: FRED API → monthly frequency alignment (downsampling/Hermite interpolation) → spectral decomposition (Σ = UΛUᵀ) → S3, covered by 240+ unit tests Core Hypothesis Markets respond to unexpected economic information, not levels. Raw covariance is misleading—it captures expected relationships.\nStandard assumption: E[r_t | X_t] = α + βX_t\nReality: Markets react to surprises: ΔX_t - E_t[ΔX_t]\nSolution: Decompose surprise covariance via PCA to identify 3-4 interpretable macro factors (growth, inflation, policy, volatility) and measure ES exposure across regimes.\n5-Stage Architecture Data Acquisition: FRED API (CPI, GDP, Unemployment, Sentiment, Fed Funds, Treasury yields) + VIX Temporal Alignment: Daily → month-end LOCF, Quarterly → Hermite interpolation to monthly Covariance Estimation: Σ_ij = (1/(n-1)) ∑_t (X_it - X̄_i)(X_jt - X̄_j) Spectral Decomposition: Σ = UΛUᵀ → First 3-4 eigenvalues capture 85-92% variance Regime Quantification: Frobenius norm ‖Σ‖_F = √(∑_i,j σ²_ij) maps to regime state Regime Classification:\nStable (‖Σ‖_F \u0026lt; μ - σ): Low uncertainty, tight co-movement Neutral (μ - σ ≤ ‖Σ‖_F ≤ μ + σ): Moderate uncertainty Elevated (‖Σ‖_F \u0026gt; μ + σ): High uncertainty, factor decoupling Infrastructure AWS Stack:\nEventBridge (cron: daily 12:00 UTC) → Lambda (512MB, 600s) → S3 (versioned, encrypted) Native C++ binary with Eigen, AWS SDK, curl IaC: AWS CDK 2.x (TypeScript, dev/staging/prod) Validation:\n240+ unit tests: covariance symmetry, eigenvalue ordering, numerical stability, Sharpe/drawdown/P\u0026amp;L accuracy 17-year backtest (2007–2024, 468 trades): 0.28 Sharpe, 3.23% max drawdown, 2.10% annualized return, 87.2% win rate (selectivity, not edge); NASDAQCOM proxy for 2007–2016 equity returns; figures from the repo README, not a captured run Data Inputs 8 economic indicators at mixed frequencies:\nCPI (monthly) - Inflation anchor Real GDP (quarterly) - Earnings driver Unemployment (monthly) - Labor tightness Consumer Sentiment (monthly) - Forward demand Fed Funds (monthly) - Discount rate 10Y/2Y Treasury (daily) - Yield curve VIX (daily) - Realized volatility ","permalink":"https://ryanjhamby.dev/projects/macro-trading-system/","summary":"\u003cp\u003e\u003cstrong\u003eAug 2023 - May 2026\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/RyanJHamby/macro-factor-decomposition\"\u003eGitHub repository\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eSystematic ES futures trading system powered by eigendecomposition of economic indicator covariance matrices. Decomposes 8×8 macro factor covariance to identify uncertainty regimes and size positions via macro surprise exposure.\u003c/p\u003e\n\u003cdiv class=\"achievement\"\u003e\nBacktested on real FRED data over \u003cspan class=\"metric\"\u003e17 years (2007–2024)\u003c/span\u003e: \u003cspan class=\"metric\"\u003e0.28 Sharpe, 2.10% annualized return, 3.2% max drawdown\u003c/span\u003e with half-Kelly, regime-adjusted position sizing\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nC++ factor decomposition engine (Eigen 3.4.0) processing 8 FRED indicators + VIX with AWS Lambda daily automation\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nEnd-to-end pipeline: FRED API → monthly frequency alignment (downsampling/Hermite interpolation) → spectral decomposition (Σ = UΛUᵀ) → S3, covered by \u003cspan class=\"metric\"\u003e240+ unit tests\u003c/span\u003e\n\u003c/div\u003e\n\u003ch2 id=\"core-hypothesis\"\u003eCore Hypothesis\u003c/h2\u003e\n\u003cp\u003eMarkets respond to \u003cstrong\u003eunexpected\u003c/strong\u003e economic information, not levels. Raw covariance is misleading—it captures expected relationships.\u003c/p\u003e","title":"Covariance-Based Macro Trading System"},{"content":"Introduction There\u0026rsquo;s a version of this story where Kubernetes \u0026ldquo;came back.\u0026rdquo; That framing is wrong.\nKubernetes never left—it lost the narrative. Between 2020 and 2023, the conversation was dominated by YAML-hell complaints, cluster complexity horror stories, and a genuine question: is this worth it for most teams? The backlash was real. But the adoption curve was not affected. Production usage kept climbing even as the discourse soured.\nNow, in 2026, the framing has shifted entirely. Kubernetes is the de facto substrate for running inference workloads at scale. The 2026 CNCF survey puts production adoption at 82%—not \u0026ldquo;emerging,\u0026rdquo; not \u0026ldquo;early majority.\u0026rdquo; It\u0026rsquo;s baseline infrastructure, like TCP/IP. You don\u0026rsquo;t get excited about TCP/IP. You also can\u0026rsquo;t run a distributed system without it.\nThe question in 2026 isn\u0026rsquo;t \u0026ldquo;should we use Kubernetes?\u0026rdquo; It\u0026rsquo;s two more precise questions: What does Kubernetes actually do in the AI stack? And when does the added complexity actually pay off versus a simpler Docker deployment?\nThose are the questions this post answers.\nHistorical Context: How We Got Here Kubernetes came out of Google\u0026rsquo;s internal Borg system in 2014. The core problem Borg solved was operational at scale: Google was running hundreds of thousands of tasks across massive clusters, and manual bin-packing was impossible. You needed a system that could look at a fleet of machines, figure out where to place a job given CPU, memory, and locality constraints, and handle failures automatically.\nThe open-source version—Kubernetes—inherited that design philosophy but applied it to a world of containers. The early years (2015–2018) were genuinely rough. The API surface was enormous, networking was tribal knowledge, and RBAC was arcane. \u0026ldquo;Just use Kubernetes\u0026rdquo; was advice that came with an implicit weeks-long onboarding tax.\nWhat changed by 2020 was managed Kubernetes. EKS, GKE, and AKS absorbed most of the control-plane complexity. You still wrote YAML, but you stopped racking and stacking etcd clusters. The operational floor dropped significantly.\nThen came the inference wave.\nIn 2024–2025, the AI industry crossed a threshold: the hard, expensive problem shifted from training large models to running them at scale. Training a frontier model is something maybe a dozen organizations do. Running inference against that model—serving millions of requests against expensive GPU hardware—is a problem every organization building with AI faces.\nThis is where Kubernetes\u0026rsquo; architecture became decisive.\nCore Mechanics: What Kubernetes Actually Does for AI GPU Scheduling and the Bin-Packing Problem An NVIDIA H100 costs roughly $30,000 and runs ~$2.60/hour in cloud. If you\u0026rsquo;re running a fleet of AI agents or inference endpoints, leaving those GPUs idle is not a financial abstraction—it\u0026rsquo;s a real number on a real invoice.\nThe challenge is that AI workloads are bursty and heterogeneous. A \u0026ldquo;Research Agent\u0026rdquo; might need 40GB of VRAM for complex chain-of-thought reasoning. A \u0026ldquo;Summarizer Agent\u0026rdquo; might only need 12GB. Without orchestration, you either over-provision (wasting money) or under-provision (hitting OOM errors).\nKubernetes solves this with the resources specification and, more importantly, with support for Multi-Instance GPU (MIG) partitioning:\nSingle H100 (80GB) → partitioned via MIG: ┌─────────────────────────────────────────┐ │ Agent Pod A │ Agent Pod B │ │ (3g.40gb MIG) │ (1g.20gb MIG x2) │ │ VRAM: 40GB │ VRAM: 10GB each │ └─────────────────────────────────────────┘ Without MIG (single Docker container): ┌─────────────────────────────────────────┐ │ Agent Pod A │ │ (entire GPU) VRAM: 80GB allocated │ │ VRAM: ~40GB actually │ │ used -- 50% waste │ └─────────────────────────────────────────┘ With MIG and Kubernetes\u0026rsquo; nvidia.com/mig-3g.40gb resource requests, the scheduler can bin-pack multiple agents onto a single GPU. A 2024 NVIDIA benchmarking study showed MIG can improve effective GPU utilization from ~40% to ~85% for mixed inference workloads. At $2.60/hour per GPU, that delta compounds fast across a fleet.\nThe Self-Healing Property for Long-Running Agents Traditional web services are stateless and short-lived. A request comes in, a response goes out, the container resets. The failure mode is well-understood: a crash loses one request, the pod restarts, traffic routes elsewhere.\nAI agents in 2026 don\u0026rsquo;t fit this model. Agentic pipelines—where a model is given a task, tools, and autonomy to complete a multi-step workflow over hours—are stateful and long-running. An OOM event halfway through a 4-hour research task is not a minor inconvenience; it\u0026rsquo;s a lost work unit.\nKubernetes\u0026rsquo; health probe system handles this differently than a simple docker run --restart=always:\nK8s Health Probe Sequence (Long-Running Agent): Agent Pod kubelet Scheduler | | | | [Running: Step 3/12] | | | | | | OOM kill (step 5) | | |\u0026lt;--- liveness probe-----| | | [fails] | | | |--- mark Unhealthy -------\u0026gt;| | | | |\u0026lt;--- terminate ----------| | | | | | |--- schedule replacement -\u0026gt;| | | | | [New pod on healthy | | | node, step 1/12] | | The critical detail is \u0026ldquo;healthy node.\u0026rdquo; If the original node is under memory pressure, Kubernetes will schedule the replacement pod on a different node. docker restart restarts on the same machine—if that machine is the problem, you\u0026rsquo;re in a loop.\nKEDA and Scale-to-Zero KEDA (Kubernetes Event-Driven Autoscaling) is the mechanism that makes Kubernetes economically viable for bursty AI workloads. The core idea: instead of keeping agent pods running 24/7, you scale them to zero when no work is queued and spin them up when a trigger fires.\nA concrete example: suppose you have a \u0026ldquo;Document Processor Agent\u0026rdquo; that runs when a PDF lands in an S3 bucket. KEDA watches the S3 queue depth:\nQueue Depth → KEDA Decision → Pod Count 0 → scale to 0 → 0 pods ($0 GPU cost) 1-5 → scale up → 2 pods 6-20 → scale up → 5 pods 21+ → scale up → 10 pods (max) In a flat Docker deployment, you keep one container running continuously. If your agent processes 500 PDFs per month—roughly 17/day—that idle container is burning GPU-hours for the other 23 hours daily. At GPU-class pricing, that\u0026rsquo;s real money for what is fundamentally waiting.\nKEDA supports scale triggers from Kafka, SQS, RabbitMQ, Prometheus metrics, and HTTP request rate. The startup latency for a typical inference container (pulling model weights from cache) is 15–30 seconds, which is acceptable for async queue-backed work.\nThe \u0026ldquo;Invisible Kubernetes\u0026rdquo; Shift The reason the complaint volume dropped isn\u0026rsquo;t that Kubernetes got simpler. It\u0026rsquo;s that most developers stopped touching it directly.\nPlatform Engineering is the discipline that emerged to put a usable interface in front of Kubernetes. The pattern: a small platform team (3–6 engineers) runs the cluster, configures security policies, and builds Internal Developer Portals (IDPs) on top of tools like Backstage or Cortex. Application developers interact with a service catalog—they fill out a form with their container image, resource requirements, and environment variables. The IDP generates the Kubernetes manifests and applies them.\nThe developer never writes a Deployment YAML. They never touch kubectl. The cognitive overhead of Kubernetes is absorbed by the platform layer.\nAmazon\u0026rsquo;s EKS Auto Mode, released in late 2025, takes this further at the managed service level. Auto Mode manages node provisioning, patching, and scaling automatically. You submit a pod spec; EKS figures out which EC2 instance type to provision, when to scale in, and how to apply OS patches without disrupting running workloads. It\u0026rsquo;s not quite \u0026ldquo;serverless Kubernetes,\u0026rdquo; but it removes enough of the operational surface that small teams can run production clusters without dedicated SRE headcount.\nThe practical implication: the Kubernetes skill gap is bifurcating. You either need deep expertise (platform/infra engineers who understand control-plane internals, CNI plugins, and admission controllers) or surface-level familiarity (application engineers who understand how to write a Deployment spec and read a pod status). The \u0026ldquo;middle\u0026rdquo; expertise—knowing kubectl well but not deeply—is becoming less relevant.\nTechnical Re-Engineering: eBPF and Wasm Two technologies are changing Kubernetes\u0026rsquo; internal architecture in ways that matter for resource efficiency.\neBPF and the Sidecar Tax The traditional service mesh pattern (Istio, Linkerd) works by injecting a sidecar proxy container into every pod. The proxy handles mTLS, traffic policy, observability. The problem: every pod now runs two containers, and the sidecar is not free. Estimates from the Istio project and independent benchmarks put the sidecar overhead at 10–15% of CPU and memory per pod, plus added latency on every network call.\neBPF (Extended Berkeley Packet Filter) eliminates this by moving network policy and observability into the Linux kernel itself. Cilium\u0026rsquo;s implementation, and Istio\u0026rsquo;s Ambient Mode (released for production use in 2024), both use eBPF to implement service mesh features without sidecars.\nThe performance delta is meaningful:\nSidecar Mesh (Istio classic): Pod = App Container + Envoy Proxy Memory overhead: ~50-100MB per pod Latency added: ~2-5ms per request (proxy hops) CPU overhead: ~100-200m per pod eBPF Mesh (Cilium / Ambient Mode): Pod = App Container only Memory overhead: ~5-10MB (kernel, shared) Latency added: ~0.1-0.3ms (kernel bypass) CPU overhead: ~10-30m (kernel, shared) For a cluster running 200 pods, eliminating sidecars recovers roughly 20GB of RAM and significant compute headroom. That headroom can run more inference workloads.\nWebAssembly and the Container Alternative WebAssembly (Wasm) modules running in Kubernetes via WASI (WebAssembly System Interface) represent a more speculative but real shift. A Wasm module cold-starts in milliseconds versus the 2–5 seconds for a typical container. Memory footprint can be 10–100x smaller depending on the runtime.\nIn 2026, Wasm is in production for specific workloads—primarily lightweight pre/post-processing pipelines, not heavy inference. The toolchain (Wasmtime, SpinKube) has matured enough that teams are using it at the edges of their AI pipelines. The pattern is a hybrid: a Wasm module handles request validation and preprocessing, hands off to a container running the actual model, and a Wasm module post-processes the response. The GPU-bound inference step stays in a container; the surrounding logic doesn\u0026rsquo;t have to.\nWhen Docker Wins All of the above is real. It\u0026rsquo;s also operational complexity that many teams don\u0026rsquo;t need.\nThree scenarios where staying on Docker is the correct engineering decision:\nPrototyping and early architecture. If you\u0026rsquo;re still iterating on your agent\u0026rsquo;s core reasoning loop, RAG pipeline, or tool-calling strategy, Docker Compose gives you a full local environment in minutes:\nservices: agent: image: my-agent:dev depends_on: [redis, vectordb] redis: image: redis:7 vectordb: image: milvus/milvus:2.4 You can iterate on agent behavior, swap vector databases, and change tool implementations without a cluster in the loop. Kubernetes adds nothing here except friction.\nEdge and constrained deployments. The Kubernetes control plane (API server, etcd, scheduler, controller-manager) consumes 2–4GB of RAM at minimum even for a single-node cluster. On a factory floor device or edge inference server with 8GB total RAM, that overhead is unacceptable. Docker or containerd with a simple systemd service unit is the right tool.\nSmall teams with predictable workloads. If you have two engineers, three internal users, and one agent with steady request volume, Kubernetes RBAC configuration, network policies, and upgrade management are liabilities without corresponding benefits. The operational surface is a maintenance burden your team doesn\u0026rsquo;t have capacity to manage well. Managing a Kubernetes cluster poorly is worse than not using Kubernetes at all.\nThe 2026 heuristic that holds up empirically: under 5 engineers and under 3 agents, stay on Docker. It\u0026rsquo;s not a hard line, but it\u0026rsquo;s a reasonable prior.\nPractical Comparison Feature Docker / Compose Kubernetes ───────────────────────────────────────────────────────────────── Setup time Minutes Days to weeks GPU efficiency Poor (full-GPU per ctn) High (MIG, slicing) Self-healing Manual / restart=always Automated, node-aware Scale to zero No Yes (KEDA) Agent networking Docker bridge / hosts DNS-based service disc. Control plane cost None 2-4GB RAM min Operational burden Low Medium to High Best for Dev, edge, MVPs Production, multi-agent ───────────────────────────────────────────────────────────────── The GPU efficiency row is the decisive one for AI workloads. At scale, poor GPU utilization translates directly to cost. If you\u0026rsquo;re managing a fleet of inference workloads and aren\u0026rsquo;t doing MIG or time-slicing, you\u0026rsquo;re likely paying 30–50% more than you need to.\nReal-World Adoption Data The adoption numbers are not soft marketing metrics. The 2026 CNCF annual survey—2,800+ respondents across engineering organizations—shows:\n82% production Kubernetes adoption (up from 71% in 2023) 66% of organizations running generative AI inference use Kubernetes to manage those workloads 54% are using or evaluating Platform Engineering / IDPs to abstract Kubernetes from application developers eBPF-based networking (Cilium or equivalent) is now the default in 44% of new cluster deployments, up from 12% in 2022 The 66% inference number is the one worth sitting with. It means that of all the organizations actually running inference pipelines in production, nearly two-thirds chose Kubernetes to manage it. That\u0026rsquo;s a strong signal that the GPU orchestration and reliability properties are driving real decisions, not just architectural preferences.\nWhat This Means for Engineers in 2026 If you\u0026rsquo;re building AI-backed products, here\u0026rsquo;s the practical framing:\nUnderstand the platform, even if you don\u0026rsquo;t operate it. Even if your team uses an IDP that generates Kubernetes manifests, understanding the underlying primitives—Deployments, Services, resource limits, liveness probes—lets you debug production issues that the abstraction layer won\u0026rsquo;t surface clearly. You need to read a pod spec, not write it from scratch.\nGPU scheduling is where the money is. If you\u0026rsquo;re paying for GPU infrastructure, instrument your utilization. If it\u0026rsquo;s consistently below 60%, investigate MIG partitioning or time-slicing. The engineering time to set this up is often recovered in the first month\u0026rsquo;s bill.\nKEDA is underused. Scale-to-zero for async, queue-backed agent workloads is well-understood technology in 2026 and can cut idle GPU costs significantly. If your agents run on triggers rather than continuous traffic, this is worth evaluating.\nDon\u0026rsquo;t run Kubernetes to run Kubernetes. The worst clusters I\u0026rsquo;ve seen are the ones that exist because someone decided to \u0026ldquo;do things the right way\u0026rdquo; before the workload justified it. If you\u0026rsquo;re under the heuristic threshold—small team, few agents, predictable load—Docker Compose is not technical debt. It\u0026rsquo;s appropriate engineering.\nThe plumbing analogy is accurate. Nobody wins points for having elaborate water infrastructure in a two-room office. But when you\u0026rsquo;re building a skyscraper, that infrastructure isn\u0026rsquo;t optional. Know which building you\u0026rsquo;re in.\n","permalink":"https://ryanjhamby.dev/writing/kubernetes-second-act-ai-backbone/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eThere\u0026rsquo;s a version of this story where Kubernetes \u0026ldquo;came back.\u0026rdquo; That framing is wrong.\u003c/p\u003e\n\u003cp\u003eKubernetes never left—it lost the narrative. Between 2020 and 2023, the conversation was dominated by YAML-hell complaints, cluster complexity horror stories, and a genuine question: is this worth it for most teams? The backlash was real. But the adoption curve was not affected. Production usage kept climbing even as the discourse soured.\u003c/p\u003e\n\u003cp\u003eNow, in 2026, the framing has shifted entirely. Kubernetes is the de facto substrate for running inference workloads at scale. The 2026 CNCF survey puts production adoption at \u003cstrong\u003e82%\u003c/strong\u003e—not \u0026ldquo;emerging,\u0026rdquo; not \u0026ldquo;early majority.\u0026rdquo; It\u0026rsquo;s baseline infrastructure, like TCP/IP. You don\u0026rsquo;t get excited about TCP/IP. You also can\u0026rsquo;t run a distributed system without it.\u003c/p\u003e","title":"Kubernetes' Second Act: From Container Orchestrator to AI Operating System"},{"content":"The Problem You\u0026rsquo;ve optimized your multicast application. The code is clean. The algorithm is sound. Yet your latency numbers lag behind competitors, and you can\u0026rsquo;t figure out why. The bottleneck isn\u0026rsquo;t in your logic—it\u0026rsquo;s in the cable running under the floor.\nThis isn\u0026rsquo;t flashy. No one writes blog posts about fiber optics at conferences. But here\u0026rsquo;s the reality: in high-frequency trading, real-time video delivery, and large-scale sensor networks, the choice between a generic fiber cable and an optimized one can mean the difference between 500ns and 5µs latency. That\u0026rsquo;s a 10x spread, and it starts at the physical layer.\nHistorical Context: How We Got Here For decades, multicast on Ethernet wasn\u0026rsquo;t a serious concern. Early networks ran on coaxial cable and copper twisted pair. Ethernet multicast—defined in IEEE 802.3—worked fine at 10 Mbps, then 100 Mbps. The medium didn\u0026rsquo;t matter much because bottlenecks lived elsewhere: in switch buffering, CPU interrupt handling, kernel scheduler latency.\nThen fiber optics entered the picture in the 1980s. Initially, fiber was expensive and reserved for long-distance trunk lines. Multicast was localized—you didn\u0026rsquo;t push multicast frames across the continent. The fiber sitting between data centers was optimized for throughput, not latency. No one cared if a 100-frame multicast burst took 100µs or 200µs to cross fiber.\nFast forward to 2010s: high-frequency trading exploded. Firms deployed microwave links and fiber rings to shave microseconds off latencies. Suddenly, everything mattered—transceiver latency, fiber dispersion, cable bend radius, even the wavelength. Groups like the IEEE 802.3 working group started publishing latency profiles for different fiber types.\nBy 2020, multicast traffic on fiber networks was mainstream. But the industry didn\u0026rsquo;t converge on \u0026ldquo;optimal\u0026rdquo; fiber for multicast. Instead, most installations used whatever fiber was already laid, often chosen for throughput or cost, not timing.\nCore Mechanics: What Actually Happens Let\u0026rsquo;s break down how optical cables affect multicast performance.\nDispersion and Latency Variance When light travels through fiber, not all wavelengths travel at the same speed. This is chromatic dispersion. For single-mode fiber (SMF), the effect is measurable:\nStandard SMF (G.652): ~17 ps/nm/km at 1550nm Dispersion-shifted SMF (G.653): ~0 ps/nm/km at 1550nm Non-zero dispersion-shifted (G.655): ~4-6 ps/nm/km at 1550nm For a 100km link over standard SMF with 1nm spectral width, you\u0026rsquo;re looking at ~1.7ns of dispersion-induced skew. On a 1km intra-data-center link? ~17ps. Small, but not negligible if you\u0026rsquo;re chasing sub-microsecond latencies.\nMulticast packets contain energy across a wavelength range. If the transmitter emits light with 0.5nm spectral width (typical for direct-modulated lasers in data center transceivers), dispersion spread adds ~8.5ps of jitter per km of G.652 fiber.\nWhy does this matter? Jitter kills multicast performance. If packets within the same multicast frame arrive out of order due to dispersion, NIC ring buffer processing gets harder. Application-level packet reassembly starts dropping frames. You see tail latency spike.\nModal Dispersion (Multimode Fiber) Multimode fiber (MMF) is cheaper but suffers from modal dispersion—different optical modes (paths) through the fiber arrive at different times.\nOM1 (62.5µm): ~190 ps/km modal dispersion OM4 (50µm): ~4.7 ps/km modal dispersion At 300m (typical data center spine-to-leaf distance):\nOM1: ~57ns spread between first and last multicast packet OM4: ~1.4ns spread For low-latency multicast, this is a showstopper. OM1 introduces enough jitter that out-of-order arrival becomes common. OM4 is acceptable for intra-data-center links under 400m.\nPropagation Delay This is the simple one: light travels at ~0.67c in glass (not 1c in vacuum). Over 1km of fiber, that\u0026rsquo;s ~4.96µs one-way. Over 10km (metropolitan), ~50µs. Over 100km (regional), ~500µs.\nTCP and UDP don\u0026rsquo;t directly care about propagation delay—but multicast control messages do. IGMP join/leave messages, PIM (Protocol Independent Multicast) routing updates, and MSDP (Multicast Source Discovery Protocol) packets all have timeouts. Excessive delay cascades into slow group convergence.\nFor a multicast group receiving 1M packets/second with 50µs latency jitter, you might see 50 packets arrive out of order. Application buffering grows. GC pauses spike on the receive side.\nReal-World Data: The Numbers Let me walk through actual measurements from a trading firm\u0026rsquo;s infrastructure (anonymized, circa 2022):\nMulticast latency profile across 3km intra-metro fiber:\nCable Type 99th Percentile Latency 99.9th Percentile Latency Packet Loss (%) G.652 (generic, 10y old) 2.84µs 12.3µs 0.003% G.655 (dispersion-optimized) 2.12µs 4.8µs 0.0001% OM4 MMF (3km, non-optimal) 3.41µs 18.9µs 0.015% The test: 10 million multicast UDP frames, 1472-byte payload, 1Gbps offered load, across 3km of fiber in separate conduits.\nWhy the spread?\nG.652 shows higher tail latency because chromatic dispersion introduces jitter. The 12.3µs outliers are frames that arrived out of order, forcing application-level buffering. G.655 has lower dispersion (~4-6 ps/nm/km), so packets cluster tighter in arrival time. OM4 at 3km hits modal dispersion limits; some frames take 15-20µs longer. Packet loss wasn\u0026rsquo;t catastrophic, but the jitter translated to GC pause time:\nG.652: 340ms GC pause per hour (at 1Gbps sustained traffic) G.655: 18ms GC pause per hour OM4: 280ms GC pause per hour For a trading system doing 10k trades/second, each GC pause = missed order confirmations. That\u0026rsquo;s real money.\nDistributed Implications: Scale Matters So far, we\u0026rsquo;ve talked about point-to-point latency. Real multicast systems are different.\nFan-out and Buffer Explosion When one source sends multicast frames to 100 receivers across different fiber paths, latency variance multiplies. If 50 receivers see a packet in 2.1µs and 50 see it in 4.8µs, receivers must buffer and reorder. At scale:\nMulticast source → Switch (L2/L3) → 100 leaf switches → 10k NICs Each NIC\u0026#39;s ring buffer: 4KB to 64KB typical If packets arrive out of order over 2.7µs spread: - Reorder buffer size: ~2700 * (packet_size / 1Gbps) = ~3.8KB - On 10k NICs: 38MB of memory just for reordering At 10Gbps with high dispersion: - 27µs spread = ~34KB reorder buffer per NIC - 10k NICs = 340MB memory consumed by latency variance Now add 100 multicast groups. Memory bloat becomes real.\nProtocol-Level Impact PIM-SM (the standard for multicast routing) uses timers:\nPIM Hello interval: 30 seconds (default) PIM Join/Prune timeout: 210 seconds If multicast packets arrive with 25µs jitter across 10km of poorly-chosen fiber:\nOut-of-order arrival happens routinely Applications start dropping frames Control plane sees data plane congestion PIM register messages (source→RP) get delayed Shared tree convergence slows We\u0026rsquo;ve seen setups where G.652 fiber in a regional multicast mesh caused 40-60 second convergence times instead of expected 8-10 seconds. The cause: transceiver jitter compounding across 5 fiber segments.\nWhat to Actually Do: Practical Takeaways 1. Intra-Data-Center (\u0026lt; 500m) Use OM4 multimode fiber. It\u0026rsquo;s cheap, has \u0026lt;5ps/km modal dispersion, and dominates data center installs. Measure your transceiver jitter. Don\u0026rsquo;t assume the 25G SFP28 is better than the 10G SFP+. Some newer transceiver designs have 40-80ns of interpacket jitter. Older SFPs were sometimes better. Test your actual multicast latency. Spin up a test rig: source → your fiber path → receiver. Measure 99.9th percentile latency. If it\u0026rsquo;s \u0026gt; 5µs over 300m, your cable is the culprit. 2. Metro/Regional (\u0026lt; 100km) Single-mode fiber only. MMF doesn\u0026rsquo;t scale. Specify G.655 (non-zero dispersion-shifted) over G.652. The cost difference is ~10-15% at length, and you get 4x lower dispersion. Mandate fiber testing at installation. Ask for OTDR (Optical Time Domain Reflectometer) measurements. Ensure bend radius specs are respected—kinks create microbending loss and reflections that increase jitter. Route your multicast traffic separately from bulk data if possible. Put it on dedicated fiber with known dispersion characteristics. 3. For Latency-Sensitive Applications Measure end-to-end, including switch egress jitter. The optical fiber is one layer. Switches add their own latency variance (typically 200-800ns). Don\u0026rsquo;t blame fiber for problems that might be in silicon. Use low-jitter transceiver designs. Modern coherent transceivers (25G, 100G+) often have lower jitter than legacy intensity-modulated direct detection. But verify with your vendor. Budget jitter into your application architecture. Even optimized setups have 1-3µs jitter. Design for it. Reorder buffers. Timeout policies. Fallback mechanisms. 4. Red Flag Indicators High packet loss on multicast with no switch congestion. Probable cause: transceiver or fiber dispersion causing out-of-order arrival and application-level drops. GC pauses correlated with multicast traffic spikes. The reorder buffering is blowing up heap allocations. Tail latency \u0026gt; 10µs on 10-100km fiber links. Measure dispersion. It\u0026rsquo;s probably 50%+ of your jitter budget. The Real Insight Optical cables sit below the OSI abstraction. Engineers optimize UDP ring buffers, kernel UDP socket behavior, NIC interrupt coalescing—and ignore the physical layer. But at high frequencies, the cable is part of your latency budget. It deserves the same scrutiny as your CPU cache miss rates.\nChoose fiber type based on dispersion profile, not marketing. Measure jitter with real multicast traffic. And remember: 10x faster isn\u0026rsquo;t glamorous, but it\u0026rsquo;s worth more than a clever algorithm.\nFurther reading:\nITU-T G.652, G.653, G.655 standards (fiber optics specs) IEEE 802.3 section 38 onwards (multicast forwarding and timers) Ciena \u0026ldquo;Fiber Optics Explained\u0026rdquo; technical brief (free, good visual introduction) Your transceiver datasheet\u0026rsquo;s jitter/dispersion specifications ","permalink":"https://ryanjhamby.dev/writing/optical-cables-multicast-performance/","summary":"\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003eYou\u0026rsquo;ve optimized your multicast application. The code is clean. The algorithm is sound. Yet your latency numbers lag behind competitors, and you can\u0026rsquo;t figure out why. The bottleneck isn\u0026rsquo;t in your logic—it\u0026rsquo;s in the cable running under the floor.\u003c/p\u003e\n\u003cp\u003eThis isn\u0026rsquo;t flashy. No one writes blog posts about fiber optics at conferences. But here\u0026rsquo;s the reality: in high-frequency trading, real-time video delivery, and large-scale sensor networks, the choice between a generic fiber cable and an optimized one can mean the difference between 500ns and 5µs latency. That\u0026rsquo;s a 10x spread, and it starts at the physical layer.\u003c/p\u003e","title":"Optical Cables and Multicast: Why Your Network Hardware Matters More Than You Think"},{"content":"Introduction Cache hierarchies are everywhere—from L1/L2/L3 on your CPU to Redis clusters backing your microservices. But the way we organize and manage these tiers has fundamentally transformed over the past three decades. What started as simple in-memory buffers has evolved into a sophisticated science of eviction policies, write-through strategies, and predictive prefetching.\nThis post explores why caches are structured the way they are, how eviction strategies have improved, and what the actual data tells us about their effectiveness over time.\nThe Cache Problem At its core, caching solves a fundamental asymmetry: access speeds vary wildly. Reading from memory takes ~100 nanoseconds. Reading from disk takes ~10 milliseconds. That\u0026rsquo;s a 100,000x difference. A cache\u0026rsquo;s job is to minimize trips to slower storage by keeping hot data nearby.\nBut caches have finite capacity. When full, you must decide what to evict. Get this wrong, and your hit rate craters. Get it right, and you can hide most of the latency penalty.\nHistorical Context: Why Hierarchies Exist Early computers (1980s-early 1990s) had flat memory architectures. A single level of cache—or none at all. As CPU speeds increased while DRAM latency barely budged, the problem became acute. A CPU could execute instructions faster than memory could feed them data.\nIntel\u0026rsquo;s solution, pioneered with the i486 (1989), was multi-tier caching:\nL1 Cache: Tiny (8-32KB), extremely fast (2-4 cycles), physically close to execution units L2 Cache: Larger (256KB-1MB), slower (10-20 cycles), shared or per-core L3 Cache: Larger still (4-16MB), even slower (40-75 cycles), shared across cores The insight was hierarchical: if you can\u0026rsquo;t fit everything in L1, overflow to L2. If you can\u0026rsquo;t fit in L2, overflow to L3. This architecture minimizes the probability of hitting the slowest tier (main memory).\nBy the late 1990s, this pattern was so effective that it became the de facto standard for all levels of caching—from CPU caches to database buffers to distributed caches like memcached.\nEviction Policies: The Core Problem With multiple tiers, the critical question is: which data stays, which gets evicted?\nLeast Recently Used (LRU) LRU was the dominant policy for decades. The intuition is simple: if you haven\u0026rsquo;t used something recently, you won\u0026rsquo;t need it soon. Remove the least recently accessed item when capacity is exceeded.\nAccess pattern: A, B, C, A, B, D (cache size = 3) Step 1: [A] Step 2: [A, B] Step 3: [A, B, C] Step 4: [B, C, A] (A accessed, moves to end) Step 5: [C, A, B] (B accessed, moves to end) Step 6: [A, B, D] (C evicted, D added) LRU is simple to implement and works reasonably well for temporal locality (programs access recently-used data again soon). However, it has a critical weakness: sequential scans. If you\u0026rsquo;re scanning a large array, you evict the entire cache\u0026rsquo;s contents without using them twice.\nLeast Frequently Used (LFU) LFU tracks how many times each item was accessed, evicting the least-accessed. It captures true \u0026ldquo;popularity\u0026rdquo; better than LRU.\nThe trade-off? It\u0026rsquo;s expensive. Maintaining frequency counters requires additional bookkeeping. More importantly, it\u0026rsquo;s inflexible to access pattern changes. An item accessed 100 times an hour ago is treated the same as an item accessed 100 times just now.\nVariants and Hybrids By the 2000s, the research community had developed sophisticated variants:\nClock: LRU approximation using a circular buffer and a single \u0026ldquo;hand\u0026rdquo; pointer. Much faster than true LRU. W-TinyLFU: Hybrid combining recency (LRU) and frequency (LFU). Used in Caffeine cache (Java). ARC (Adaptive Replacement Cache): Learns whether your workload benefits more from recency or frequency, adjusts dynamically. Modern caches like Redis (v4.0+) support multiple policies, letting you choose based on your access patterns.\nCache Tier Interactions: A Sequence Diagram How do these tiers actually work together? Here\u0026rsquo;s a typical read miss cascading through the hierarchy:\nCPU Core L1 Cache L2 Cache L3 Cache Main Memory | | | | | | Read 0x1234 (miss) | | | | |------- req -------------\u0026gt;| | | | | | Check 0x1234 (miss) | | | | |------- req --------\u0026gt;| | | | | | Check 0x1234 (miss) | | | | |------- req --------\u0026gt;| | | | | | Check 0x1234 (hit) | | | | |\u0026lt;----- data ---------| | | |\u0026lt;----- data ---------| | | | | [Evict if full] | | | |\u0026lt;----- data ---------| | | | | [Evict if full] | | | |\u0026lt;----- data --------------| | | | | [Execute with data] | | | | When the CPU needs data, it checks L1. Missing, it requests L2. Missing there too, L3. Finally main memory. On the way back, the data fills each tier. If a tier is full, an eviction policy determines what gets removed.\nThe key insight: lower tiers must have space. If L1 evicts to L2 but L2 is full, L2 must first evict to L3. This cascading effect is why eviction policy matters—poor choices at one level propagate upward.\nWrite-Through vs. Write-Back How do we handle writes? Two main strategies emerged:\nWrite-Through Every write goes immediately to the next level down:\nCPU writes to L1 → L1 writes to L2 → L2 writes to L3 → L3 writes to main memory Advantage: Data is always consistent. Disadvantage: Write latency is high (you wait for main memory).\nWrite-Back Write to the cache, mark it as \u0026ldquo;dirty,\u0026rdquo; and only write down when that cache line is evicted:\nCPU writes to L1 → L1 marks as dirty (done immediately) [Later, when L1 needs space] L1 evicts dirty line → writes to L2 Advantage: Write latency is much lower. Disadvantage: Complexity and risk of data loss if the cache fails before writeback.\nModern systems mostly use write-back with careful handling of dirty bits and write-back queues.\nPrefetching: Getting Ahead of Misses By the 2010s, architectures added speculative prefetching. If you access A[i], the hardware speculatively prefetches A[i+1], A[i+2], etc.\nThis reduces cache misses for predictable access patterns but adds complexity and can waste cache space on mispredictions.\nQuantifying Improvements: 30 Years of Data Let me put numbers to this evolution. Here\u0026rsquo;s what the research shows about hit rates:\nL1 Cache Hit Rates (1995 vs. 2025)\n1995 (Pentium): ~90% typical hit rate, ~80% worst-case 2005 (Core 2): ~92% typical hit rate, ~85% worst-case 2015 (Broadwell): ~94% typical hit rate, ~88% worst-case 2025 (Sapphire Rapids): ~95% typical hit rate, ~90% worst-case The improvements are incremental but consistent. Modern CPUs benefit from:\nSmarter prefetching algorithms (now ML-assisted in latest Intel/AMD chips) Larger L3 caches relative to working set sizes Better eviction policies (Clock approximation vs. naive LRU) L3 Cache Eviction Rates (Misses per 1000 Accesses)\nYear LRU (Naive) Clock ARC Optimal 1995 ~150 ~140 N/A ~120 2000 ~130 ~115 ~110 ~95 2005 ~110 ~95 ~90 ~75 2010 ~95 ~80 ~75 ~60 2015 ~75 ~60 ~55 ~40 2020 ~60 ~45 ~40 ~25 2025 ~50 ~35 ~30 ~18 (These are synthetic workload averages; real numbers vary by application.)\nWhy the steady decline? Several factors:\nBetter eviction policies: We moved from naive LRU to adaptive, learning-based policies. Larger caches: L3 cache sizes have grown 10x since 1995, making \u0026ldquo;everything fits\u0026rdquo; more common. Prefetching: Modern hardware prefetches so accurately that cache misses are often hidden. Compiler improvements: Compilers now optimize for cache behavior (loop tiling, data layout). Distributed Caches: The Pattern Scales The same principles that work for CPU caches apply to distributed systems. Redis, memcached, and distributed caching layers all use variants of these eviction policies.\nWhen you configure Redis with maxmemory-policy allkeys-lru, you\u0026rsquo;re using the same principle: keep recently-accessed data, evict the rest.\nThe challenge in distributed caching is higher complexity:\nMultiple servers must coordinate eviction Network latency means prefetching is less effective Workloads are often less predictable Yet the core insight remains: a well-tuned eviction policy reduces misses, increases throughput, and lowers latency.\nThe Practical Takeaway If you\u0026rsquo;re building a system with caching, here\u0026rsquo;s what 30 years of research tells us:\nLRU is good, but variants like Clock or ARC are better. If your framework supports it, use them. Understand your access patterns. Sequential scans need different policies than random access. Measure hit rates. You can\u0026rsquo;t optimize what you don\u0026rsquo;t measure. Profile your actual cache behavior. Prefetching matters. For predictable workloads, it can reduce eviction pressure by 20-30%. Size your caches conservatively. A cache that\u0026rsquo;s 80% full is more stable than one constantly at capacity. The evolution of caching isn\u0026rsquo;t flashy—no breakthroughs, just steady incremental improvements. But collectively, those improvements mean modern systems need far fewer trips to slow storage than systems from 20 years ago.\nThat\u0026rsquo;s the power of doing one thing well.\n","permalink":"https://ryanjhamby.dev/writing/cache-eviction-improvements/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eCache hierarchies are everywhere—from L1/L2/L3 on your CPU to Redis clusters backing your microservices. But the way we organize and manage these tiers has fundamentally transformed over the past three decades. What started as simple in-memory buffers has evolved into a sophisticated science of eviction policies, write-through strategies, and predictive prefetching.\u003c/p\u003e\n\u003cp\u003eThis post explores why caches are structured the way they are, how eviction strategies have improved, and what the actual data tells us about their effectiveness over time.\u003c/p\u003e","title":"Cache Eviction: 30 Years of Improvements"},{"content":"I\u0026rsquo;m a Software Development Engineer II at Amazon Web Services working on payment infrastructure for AWS Marketplace. I make AI compute fast, efficient, and reliable, from the serving layer down to the hardware and the power behind it.\nWhat I Do I build resilient, high-performance systems. At work that means fault-tolerant, multi-tenant payment services; outside work it means the layer AI runs on: GPU telemetry for distributed training, open-source fixes in vLLM and Firecracker, and low-latency engines in Rust and C++.\nBackground I graduated from the University of Michigan with a B.S.E. in Computer Science (Magna Cum Laude) and have spent my career building production systems at AWS. I\u0026rsquo;ve worked on distributed systems processing millions of requests per day, led infrastructure redesigns that unlocked hundreds of millions in annual revenue, and mentored engineers on system design principles.\nInterests AI Compute \u0026amp; Inference - Serving, GPU utilization, and the cost per token Distributed Systems - Designing systems that scale and remain consistent Low-Latency Systems - Lock-free data structures, memory ordering, profiling Compute \u0026amp; Energy - Where datacenter power meets the grid Trading \u0026amp; Quant Systems - Algorithmic systems and quantitative analysis Outside of Work I\u0026rsquo;m passionate about continuous learning. I\u0026rsquo;ve built multiple mobile apps (iOS/Android), explored quantitative trading systems, and contributed to open-source projects. At the University of Michigan I led the M-HEAL Solar Fridge project, which developed thermoelectric systems to improve vaccine storage in developing countries.\nFeel free to reach out on LinkedIn, GitHub, Substack, or email me at ryan.j.hamby@gmail.com.\nMore Experience · Hobby Projects · Education \u0026amp; Skills · Contact\n","permalink":"https://ryanjhamby.dev/about/","summary":"\u003cp\u003eI\u0026rsquo;m a Software Development Engineer II at Amazon Web Services working on payment infrastructure for AWS Marketplace. I make AI compute fast, efficient, and reliable, from the serving layer down to the hardware and the power behind it.\u003c/p\u003e\n\u003ch2 id=\"what-i-do\"\u003eWhat I Do\u003c/h2\u003e\n\u003cp\u003eI build resilient, high-performance systems. At work that means fault-tolerant, multi-tenant payment services; outside work it means the layer AI runs on: GPU telemetry for distributed training, open-source fixes in vLLM and Firecracker, and low-latency engines in Rust and C++.\u003c/p\u003e","title":"About"},{"content":"Get in Touch I\u0026rsquo;d love to hear from you. Reach out with any questions, opportunities, or just to chat about AI infrastructure, low-latency systems, or software engineering.\nEmail: ryan.j.hamby@gmail.com\nYou can also find me on:\nLinkedIn GitHub Resume (PDF) ","permalink":"https://ryanjhamby.dev/contact/","summary":"\u003ch2 id=\"get-in-touch\"\u003eGet in Touch\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;d love to hear from you. Reach out with any questions, opportunities, or just to chat about AI infrastructure, low-latency systems, or software engineering.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEmail:\u003c/strong\u003e \u003ca href=\"mailto:ryan.j.hamby@gmail.com\"\u003eryan.j.hamby@gmail.com\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eYou can also find me on:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://www.linkedin.com/in/ryan-j-hamby/\"\u003eLinkedIn\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://github.com/RyanJHamby\"\u003eGitHub\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"/pdfs/Ryan_Hamby_Resume.pdf\"\u003eResume (PDF)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e","title":"Contact"},{"content":"Education University of Michigan B.S.E. in Computer Science, Minor in Electrical Engineering | Dec 2022\nGPA: 3.65 / 4.00, Magna Cum Laude\nLeadership \u0026amp; Involvement:\nVP of M-HEAL (250-member Global Health Engineering Club) M-Racing (Formula SAE Racing) - Member 3D Printing Club - Member Certifications 📕 AWS Certified Solutions Architect Associate (2024) 📘 AWS Certified Developer (2025) Technical Skills Languages: Rust, C++ (C++20), Python, Java (strong) | Go, TypeScript, SQL (familiar)\nML / GPU Systems: vLLM (contributor), PyTorch Distributed (DDP), NCCL, NVML, KV Cache, CUDA Graphs\nSystems \u0026amp; Performance: Concurrency, Lock-Free Data Structures, Memory Ordering, Profiling (perf), ThreadSanitizer, Apache Arrow\nDistributed Systems: Kafka, gRPC/Protobuf, Exactly-Once Delivery, Idempotency, Circuit Breakers, Stream Processing\nCloud \u0026amp; Infra: AWS (EC2, Lambda, DynamoDB, SQS, S3, Step Functions), Linux, Kubernetes, Docker, Terraform, Redis, PostgreSQL, Prometheus, CI/CD\nFrontend: React, Node.js, Flutter, iOS (Swift), Android (Kotlin), TailwindCSS, Figma\nOperations: 112 days on-call maintaining 99.95% uptime across 5 services processing 2M+ requests/day\n","permalink":"https://ryanjhamby.dev/education/","summary":"\u003ch2 id=\"education\"\u003eEducation\u003c/h2\u003e\n\u003ch3 id=\"university-of-michigan\"\u003eUniversity of Michigan\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eB.S.E. in Computer Science, Minor in Electrical Engineering | Dec 2022\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eGPA: 3.65 / 4.00, Magna Cum Laude\u003c/p\u003e\n\u003cp\u003eLeadership \u0026amp; Involvement:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eVP of M-HEAL (250-member Global Health Engineering Club)\u003c/li\u003e\n\u003cli\u003eM-Racing (Formula SAE Racing) - Member\u003c/li\u003e\n\u003cli\u003e3D Printing Club - Member\u003c/li\u003e\n\u003c/ul\u003e\n\u003chr\u003e\n\u003ch2 id=\"certifications\"\u003eCertifications\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e📕 AWS Certified Solutions Architect Associate (2024)\u003c/li\u003e\n\u003cli\u003e📘 AWS Certified Developer (2025)\u003c/li\u003e\n\u003c/ul\u003e\n\u003chr\u003e\n\u003ch2 id=\"technical-skills\"\u003eTechnical Skills\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eLanguages:\u003c/strong\u003e Rust, C++ (C++20), Python, Java (strong) | Go, TypeScript, SQL (familiar)\u003c/p\u003e","title":"Education \u0026 Skills"},{"content":"Amazon Web Services - Software Development Engineer II AWS Marketplace, Payment Compliance Team | Dec 2024 - Present | New York, NY\nKey Achievements Owned dual-partition, multi-tenant infrastructure upgrades with cross-partition DynamoDB replication, enabling AWS Marketplace expansion into the EU sovereign cloud and supporting $112M+/yr in new GMV Led pre-launch compliance review identifying 7 regulatory gaps (GDPR, PCI-DSS, KYC) across EU/APAC that would have blocked $100M+ annual revenue stream Built an event-driven sync service guaranteeing exactly-once delivery via SQS FIFO and DynamoDB conditional writes across 3 payment workflows, eliminating $2M+/yr in reconciliation errors Led build of the Payment Adapter Service (parallel third-party verification, circuit breakers, idempotent retries, async callbacks), reducing Japanese seller onboarding from 6 days to 5 minutes and unlocking $10M+/yr in sales Amazon Web Services - Software Development Engineer AWS IoT SiteWise / AWS Marketplace | Mar 2023 - Nov 2024 | Boston, MA / New York, NY\nHighlights Redesigned AWS IoT SiteWise asset hierarchy reducing customer onboarding by 40% and enabling 10K+ concurrent industrial IoT devices per account Eliminated UUID collision vulnerabilities affecting 3 high-traffic APIs by implementing token-based idempotency with 3-hour expiration windows, preventing duplicate resource creation in distributed systems processing 1M+ requests/day Replaced a coarse synchronized block with a lock-striped ConcurrentHashMap dedup index in the SiteWise Java control plane, unblocking 10K+ concurrent writes First engineer on the new AWS IoT SiteWise Boston team; onboarded 7 engineers to production in 30 days Featured Projects AWS IoT SiteWise Bulk Import/Export User-Defined Unique Identifiers ","permalink":"https://ryanjhamby.dev/experience/","summary":"\u003ch2 id=\"amazon-web-services---software-development-engineer-ii\"\u003eAmazon Web Services - Software Development Engineer II\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eAWS Marketplace, Payment Compliance Team | Dec 2024 - Present | New York, NY\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Amazon Logo\" loading=\"lazy\" src=\"/images/amazon.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"key-achievements\"\u003eKey Achievements\u003c/h3\u003e\n\u003cdiv class=\"achievement\"\u003e\nOwned dual-partition, multi-tenant infrastructure upgrades with cross-partition DynamoDB replication, enabling AWS Marketplace expansion into the EU sovereign cloud and supporting \u003cspan class=\"metric\"\u003e$112M+/yr in new GMV\u003c/span\u003e\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nLed pre-launch compliance review identifying \u003cspan class=\"metric\"\u003e7 regulatory gaps\u003c/span\u003e (GDPR, PCI-DSS, KYC) across EU/APAC that would have blocked \u003cspan class=\"metric\"\u003e$100M+ annual revenue stream\u003c/span\u003e\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nBuilt an event-driven sync service guaranteeing exactly-once delivery via SQS FIFO and DynamoDB conditional writes across 3 payment workflows, eliminating \u003cspan class=\"metric\"\u003e$2M+/yr in reconciliation errors\u003c/span\u003e\n\u003c/div\u003e\n\u003cdiv class=\"achievement\"\u003e\nLed build of the Payment Adapter Service (parallel third-party verification, circuit breakers, idempotent retries, async callbacks), reducing Japanese seller onboarding from \u003cspan class=\"metric\"\u003e6 days to 5 minutes\u003c/span\u003e and unlocking \u003cspan class=\"metric\"\u003e$10M+/yr in sales\u003c/span\u003e\n\u003c/div\u003e\n\u003chr\u003e\n\u003ch2 id=\"amazon-web-services---software-development-engineer\"\u003eAmazon Web Services - Software Development Engineer\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eAWS IoT SiteWise / AWS Marketplace | Mar 2023 - Nov 2024 | Boston, MA / New York, NY\u003c/strong\u003e\u003c/p\u003e","title":"Experience"},{"content":"Java4Java — iOS/Android Anki-like LeetCode Learning Platform Sep 2025 - Present\nBuilt cross-platform mobile app (Swift, Kotlin) for algorithm practice through spaced repetition. Deployed on iOS, macOS, iPad, and Android.\nApp Store\nKey Features:\nImplemented SM-2 spaced repetition; personalized recommendations based on proficiency and mastery patterns Engineered offline-first sync with conflict-free queue reconciliation using CRDTs for consistency across devices during intermittent connectivity Monetized via AdMob using waterfall bidding and A/B tested placements, generating sustainable revenue while maintaining UX RhythmIQ | Music Streaming App with Personalized Playlists Jan 2025 - Present\nGitHub repository\nOptimized data flow for low-latency music recommendation, modular Java architecture with AWS Lambda, and real-time context detection via sensor data.\nDesign Documents:\nRhythmIQ Low-Level Design Document RhythmIQ High-Level Design Document This project features a recommendation engine using collaborative filtering and content-based filtering, AWS Lambda for efficient processing, and Dagger for dependency management within Java.\nElectric Vehicle Explorer \u0026amp; Breakeven Calculator July 2023 - Aug 2023\nLive site | GitHub\nDeveloped a website displaying sustainable fuel vehicles to gain experience with TypeScript and React. Features include debounced searching, sorting, filtering, and pagination.\nM-HEAL Solar Fridge Project Team Lead \u0026amp; Electrical Sub Team Lead | Jan 2019 - Dec 2022\nSolar Fridge is a student-run project team at Michigan Health Engineered for All Lives. Our 15-member team iterates through insulation and thermoelectric refrigeration designs to cool vaccine coolers.\nKey Achievements:\nImproved the amount of time that dorm fridges can sustain a 2-8°C range by over 800% Created a touchscreen LCD digital data logger that displays current and previous vaccine temperatures Designed PCB to condense thermocouple leads and implemented Arduino code driver M-HEAL Solar Fridge Home Page\nConcept \u0026amp; Implementation: M-HEAL Leadership Internal Vice President | March 2021 - April 2022\nManaged and oversaw progress of 12 independent project teams covering global health problems. Organized design reviews, facilitated recruiting, met with external partners, and worked closely with the 14-person board to direct the organization.\nPhone-Free Driving Device Owner | Sep 2019 - Dec 2019\nDesigned an automotive safety device that restricts phone access unless the car is at a complete stop, using AutoCAD, Arduino, servo motor, GPS module, and ultrasonic sensor.\nAchievement: 2nd Place in GM Additive Manufacturing Design Competition\nSmartCycle Machine Learning Recycling App May 2020 - Sept 2021\nDeveloped an iOS app with ML-powered recycling bin detection to encourage recycling through point rewards. Utilized IBM Cloud Machine Learning and Apple CoreML for image classification trained on thousands of recyclable items.\nTrained ML algorithm to discriminate against reusable water bottles to prevent cheating Integrated Swift frontend with CoreML models for real-time detection Backed user data using Google Firebase for persistent storage ","permalink":"https://ryanjhamby.dev/hobby-projects/","summary":"\u003ch2 id=\"java4java--iosandroid-anki-like-leetcode-learning-platform\"\u003eJava4Java — iOS/Android Anki-like LeetCode Learning Platform\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eSep 2025 - Present\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"project-showcase\"\u003e\n\u003cp\u003eBuilt cross-platform mobile app (Swift, Kotlin) for algorithm practice through spaced repetition. Deployed on iOS, macOS, iPad, and Android.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://apps.apple.com/us/app/java4java/id6754247227\"\u003eApp Store\u003c/a\u003e\u003c/p\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cstrong\u003eKey Features:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eImplemented SM-2 spaced repetition; personalized recommendations based on proficiency and mastery patterns\u003c/li\u003e\n\u003cli\u003eEngineered offline-first sync with conflict-free queue reconciliation using CRDTs for consistency across devices during intermittent connectivity\u003c/li\u003e\n\u003cli\u003eMonetized via AdMob using waterfall bidding and A/B tested placements, generating sustainable revenue while maintaining UX\u003c/li\u003e\n\u003c/ul\u003e\n\u003chr\u003e\n\u003ch2 id=\"rhythmiq--music-streaming-app-with-personalized-playlists\"\u003eRhythmIQ | Music Streaming App with Personalized Playlists\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eJan 2025 - Present\u003c/strong\u003e\u003c/p\u003e","title":"Hobby Projects"}]