Monitoring LLM behavior: Drift, retries, and refusal patterns

The stochastic challenge Traditional software is predictable: Input. A plus function B always equals output C. This determinism allows engineers to develop robust tests. On the other hand, generative AI is stochastic and unpredictable. The exact same prompt often yields different results on Monday versus Tuesday, breaking the traditional unit testing that engineers know and love.To ship enterprise-ready AI, engineers cannot rely on mere “vibe checks” that pass today but fail when customers use the product. Product builders need to adopt a new infrastructure layer: The AI Evaluation Stack.This framework is informed by my extensive experience shipping AI products for Fortune 500 enterprise customers in high-stakes industries, where “hallucination” is not funny — it’s a huge compliance risk.Defining the AI evaluation paradigmTraditional software tests are binary assertions (pass/fail). While some AI evals use binary asserts, many evaluate on a gradient. An eval is not a single script; it is a structured pipeline of assertions — ranging from strict code syntax to nuanced semantic checks — that verify the AI system’s intended function.The taxonomy of evaluation checksTo build a robust, cost-effective pipeline, asserts must be separated into two distinct architectural layers:Layer 1: Deterministic assertionsA surprisingly large share of production AI failures aren't semantic "hallucinations" — they are basic syntax and routing failures. Deterministic assertions serve as the pipeline's first gate, using traditional code and regex to validate structural integrity.Instead of asking if a response is "helpful," these assertions ask strict, binary questions:Did the model generate the correct JSON key/value schema?Did it invoke the correct tool call with the required arguments?Did it successfully slot-fill a valid GUID or email address?// Example: Layer 1 Deterministic Tool Call Assertion{ "test_scenario": "User asks to look up an account", "assertion_type": "schema_vali