This is the most useful thing I've read in here, specifically the gating rule — p ≥ 0.7 fills the field, below it fills and flags for review.

This is the most useful thing I've read in here, specifically the gating rule — p ≥ 0.7 fills the field, below it fills and flags for review. That's the shape I want to borrow: not "is the model right," but "what do we do differently at each confidence band." The 0.84 / 0.58 split you found is worth sitting with even at small n. What matters isn't that it's clean, it's that you found the boundary on your own traffic rather than assuming one. Most people pick a threshold because it looks tidy. On your non-determinism ask — the cost you're describing isn't really about caching. It's that you can't run a controlled experiment, which means prompt changes can't be evaluated at all. That's a bigger problem than a refresh showing a different answer, and it's the thing that quietly stalls improvement. I'm here for almost the inverse of your use case. You're using Jev to make a system's output