I've been training models using SFT/RL and then measuring the training methods using benchmark evaluations.

I've been training models using SFT/RL and then measuring the training methods using benchmark evaluations. One of the tedious and costly parts of the research is breaking down the taxonomy of failures when the models get things wrong. Using Fable to do perform the failure taxonomy on 40 samples, it takes about 11 minutes of wall clock time and roughly 102k context tokens. Typesafe did the same work with over 80% accuracy in 9 sec for 77k tokens. It's totally saving me time and money on this part of the iterative cycle of research.