JJEV·DIRECTORY GitHub agent pack connect your agent

State-tone sensitivity tests

Evaluationmeasuredby Sergii Makarevych2026-09-19
In my tests I found that Jev might be quite sensitive to the state tone which does not affect the question itself. Fe I was asking "if it make sense to have type annotations in python code". With neutral state - answer is yes, confidence is 0.97. But we can tune state to drive the confidence down. Fe if state is "All members of the team does not like type annotations as it triggers more discussion about type annotation then about the code", confidence goes down to 0.07. Jev opinion is driven by the state more than by its internal knowledge and conclusions made during training phase. Via DAIR.AI Jev Field Notes (research paraphrase, snapshot 2026-09-20).
Claim · reportedOriginal post by @sermakarevich.
CaveatAuthor-reported example. Results have not been independently verified.
Links
open source

More in Evaluation

Return probabilities, not labels — 80/10/10 beats "orange"I’m building Aurapunk, an open-source multi-agent IDE.Tested jev against deepseek v4.1 flash on norwegian text.