An agent that operates has to prove it decides well. You design how that is measured: test cases, evaluation sets, metrics and the threshold at which an agent can move from proposing to acting.
The role
- Build evaluation sets from real operational cases.
- Measure each agent's hits, errors, cost and time.
- Compare models and configurations with method, not impressions.
- Define the thresholds that allow moving up a level of autonomy.
Your first 90 days
- Day 30
You learn the agents and their logs; you measure one end to end.
- Day 60
You build the evaluation that runs before every change to an agent.
- Day 90
An autonomy decision gets made with your numbers.
Stack and tools
PythonSQLLanguage model APIspgvector
What we look for
Must have
- Fluency in statistics and experimental method.
- Python to analyze and automate evaluations.
- You distrust a number until you know where it comes from.
- You can explain a result to someone who isn't technical.
Nice to have
- Language model evaluation.
- Semantic search or retrieval.
- Experience with operational data.
What working here is like
What you build goes into products that operate real organizations. You work directly with the founders, with no layers between your code and production, under the same rule we give the intelligence: a person signs what is critical.