Custom Evals
Need to use your own custom evals?
There are a number of ways to use custom evals in Athina.
GradingCriteria: Pass / fail based on a custom criterion. "If X, then fail. Otherwise pass."CustomPrompt: Use a completely custom prompt for evaluation.ApiCall: Use theApiCallevaluator to make a call to a custom endpoint where you are hosting your evaluation logic.- Create your own evaluator: Create your own evaluator by extending the
BaseEvaluator(opens in a new tab) class.
Contribute your evals!
This library is open source and we welcome contributions.
If you have an idea for a new evaluator, please open an issue (opens in a new tab) or submit a PR (opens in a new tab).