API Call

If you want to use your own evaluation functions, you can do so by using the ApiCall evaluator.

Description:
Performs an API call to a specified endpoint and picks up the evaluation result from the response.

This evaluator is useful if you:

• Want to run some complex or custom logic
• Need to run evaluations on your own servers
• Want to use an evaluation metric that we don't support currently (Send us an email at hello@athina.ai to request a new metric - we can integrate pretty fast).

Info:

• Input: response, query, context, response
• Type: boolean
• Metrics: passed (0 or 1)

Arguments:

• url: string - API endpoint to call. Note that this API should accept POST request.
• headers: dict - Headers to include in the API call.
• payload: dict - Body to send with the API call. This payload will have the Response added to it.

Sample Code:

from athina.evals import ApiCall
from athina.loaders import ResponseLoader
 
# Raw data must contain response and optionally the query, context and expected_response to be passed to the API
raw_data = [
    {
        "response": "Response to be sent to the your own API based evaluator",
        "query": "Query to be sent to the your own API based evaluator"
    }
]
dataset = ResponseLoader().load_dict(raw_data)
 
ApiCall(
    url="https://8e714940905f4022b43267e348b8a713.api.mockbin.io/",
    payload={"evaluator": "custom_api_based_evaluator"},
    headers={"Authorization": "Bearer token"}
).run_batch(data=dataset).to_df()

The dataset should contain the response and optionally the query, context and expected_response to be passed to the API.