execute_tests

Attributes

executor

Classes

LLMExecutor

Module Contents

class execute_tests.LLMExecutor(config_file='config/config.yaml')

load_config(config_file)

call_llm_api(question, api_name=None, **kwargs)

deep_format(obj, **kwargs)

Recursively formats strings in a nested structure.

extract_answer(response_json, response_path)

Extracts the answer from the response JSON based on the response path.

parse_response_path(path_str)

Parses the response path string into a list of keys/indexes. For example: ‘choices[0].message.content’ => [‘choices’, 0, ‘message’, ‘content’]

execute_tests.executor