utils
check_either_required_env_variables_set
check_either_required_env_variables_set(
required_env_variables: list[list[str]],
) -> list[Exception]
Check if at least one of the required environment variables is set in a list for a given list of lists of environment variables.
For example, if required_env_variables is [['A', 'B'], ['C', 'D']]
,
then we first look at ['A', 'B']
, and check at least one of the
environment variables ‘A’ or ‘B’ are set. If either ‘A’ or ‘B’ are not set,
we add a Warning to the returned list. IF neither ‘A’ or ‘B’ are set, we add
a KeyError to the returned list. We then repeat this process for ['C', 'D']
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
required_env_variables
|
list[list[str]]
|
List of lists of environment variables where at least one of the environment variables must be set. |
required |
Returns:
Type | Description |
---|---|
list[Exception]
|
List of exceptions of either Warnings to say an environment variable isn’t set or KeyErrors if none of the required environment variables in a list are set. |
Source code in src/prompto/utils.py
check_max_queries_dict
Check the format of the max_queries_dict dictionary.
Raises a TypeError if the dictionary is not in the correct format.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
max_queries_dict
|
dict[str, int | dict[str, int]]
|
A dictionary of maximum queries per minute for each API or group, by default {}. The dictionary keys should be either a group name (which is then used in the “group” key of the prompt_dict) or an API name. The values should be integers (the maximum queries per minute or rate limit) or itself a dictionary with keys as the model-names and values as the maximum queries per minute for that model. |
required |
Returns:
Type | Description |
---|---|
bool
|
True if the max_queries_dict is valid, otherwise raises a ValueError or TypeError. |
Source code in src/prompto/utils.py
check_optional_env_variables_set
Check if optional environment variables are set.
A list of Warnings are returned for each optional environment variables that is not set. If they are all set, an empty list is returned.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
optional_env_variables
|
list[str]
|
List of environment variables that are optional to be set. |
required |
Returns:
Type | Description |
---|---|
list[Exception]
|
List of exceptions for the optional environment variables that are not set. |
Source code in src/prompto/utils.py
check_required_env_variables_set
Check if required environment variables are set.
A list of KeyErrors are returned for each required environment variables that is not set. If they are all set, an empty list is returned.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
required_env_variables
|
list[str]
|
List of environment variables that are required to be set. |
required |
Returns:
Type | Description |
---|---|
list[Exception]
|
List of exceptions that are raised if the required environment variables are not set. |
Source code in src/prompto/utils.py
copy_file
Function to copy a file from one location to another.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
str
|
File path of the file to be moved. |
required |
destination
|
str
|
File path of the destination of the file. |
required |
Source code in src/prompto/utils.py
create_folder
Function to create a folder if it does not already exist.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
folder
|
str
|
Name of the folder to be created. |
required |
Source code in src/prompto/utils.py
get_environment_variable
Get the value of an environment variable for a specific model. We first check if the environment variable with the model name identifier exists. If it does, we return the value of that environment variable. If it does not exist, we return the value of the environment variable without the model name identifier. If neither environment variables exist, we raise a KeyError.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
env_variable
|
str
|
The name of the environment variable to get |
required |
model_name
|
str
|
The name of the model to get the environment variable for |
required |
Returns:
Type | Description |
---|---|
str
|
The value of the environment variable for the specific model. If no model-specific environment variable exists, the value of the environment variable without the model name identifier is returned. |
Source code in src/prompto/utils.py
get_model_name_identifier
Helper function to get the model name identifier.
Some model names can contain characters that are not allowed in environment variable names. This function replaces those characters (“-“, “/”, “.”, “:”, ” “) with underscores (“_”).
Parameters:
Name | Type | Description | Default |
---|---|---|---|
model_name
|
str
|
The model name |
required |
Returns:
Type | Description |
---|---|
str
|
The model name identifier with invalid characters replaced with underscores |
Source code in src/prompto/utils.py
log_error_response_chat
log_error_response_chat(
index: int | str,
model: str,
message_index: int,
n_messages: int,
message: str,
responses_so_far: list[str],
error_as_string: str,
id: int | str = "NA",
) -> str
Log an error response from a model in a chat interaction.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
index
|
int | str
|
Identifier for the query/chat from the input file. |
required |
model
|
str
|
Name of the model that generated the response. |
required |
message_index
|
int
|
Index of the message in the chat interaction. |
required |
n_messages
|
int
|
Total number of messages in the chat interaction. |
required |
message
|
str
|
Message that was sent to the model. |
required |
responses_so_far
|
list[str]
|
List of responses that have been generated so far in the chat interaction. |
required |
error_as_string
|
str
|
Error message that was generated by the model as a string. |
required |
Returns:
Type | Description |
---|---|
str
|
The log message that was written. |
Source code in src/prompto/utils.py
log_error_response_query
log_error_response_query(
index: int | str,
model: str,
prompt: str,
error_as_string: str,
id: int | str = "NA",
) -> str
Log an error response from a model to a query.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
index
|
int | str
|
Identifier for the query from the input file. |
required |
model
|
str
|
Name of the model that generated the response. |
required |
prompt
|
str
|
Prompt that was used to generate the response. |
required |
error_as_string
|
str
|
Error message that was generated by the model as a string. |
required |
Returns:
Type | Description |
---|---|
str
|
The log message that was written. |
Source code in src/prompto/utils.py
log_success_response_chat
log_success_response_chat(
index: int | str,
model: str,
message_index: int,
n_messages: int,
message: str,
response_text: str,
id: int | str = "NA",
) -> str
Log a successful chat interaction with a model.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
index
|
int | str
|
Identifier for the query/chat from the input file. |
required |
model
|
str
|
Name of the model that generated the response. |
required |
message_index
|
int
|
Index of the message in the chat interaction. |
required |
n_messages
|
int
|
Total number of messages in the chat interaction. |
required |
message
|
str
|
Message that was sent to the model. |
required |
response_text
|
str
|
Response text generated by the model. |
required |
Returns:
Type | Description |
---|---|
str
|
The log message that was written. |
Source code in src/prompto/utils.py
log_success_response_query
log_success_response_query(
index: int | str,
model: str,
prompt: str,
response_text: str,
id: int | str = "NA",
) -> str
Log a successful response from a model to a query.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
index
|
int | str
|
Identifier for the query from the input file. |
required |
model
|
str
|
Name of the model that generated the response. |
required |
prompt
|
str
|
Prompt that was used to generate the response. |
required |
response_text
|
str
|
Response text generated by the model. |
required |
Returns:
Type | Description |
---|---|
str
|
The log message that was written. |
Source code in src/prompto/utils.py
move_file
Function to move a file from one location to another.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
source
|
str
|
File path of the file to be moved. |
required |
destination
|
str
|
File path of the destination of the file. |
required |
Source code in src/prompto/utils.py
parse_list_arg
Splits a string into a list by separating on commas. Will remove any whitespace and removes duplicates. Used to parsing argument which is a list in CLI commands.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
argument
|
str
|
A string separated with commas, e.g. “judge1, judge2” or “judge1,judge2,judge1”. Whitespace will be removed. |
required |
Returns:
Type | Description |
---|---|
list[str]
|
A list of the comma separated items in the input string, with no duplicates and whitespaces, e.g. [“judge1”, “judge2”]. |
Source code in src/prompto/utils.py
sort_input_files_by_creation_time
Function sorts the jsonl or csv files in the input folder by creation/change time in a given directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
input_folder
|
str
|
Folder which contains the files to be processed. |
required |
Returns:
Type | Description |
---|---|
list[str]
|
Ordered list of jsonl or csv filenames in the input folder. |
Source code in src/prompto/utils.py
sort_prompts_by_model_for_api
For a list of prompt dictionaries, sort the dictionaries with "api": api
by the “model_name” key. The rest of the dictionaries are kept in the same order.
For Ollama API, if the model requested is not currently loaded, the model will be loaded on demand. This can take some time, so it is better to sort the prompts by the model name to reduce the time taken to load the models.
If no dictionaries with "api": api
are present, the original list is returned.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
prompt_dicts
|
list[dict]
|
List of dictionaries containing the prompt and other parameters to be sent to the API. Each dictionary must have keys “prompt” and “api” |
required |
api
|
str
|
The API name to sort the prompt dictionaries by the “model_name” key |
required |
Returns:
Type | Description |
---|---|
list[dict]
|
List of dictionaries containing the prompt and other parameters
where the dictionaries with |
Source code in src/prompto/utils.py
write_log_message
Helper function to write a log message to a log file with the current date and time of the log message.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
log_file
|
str
|
Path to the log file. |
required |
log_message
|
str
|
Message to be written to the log file. |
required |