Using prompto with Gemini¶
from prompto.settings import Settings
from prompto.experiment import Experiment
from dotenv import load_dotenv
import os
When using prompto to query models from the Gemini API, lines in our experiment .jsonl files must have "api": "gemini" in the prompt dict.
Environment variables¶
For the Gemini API, there are two environment variables that could be set:
GEMINI_API_KEY: the API key for the Gemini API
As mentioned in the environment variables docs, there are also model-specific environment variables too which can be utilised. In particular, when you specify a model_name key in a prompt dict, one could also specify a GEMINI_API_KEY_model_name environment variable to indicate the API key used for that particular model (where "model_name" is replaced to whatever the corresponding value of the model_name key is). We will see a concrete example of this later.
To set environment variables, one can simply have these in a .env file which specifies these environment variables as key-value pairs:
GEMINI_API_KEY=<YOUR-GEMINI-KEY>
If you make this file, you can run the following which should return True if it's found one, or False otherwise:
load_dotenv(dotenv_path=".env")
True
Now, we obtain those values. We raise an error if the GEMINI_API_KEY environment variable hasn't been set:
GEMINI_API_KEY = os.environ.get("GEMINI_API_KEY")
if GEMINI_API_KEY is None:
raise ValueError("GEMINI_API_KEY is not set")
If you get any errors or warnings in the above cell, try to fix your .env file like the example we have above to get these variables set.
Types of prompts¶
With the Gemini API, the prompt (given via the "prompt" key in the prompt dict) can take several forms:
- a string: a single prompt to obtain a response for
- a list of strings: a sequence of prompts to send to the model
- this is useful in the use case of simulating a conversation with the model by defining the user prompts sequentially
- a list of dictionaries with keys "role" and "parts", where "role" is one of "user", "model", or "system" and "parts" is the message
- this is useful in the case of passing in some conversation history or to pass in a system prompt to the model
- note that only the prompt in the list can be a system prompt, and the rest must be user or model prompts
The last format is also useful for the case where you want to pass in some conversation history to the model. It is also how we can define multimodal prompts to the model - more details in the Multimodal prompting with Gemini API notebook.
We have created an input file in data/input/gemini-example.jsonl with an example of each of these cases as an illustration.
settings = Settings(data_folder="./data", max_queries=30)
experiment = Experiment(file_name="gemini-example.jsonl", settings=settings)
We set max_queries to 30 so we send 30 queries a minute (every 2 seconds).
print(settings)
Settings: data_folder=./data, max_queries=30, max_attempts=3, parallel=False Subfolders: input_folder=./data/input, output_folder=./data/output, media_folder=./data/media
len(experiment.experiment_prompts)
6
We can see the prompts that we have in the experiment_prompts attribute:
experiment.experiment_prompts
[{'id': 0,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': 'How does technology impact us?',
'safety_filter': 'none',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}},
{'id': 1,
'api': 'gemini',
'model_name': 'gemini-1.0-pro',
'prompt': 'How does technology impact us?',
'safety_filter': 'few',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}},
{'id': 2,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': ['How does international trade create jobs?',
'I want a joke about that'],
'safety_filter': 'some',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}},
{'id': 3,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'},
{'role': 'user',
'parts': 'What efforts are being made to keep the hakka language alive?'}],
'safety_filter': 'default',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}},
{'id': 4,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'user',
'parts': 'What efforts are being made to keep the hakka language alive?'},
{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'}],
'safety_filter': 'most',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}},
{'id': 5,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'},
{'role': 'user', 'parts': "Hello, I'm Bob and I'm 6 years old"},
{'role': 'model', 'parts': 'Hi Bob, how may I assist you?'},
{'role': 'user', 'parts': 'How old will I be next year?'}],
'safety_filter': 'most',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100}}]
- In the first prompt (
"id": 0), we have a"prompt"key which is a string and specify a"model_name"key to be "gemini-1.5-flash" - In the second prompt (
"id": 1), we have a"prompt"key is also a string but we specify a"model_name"key to be "gemini-1.0-pro". - In the third prompt (
"id": 2), we have a"prompt"key which is a list of strings. - In the fourth prompt (
"id": 3), we have a"prompt"key which is a list of dictionaries. These dictionaries have a "role" and "parts" key. This acts as passing in a system prompt. Here, we just have a system prompt before a user prompt. - In the fifth prompt (
"id": 4), we have a"prompt"key which is a list of dictionaries. These dictionaries have a "role" and "parts" key but here, we have a user prompt and then a system prompt. As mentioned above, only the first prompt in the list can be a system prompt. We should get an error for this particular prompt. - In the sixth prompt (
"id": 5), we have a"prompt"key which is a list of dictionaries. These dictionaries have a "role" and "parts" key. Here, we have a system prompt and a series of user/model interactions before finally having a user prompt. This acts as passing in a system prompt and conversation history.
Note that for each of these prompt dicts, we have "model_name": "gemini-1.5-flash", besides "id": 1 where we have "model_name": "gemini-1.0-pro".
Safety filters with Gemini API¶
With the Gemini API, it is possible to configure the safety filters (see the safety settings docs). We can set the "safety_filter" key in the prompt dict where the options are:
"none": corresponds to "Block none" orBLOCK_NONE"few": corresponds to "Block few" orBLOCK_ONLY_HIGH"default"or"some": corresponds to "Block some" orBLOCK_HIGH_AND_MEDIUM"most": corresponds to "Block most" orBLOCK_LOW_AND_ABOVE
In the example input file, we have set the "safety_filter" key to each of these options.
Running the experiment¶
We now can run the experiment using the async method process which will process the prompts in the input file asynchronously. Note that a new folder named timestamp-gemini-example (where "timestamp" is replaced with the actual date and time of processing) will be created in the output directory and we will move the input file to the output directory. As the responses come in, they will be written to the output file and there are logs that will be printed to the console as well as being written to a log file in the output directory.
responses, avg_query_processing_time = await experiment.process()
Sending 6 queries at 30 QPM with RI of 2.0s (attempt 1/3): 100%|██████████| 6/6 [00:12<00:00, 2.00s/query] Waiting for responses (attempt 1/3): 100%|██████████| 6/6 [00:00<00:00, 10.17query/s]
We can see that the responses are written to the output file, and we can also see them as the returned object. From running the experiment, we obtain prompt dicts where there is now a "response" key which contains the response(s) from the model.
For the case where the prompt is a list of strings, we see that the response is a list of strings where each string is the response to the corresponding prompt.
responses
[{'id': 0,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': 'How does technology impact us?',
'safety_filter': 'none',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-31',
'response': "Technology impacts us in profound and multifaceted ways, shaping our lives in every aspect from communication and work to entertainment and healthcare. Here's a breakdown of key impacts:\n\n**Positive Impacts:**\n\n* **Enhanced Communication:** Technology has revolutionized communication. We can connect with people across the globe instantly through video calls, instant messaging, and social media. This fosters stronger relationships, facilitates business, and enables global collaboration.\n* **Increased Efficiency and Productivity:** Technology automates tasks, streamlines processes,",
'safety_attributes': {'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '1',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'MAX_TOKENS'}},
{'id': 1,
'api': 'gemini',
'model_name': 'gemini-1.0-pro',
'prompt': 'How does technology impact us?',
'safety_filter': 'few',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-33',
'response': '**Positive Impacts of Technology**\n\n* **Enhanced Communication:** Social media, messaging apps, and video conferencing facilitate seamless connection and information exchange globally.\n* **Increased Accessibility:** Technology provides equal access to education, healthcare, and information for individuals regardless of location or circumstances.\n* **Improved Productivity:** Automation, artificial intelligence, and data analytics streamline processes, improve efficiency, and reduce time consumption.\n* **Medical Advancements:** Medical devices, surgical robots, and telemedicine enhance patient care, improve',
'safety_attributes': {'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '1',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'MAX_TOKENS'}},
{'id': 2,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': ['How does international trade create jobs?',
'I want a joke about that'],
'safety_filter': 'some',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-35',
'response': ["International trade creates jobs in a multitude of ways, boosting both domestic and global economies. Here's a breakdown of how it works:\n\n**1. Increased Production and Export Jobs:**\n\n* **Specialization and Comparative Advantage:** Countries specialize in producing goods and services where they have a comparative advantage (lower opportunity cost). This leads to increased efficiency and higher production levels.\n* **Export Jobs:** Producers of these specialized goods and services find new markets abroad, leading to increased demand for their products",
'Why did the economist get a job in international trade?\n\nBecause he was tired of being a "jobless" economist! 😜 \n'],
'safety_attributes': [{'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '1',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'MAX_TOKENS'},
{'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '2',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'STOP'}]},
{'id': 3,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'},
{'role': 'user',
'parts': 'What efforts are being made to keep the hakka language alive?'}],
'safety_filter': 'default',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-37',
'response': 'Efforts to preserve Hakka include language schools, cultural festivals, and online resources. \n',
'safety_attributes': {'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '1',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'STOP'}},
{'id': 4,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'user',
'parts': 'What efforts are being made to keep the hakka language alive?'},
{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'}],
'safety_filter': 'most',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-39',
'response': "TypeError - if api == 'gemini', then the prompt must be a str, list[str], or list[dict[str,str]] where the dictionary contains the keys 'role' and 'parts' only, and the values for 'role' must be one of 'user' or 'model', except for the first message in the list of dictionaries can be a system message with the key 'role' set to 'system'."},
{'id': 5,
'api': 'gemini',
'model_name': 'gemini-1.5-flash',
'prompt': [{'role': 'system',
'parts': 'You are a helpful assistant designed to answer questions briefly.'},
{'role': 'user', 'parts': "Hello, I'm Bob and I'm 6 years old"},
{'role': 'model', 'parts': 'Hi Bob, how may I assist you?'},
{'role': 'user', 'parts': 'How old will I be next year?'}],
'safety_filter': 'most',
'parameters': {'candidate_count': 1,
'temperature': 1,
'max_output_tokens': 100},
'timestamp_sent': '18-10-2024-09-57-41',
'response': 'You will be 7 years old next year! \n',
'safety_attributes': {'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
'HARM_CATEGORY_HATE_SPEECH': '1',
'HARM_CATEGORY_HARASSMENT': '1',
'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
'blocked': '[False, False, False, False]',
'finish_reason': 'STOP'}}]
Also notice how with the Gemini API, we record some additional information related to the safety attributes.
Running the experiment via the command line¶
We can also run the experiment via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. examples/gemini):
prompto_run_experiment --file data/input/gemini-example.jsonl --max-queries 30