Using prompto for multimodal prompting with Anthropic¶
from prompto.settings import Settings
from prompto.experiment import Experiment
from dotenv import load_dotenv
import warnings
import os
When using prompto to query models from the Anthropic API, lines in our experiment .jsonl files must have "api": "anthropic" in the prompt dict.
Environment variables¶
For the Anthropic API, there are two environment variables that could be set:
ANTHROPIC_API_KEY: the API key for the Anthropic API
As mentioned in the environment variables docs, there are also model-specific environment variables too which can be utilised. In particular, when you specify a model_name key in a prompt dict, one could also specify a ANTHROPIC_API_KEY_model_name environment variable to indicate the API key used for that particular model (where "model_name" is replaced to whatever the corresponding value of the model_name key is). We will see a concrete example of this later.
To set environment variables, one can simply have these in a .env file which specifies these environment variables as key-value pairs:
ANTHROPIC_API_KEY=<YOUR-ANTHROPIC-KEY>
If you make this file, you can run the following which should return True if it's found one, or False otherwise:
load_dotenv(dotenv_path=".env")
True
Now, we obtain those values. We raise an error if the ANTHROPIC_API_KEY environment variable hasn't been set:
ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")
if ANTHROPIC_API_KEY is None:
raise ValueError("ANTHROPIC_API_KEY is not set")
If you get any errors or warnings in the above two cells, try to fix your .env file like the example we have above to get these variables set.
Types of prompts¶
With the Anthropic API, the prompt (given via the "prompt" key in the prompt dict) can take several forms:
- a string: a single prompt to obtain a response for
- a list of strings: a sequence of prompts to send to the model
- this is useful in the use case of simulating a conversation with the model by defining the user prompts sequentially
- a list of dictionaries with keys "role" and "content", where "role" is one of "user", "assistant", or "system" and "content" is the message
- this is useful in the case of passing in some conversation history or to pass in a system prompt to the model
Multimodal prompts¶
For prompting the model with multimodal inputs, we use this last format where we define a prompt by specifying the role of the prompt and then a list of parts that make up the prompt. Individual pieces of the part can be text, images or video which are passed to the model as a multimodal input. In this setting, the prompt can be defined flexibly with text interspersed with images or video.
When specifying an individual part of the prompt, we define this using a dictionary with keys:
"type"is one of"text"or"image"- if
"type"is"text", then you must have a "text" key with the text content - if
"type"is"image", then you must have a"source"key. This must also be a dictionary with keys "media" specifying the local path of an image and "media_type" specifying the type of media (e.g."image/jpeg")
This is similar to how you'd set up a multimodal prompt for the Anthropic API (see Anthropic's documentation).
You can also pass in a string which will be taken as a text prompt.
An example of a multimodal prompt is the following:
[
{
"role": "user",
"content": [
"what is in this image?",
{"type": "image", "source": {"media": "image.jpg", "media_type": "image/jpeg"}},
]
},
]
Here, we have a list of one dictionary where we specify the "role" as "user" and "content" as a list of two elements: the first specifies a text string and the second is a dictionary specifying an image.
For this notebook, we have created an input file in data/input/anthropic-multimodal-example.jsonl with several multimodal prompts with local files as an illustration.
Specifying local files¶
When specifying the local files, the file paths must be relative file paths to the media/ folder in the data folder. For example, if you have an image file image.jpg in the media/ folder, you would specify this as "source": "image.jpg" in the prompt. If you have a video file video.mp4 in the media/videos/ folder, you would specify this as "source": "videos/video.mp4" in the prompt.
settings = Settings(data_folder="./data", max_queries=30)
experiment = Experiment(
file_name="anthropic-multimodal-example.jsonl", settings=settings
)
We set max_queries to 30 so we send 30 queries a minute (every 2 seconds).
print(settings)
Settings: data_folder=./data, max_queries=30, max_attempts=3, parallel=False Subfolders: input_folder=./data/input, output_folder=./data/output, media_folder=./data/media
len(experiment.experiment_prompts)
3
We can see the prompts that we have in the experiment_prompts attribute:
experiment.experiment_prompts
[{'id': 0,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': ['describe what is happening in this image',
{'type': 'image',
'source': {'media': 'pantani_giro.jpg', 'media_type': 'image/jpeg'}}]}],
'parameters': {'temperature': 1, 'max_tokens': 100}},
{'id': 1,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': [{'type': 'image',
'source': {'media': 'mortadella.jpg', 'media_type': 'image/jpeg'}},
'what is this?']}],
'parameters': {'temperature': 1, 'max_tokens': 100}},
{'id': 2,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': ['what is in this image?',
{'type': 'image',
'source': {'media': 'pantani_giro.jpg', 'media_type': 'image/jpeg'}}]},
{'role': 'assistant',
'content': 'This is image shows a group of cyclists.'},
{'role': 'user',
'content': 'are there any notable cyclists in this image? what are their names?'}],
'parameters': {'temperature': 1, 'max_tokens': 100}}]
- In the first prompt (
"id": 0), we have a"prompt"key which specifies a prompt where we ask the model to "describe what is happening in this image" and we pass in an image which is defined using a dictionary with "type" and "source" keys pointing to a file in the media folder - In the second prompt (
"id": 1), we have a"prompt"key which specifies a prompt where we first pass in an image defined using a dictionary with "type" and "source" keys pointing to a file in the media folder and then we ask the model "what is this?" - In the third prompt (
"id": 2), we have a"prompt"key which is a list of dictionaries. Each of these dictionaries have a "role" and "content" key and we specify a user/model interaction. First we ask the model "what is in this image?" along with an image defined by a dictionary with "type" and "source" keys to point to a file in the media folder. We then have a model response and another user query
For each of these prompts, we specify a "model_name" key to be "claude-3-5-sonnet-20241022".
Running the experiment¶
We now can run the experiment using the async method process which will process the prompts in the input file asynchronously. Note that a new folder named timestamp-anthropic-example (where "timestamp" is replaced with the actual date and time of processing) will be created in the output directory and we will move the input file to the output directory. As the responses come in, they will be written to the output file and there are logs that will be printed to the console as well as being written to a log file in the output directory.
responses, avg_query_processing_time = await experiment.process()
Sending 3 queries at 30 QPM with RI of 2.0s (attempt 1/3): 100%|██████████| 3/3 [00:06<00:00, 2.00s/query] Waiting for responses (attempt 1/3): 100%|██████████| 3/3 [00:03<00:00, 1.28s/query]
We can see that the responses are written to the output file, and we can also see them as the returned object. From running the experiment, we obtain prompt dicts where there is now a "response" key which contains the response(s) from the model.
For the case where the prompt is a list of strings, we see that the response is a list of strings where each string is the response to the corresponding prompt.
responses
[{'id': 0,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': ['describe what is happening in this image',
{'type': 'image',
'source': {'media': 'pantani_giro.jpg', 'media_type': 'image/jpeg'}}]}],
'parameters': {'temperature': 1, 'max_tokens': 100},
'timestamp_sent': '29-10-2024-15-36-29',
'response': "This image shows professional cyclists competing in what appears to be a cycling race, likely from the 1990s based on the equipment and team jerseys visible. There are several riders in the frame, including one wearing the distinctive pink jersey (known as the maglia rosa in the Giro d'Italia). The cyclists are riding along a stone wall with an iron fence on top, and they're using classic road racing bikes with distinctive team color schemes - notably some turquoise Bian"},
{'id': 1,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': [{'type': 'image',
'source': {'media': 'mortadella.jpg', 'media_type': 'image/jpeg'}},
'what is this?']}],
'parameters': {'temperature': 1, 'max_tokens': 100},
'timestamp_sent': '29-10-2024-15-36-31',
'response': 'These appear to be mortadella and other Italian cold cuts or processed meats. The larger ones with the string/twine pattern wrapped around them are likely mortadella (a type of Italian bologna), while the pink spotted ones appear to be a different variety of cold cut or processed meat product. The spotted pattern is characteristic of certain Italian deli meats where small pieces of fat or other ingredients are distributed throughout the meat. These are commonly sliced and served in sandwiches or on'},
{'id': 2,
'api': 'anthropic',
'model_name': 'claude-3-5-sonnet-20241022',
'prompt': [{'role': 'user',
'content': ['what is in this image?',
{'type': 'image',
'source': {'media': 'pantani_giro.jpg', 'media_type': 'image/jpeg'}}]},
{'role': 'assistant',
'content': 'This is image shows a group of cyclists.'},
{'role': 'user',
'content': 'are there any notable cyclists in this image? what are their names?'}],
'parameters': {'temperature': 1, 'max_tokens': 100},
'timestamp_sent': '29-10-2024-15-36-33',
'response': "From the image, I can see this appears to be from a professional cycling race, likely from the 1990s based on the equipment and jerseys. While there are professional cyclists in the image, including one wearing the pink jersey (which is the leader's jersey in the Giro d'Italia), I should refrain from identifying specific individuals by name. The image shows a group of riders from various teams including Mercatone Uno and what appears to be racing in a major"}]
Also notice how with the Anthropic API, we record some additional information related to the safety attributes.
Running the experiment via the command line¶
We can also run the experiment via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. examples/anthropic):
prompto_run_experiment --file data/input/anthropic-multimodal-example.jsonl --max-queries 30