Using prompto with Azure OpenAI¶

In [1]:

from prompto.settings import Settings
from prompto.experiment import Experiment
from dotenv import load_dotenv
import warnings
import os

from prompto.settings import Settings from prompto.experiment import Experiment from dotenv import load_dotenv import warnings import os

When using prompto to query models from the OpenAI API, lines in our experiment .jsonl files must have "api": "openai" in the prompt dict.

Environment variables¶

For the AzureOpenAI API, there are four environment variables that could be set:

• AZURE_OPENAI_API_KEY: the API key for the Azure OpenAI API
• AZURE_OPENAI_API_ENDPOINT: the endpoint for the Azure OpenAI API
• AZURE_OPENAI_API_VERSION: the version of the Azure OpenAI API (optional)

As mentioned in the environment variables docs, there are also model-specific environment variables too which can be utilised. In particular, when you specify a model_name key in a prompt dict, one could also specify a AZURE_OPENAI_API_KEY_model_name environment variable to indicate the API key used for that particular model (where "model_name" is replaced to whatever the corresponding value of the model_name key is). We will see a concrete example of this later. The same applies for the AZURE_OPENAI_API_ENDPOINT_model_name and AZURE_OPENAI_API_VERSION_model_name environment variables.

To set environment variables, one can simply have these in a .env file which specifies these environment variables as key-value pairs:

AZURE_OPENAI_API_KEY=<YOUR-AZURE-OPENAI-KEY>
AZURE_OPENAI_API_ENDPOINT=<YOUR-AZURE-OPENAI-ENDPOINT>
AZURE_OPENAI_API_VERSION=<DEFAULT-AZURE-OPENAI-API-VERSION>

If you make this file, you can run the following which should return True if it's found one, or False otherwise:

In [2]:

load_dotenv(dotenv_path=".env")

load_dotenv(dotenv_path=".env")

Out[2]:

True

Now, we obtain those values. We raise an error if the AZURE_OPENAI_API_KEY or AZURE_OPENAI_API_ENDPOINT environment variables haven't been set:

In [3]:

AZURE_OPENAI_API_KEY = os.environ.get("AZURE_OPENAI_API_KEY")
if AZURE_OPENAI_API_KEY is None:
    raise ValueError("AZURE_OPENAI_API_KEY is not set")

AZURE_OPENAI_API_KEY = os.environ.get("AZURE_OPENAI_API_KEY") if AZURE_OPENAI_API_KEY is None: raise ValueError("AZURE_OPENAI_API_KEY is not set")

In [4]:

AZURE_OPENAI_API_ENDPOINT = os.environ.get("AZURE_OPENAI_API_ENDPOINT")
if AZURE_OPENAI_API_ENDPOINT is None:
    raise ValueError("AZURE_OPENAI_API_ENDPOINT is not set")

AZURE_OPENAI_API_ENDPOINT = os.environ.get("AZURE_OPENAI_API_ENDPOINT") if AZURE_OPENAI_API_ENDPOINT is None: raise ValueError("AZURE_OPENAI_API_ENDPOINT is not set")

We will only raise a warning if AZURE_OPENAI_API_VERSION hasn't been set:

In [5]:

AZURE_OPENAI_API_VERSION = os.environ.get("AZURE_OPENAI_API_VERSION")
if AZURE_OPENAI_API_VERSION is None:
    warnings.warn("AZURE_OPENAI_API_VERSION is not set")
else:
    print(f"Default AzureOpenAI version: {AZURE_OPENAI_API_VERSION}")

AZURE_OPENAI_API_VERSION = os.environ.get("AZURE_OPENAI_API_VERSION") if AZURE_OPENAI_API_VERSION is None: warnings.warn("AZURE_OPENAI_API_VERSION is not set") else: print(f"Default AzureOpenAI version: {AZURE_OPENAI_API_VERSION}")

Default AzureOpenAI version: 2024-02-01

If you get any errors or warnings in the above two cells, try to fix your .env file like the example we have above to get these variables set.

Types of prompts¶

With the OpenAI API, the prompt (given via the "prompt" key in the prompt dict) can take several forms:

• a string: a single prompt to obtain a response for
• a list of strings: a sequence of prompts to send to the model
  • this is useful in the use case of simulating a conversation with the model by defining the user prompts sequentially
• a list of dictionaries with keys "role" and "content", where "role" is one of "user", "assistant", or "system" and "content" is the message
  • this is useful in the case of passing in some conversation history or to pass in a system prompt to the model

We have created an input file in data/input/azure-openai-example.jsonl with an example of each of these cases as an illustration.

Note that for each of these (besides "id": 1), we have "model_name": "reginald-gpt4" which refers to a specific GPT-4 deployment that we have on our Azure subscription when developing this notebook. See below for an overview of each of the prompts in the input file.

In [6]:

settings = Settings(data_folder="./data", max_queries=30)
experiment = Experiment(file_name="azure-openai-example.jsonl", settings=settings)

settings = Settings(data_folder="./data", max_queries=30) experiment = Experiment(file_name="azure-openai-example.jsonl", settings=settings)

We set max_queries to 30 so we send 30 queries a minute (every 2 seconds).

In [7]:

print(settings)

print(settings)

Settings: data_folder=./data, max_queries=30, max_attempts=3, parallel=False
Subfolders: input_folder=./data/input, output_folder=./data/output, media_folder=./data/media

In [8]:

len(experiment.experiment_prompts)

len(experiment.experiment_prompts)

Out[8]:

5

We can see the prompts that we have in the experiment_prompts attribute:

In [9]:

experiment.experiment_prompts

experiment.experiment_prompts

Out[9]:

[{'id': 0,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': 'How does technology impact us?',
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100}},
 {'id': 1,
  'api': 'azure-openai',
  'model_name': 'unknown-model-name',
  'prompt': 'How does technology impact us?',
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100}},
 {'id': 2,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': ['How does international trade create jobs?',
   'I want a joke about that'],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100}},
 {'id': 3,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': [{'role': 'system',
    'content': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user',
    'content': 'What efforts are being made to keep the hakka language alive?'}],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100}},
 {'id': 4,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': [{'role': 'system',
    'content': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user', 'content': "Hello, I'm Bob and I'm 6 years old"},
   {'role': 'assistant', 'content': 'Hi Bob, how may I assist you?'},
   {'role': 'user', 'content': 'How old will I be next year?'}],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100}}]

• In the first prompt ("id": 0), we have a "prompt" key which is a string.
• In the second prompt ("id": 1), we have a "prompt" key is also a string but we specify a "model_name" key to be "unknown-model-name". We do this to illustrate what happens when a model name is not recognised as being a deployed model in your Azure subscription - we will see that we get an error for this.
• In the third prompt ("id": 2), we have a "prompt" key which is a list of strings.
• In the fourth prompt ("id": 3), we have a "prompt" key which is a list of dictionaries. These dictionaries have a "role" and "content" key. This acts as passing in a system prompt. Here, we just have a system prompt before a user prompt.
• In the fifth prompt ("id": 4), we have a "prompt" key which is a list of dictionaries. These dictionaries have a "role" and "content" key. Here, we have a system prompt and a series of user/assistant interactions before finally having a user prompt. This acts as passing in a system prompt and conversation history.

Note that for each of these prompt dicts (besides "id": 1), we have "model_name": "reginald-gpt4" which refers to a specific GPT-4 deployment that we have on our Azure subscription when developing this notebook.

Running the experiment¶

We now can run the experiment using the async method process which will process the prompts in the input file asynchronously. Note that a new folder named timestamp-openai-example (where "timestamp" is replaced with the actual date and time of processing) will be created in the output directory and we will move the input file to the output directory. As the responses come in, they will be written to the output file and there are logs that will be printed to the console as well as being written to a log file in the output directory.

In [10]:

responses, avg_query_processing_time = await experiment.process()

responses, avg_query_processing_time = await experiment.process()

Sending 5 queries  (attempt 1/3):   0%|          | 0/5 [00:00<?, ?query/s]

Sending 5 queries  (attempt 1/3): 100%|██████████| 5/5 [00:10<00:00,  2.00s/query]
Waiting for responses  (attempt 1/3): 100%|██████████| 5/5 [00:29<00:00,  5.89s/query]
Sending 1 queries  (attempt 2/3): 100%|██████████| 1/1 [00:02<00:00,  2.00s/query]
Waiting for responses  (attempt 2/3): 100%|██████████| 1/1 [00:00<00:00, 11.46query/s]
Sending 1 queries  (attempt 3/3): 100%|██████████| 1/1 [00:02<00:00,  2.00s/query]
Waiting for responses  (attempt 3/3): 100%|██████████| 1/1 [00:00<00:00,  9.39query/s]

We can see that the responses are written to the output file, and we can also see them as the returned object. From running the experiment, we obtain prompt dicts where there is now a "response" key which contains the response(s) from the model.

For the case where the prompt is a list of strings, we see that the response is a list of strings where each string is the response to the corresponding prompt.

Note here, for our specific Azure subscription, we haven't got a model with deployment name "gpt-3.5-turbo" and hence, we actually receive an error message in the response for the second prompt.

In [11]:

responses

responses

Out[11]:

[{'id': 0,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': 'How does technology impact us?',
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100},
  'response': 'Technology impacts our lives in numerous ways, both positive and negative. Here is a synthesis of the key ways technology can affect individuals and society:\n\nPositive Impacts:\n\n1. **Access to Information**: Today we have an unprecedented amount of information at our fingertips thanks to the internet, search engines, and digital libraries.\n\n2. **Communication**: With smartphones, messaging apps, and social media, we can communicate instantaneously with people across the globe, fostering relationships and collaborations.\n\n3. **Convenience**: Online'},
 {'id': 4,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': [{'role': 'system',
    'content': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user', 'content': "Hello, I'm Bob and I'm 6 years old"},
   {'role': 'assistant', 'content': 'Hi Bob, how may I assist you?'},
   {'role': 'user', 'content': 'How old will I be next year?'}],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100},
  'response': "If you're 6 years old now, you'll be 7 years old next year."},
 {'id': 3,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': [{'role': 'system',
    'content': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user',
    'content': 'What efforts are being made to keep the hakka language alive?'}],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100},
  'response': "Efforts to keep the Hakka language alive include:\n\n1. **Language Education**: The establishment of Hakka language courses in schools, particularly in regions with significant Hakka populations like Taiwan, Hong Kong, and parts of China.\n\n2. **Media Broadcasting**: Hakka TV and radio stations are dedicated to producing content in Hakka, which helps maintain the language's presence in daily life.\n\n3. **Cultural Promotion**: Cultural festivals and events that celebrate Hakka traditions often incorporate the language, promoting"},
 {'id': 2,
  'api': 'azure-openai',
  'model_name': 'reginald-gpt4',
  'prompt': ['How does international trade create jobs?',
   'I want a joke about that'],
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100},
  'response': ['International trade creates jobs through various mechanisms and over different phases, from production to distribution. Here are some ways in which engaging in international trade can lead to job creation:\n\n1. Export Opportunities:\n   - When a country exports goods and services, it generates sales and revenue from overseas markets, which can lead to the expansion of local businesses and thus the creation of new jobs.\n   - Increased demand for exports can lead to the need for more workers in the manufacturing sector, as well as in supporting sectors such',
   "Sure, here's a light-hearted take on international trade and job creation:\n\nWhy did the tariff get a job as a comedian?\n\nBecause whenever it took the stage, a bunch of jobs got created just to figure out how to work around it!"]},
 {'id': 1,
  'api': 'azure-openai',
  'model_name': 'unknown-model-name',
  'prompt': 'How does technology impact us?',
  'parameters': {'n': 1, 'temperature': 1, 'max_tokens': 100},
  'response': "An unexpected error occurred when querying the API: NotFoundError - Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.'}} after maximum 3 attempts"}]

Running the experiment via the command line¶

We can also run the experiment via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. examples/azure-openai):

prompto_run_experiment --file data/input/azure-openai-example.jsonl --max-queries 30