Using prompto with Vertex AI¶

In [1]:

from prompto.settings import Settings
from prompto.experiment import Experiment
from dotenv import load_dotenv
import warnings
import os

from prompto.settings import Settings from prompto.experiment import Experiment from dotenv import load_dotenv import warnings import os

When using prompto to query models from the Vertex AI API, lines in our experiment .jsonl files must have "api": "vertexai" in the prompt dict.

Environment variables¶

For the Vertex AI API, there are two environment variables that could be set:

• VERTEXAI_PROJECT_ID: the project-id for the Vertex AI API
• VERTEXAI_LOCATION_ID: the location-id for the Vertex AI API

As mentioned in the environment variables docs, there are also model-specific environment variables too which can be utilised. In particular, when you specify a model_name key in a prompt dict, one could also specify a VERTEXAI_PROJECT_ID_model_name or VERTEXAI_LOCATION_ID_model_name environment variables to indicate the project-id and location-id for that particular model (where "model_name" is replaced to whatever the corresponding value of the model_name key is).

To set environment variables, one can simply have these in a .env file which specifies these environment variables as key-value pairs:

VERTEXAI_PROJECT_ID=<YOUR-VERTEXAI-PROJECT-ID>
VERTEXAI_LOCATION_ID=<YOUR-VERTEXAI-LOCATION-ID>

If you make this file, you can run the following which should return True if it's found one, or False otherwise:

In [2]:

load_dotenv(dotenv_path=".env")

load_dotenv(dotenv_path=".env")

Out[2]:

True

Now, we obtain those values. We some warnings if the VERTEXAI_PROJECT_ID or VERTEXAI_LOCATION_ID environment variables haven't been set. However, note that when using Vertex AI, you can actually set the default project and location using the gcloud CLI, so these aren't strictly necessary.

In [3]:

VERTEXAI_PROJECT_ID = os.environ.get("VERTEXAI_PROJECT_ID")
if VERTEXAI_PROJECT_ID is None:
    warnings.warn("VERTEXAI_PROJECT_ID is not set")

VERTEXAI_PROJECT_ID = os.environ.get("VERTEXAI_PROJECT_ID") if VERTEXAI_PROJECT_ID is None: warnings.warn("VERTEXAI_PROJECT_ID is not set")

In [4]:

VERTEXAI_LOCATION_ID = os.environ.get("VERTEXAI_LOCATION_ID")
if VERTEXAI_LOCATION_ID is None:
    warnings.warn("VERTEXAI_LOCATION_ID is not set")

VERTEXAI_LOCATION_ID = os.environ.get("VERTEXAI_LOCATION_ID") if VERTEXAI_LOCATION_ID is None: warnings.warn("VERTEXAI_LOCATION_ID is not set")

Types of prompts¶

With the Vertex AI API, the prompt (given via the "prompt" key in the prompt dict) can take several forms:

• a string: a single prompt to obtain a response for
• a list of strings: a sequence of prompts to send to the model
  • this is useful in the use case of simulating a conversation with the model by defining the user prompts sequentially
• a list of dictionaries with keys "role" and "parts", where "role" is one of "user", "model", or "system" and "parts" is the message/prompt to the model
  • this is useful in the case of passing in some conversation history or to pass in a system prompt to the model
  • note that only one prompt in the list can be a system prompt and it must be the first - the rest must be user or model prompts

The last format is also useful for the case where you want to pass in some conversation history to the model. It is also how we can define multimodal prompts to the model - more details in the Multimodal prompting with Vertex AI notebook.

We have created an input file in data/input/vertexai-example.jsonl with an example of each of these cases as an illustration.

In [5]:

settings = Settings(data_folder="./data", max_queries=30)
experiment = Experiment(file_name="vertexai-example.jsonl", settings=settings)

settings = Settings(data_folder="./data", max_queries=30) experiment = Experiment(file_name="vertexai-example.jsonl", settings=settings)

We set max_queries to 30 so we send 30 queries a minute (every 2 seconds).

In [6]:

print(settings)

print(settings)

Settings: data_folder=./data, max_queries=30, max_attempts=3, parallel=False
Subfolders: input_folder=./data/input, output_folder=./data/output, media_folder=./data/media

In [7]:

len(experiment.experiment_prompts)

len(experiment.experiment_prompts)

Out[7]:

6

We can see the prompts that we have in the experiment_prompts attribute:

In [8]:

experiment.experiment_prompts

experiment.experiment_prompts

Out[8]:

[{'id': 0,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': 'How does technology impact us?',
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}},
 {'id': 1,
  'api': 'vertexai',
  'model_name': 'gemini-1.0-pro-002',
  'prompt': 'How does technology impact us?',
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}},
 {'id': 2,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': ['How does international trade create jobs?',
   'I want a joke about that'],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}},
 {'id': 3,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user',
    'parts': 'What efforts are being made to keep the hakka language alive?'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}},
 {'id': 4,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'user',
    'parts': 'What efforts are being made to keep the hakka language alive?'},
   {'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}},
 {'id': 5,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user', 'parts': "Hello, I'm Bob and I'm 6 years old"},
   {'role': 'model', 'parts': 'Hi Bob, how may I assist you?'},
   {'role': 'user', 'parts': 'How old will I be next year?'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000}}]

• In the first prompt ("id": 0), we have a "prompt" key which is a string and specify a "model_name" key to be "gemini-1.5-flash"
• In the second prompt ("id": 1), we have a "prompt" key is also a string but we specify a "model_name" key to be "gemini-1.0-pro".
• In the third prompt ("id": 2), we have a "prompt" key which is a list of strings.
• In the fourth prompt ("id": 3), we have a "prompt" key which is a list of dictionaries. These dictionaries have a "role" and "parts" key. This acts as passing in a system prompt. Here, we just have a system prompt before a user prompt.
• In the fifth prompt ("id": 4), we have a "prompt" key which is a list of dictionaries. These dictionaries have a "role" and "parts" key but here, we have a user prompt and then a system prompt. As mentioned above, only the first prompt in the list can be a system prompt. We should get an error for this particular prompt.
• In the sixth prompt ("id": 5), we have a "prompt" key which is a list of dictionaries. These dictionaries have a "role" and "parts" key. Here, we have a system prompt and a series of user/model interactions before finally having a user prompt. This acts as passing in a system prompt and conversation history.

Note that for each of these prompt dicts, we have "model_name": "gemini-1.5-flash", besides "id": 1 where we have "model_name": "gemini-1.0-pro".

Safety filters with VertexAI API¶

With the Gemini API, it is possible to configure the safety filters (see the safety settings docs). We can set the "safety_filter" key in the prompt dict where the options are:

• "none": corresponds to "Block none" or BLOCK_NONE
• "few": corresponds to "Block few" or BLOCK_ONLY_HIGH
• "default" or "some": corresponds to "Block some" or BLOCK_HIGH_AND_MEDIUM
• "most": corresponds to "Block most" or BLOCK_LOW_AND_ABOVE

In the example input file, we have set the "safety_filter" key to each of these options.

Running the experiment¶

We now can run the experiment using the async method process which will process the prompts in the input file asynchronously. Note that a new folder named timestamp-vertexai-example (where "timestamp" is replaced with the actual date and time of processing) will be created in the output directory and we will move the input file to the output directory. As the responses come in, they will be written to the output file and there are logs that will be printed to the console as well as being written to a log file in the output directory.

In [9]:

responses, avg_query_processing_time = await experiment.process()

responses, avg_query_processing_time = await experiment.process()

Sending 6 queries at 30 QPM with RI of 2.0s (attempt 1/3): 100%|██████████| 6/6 [00:14<00:00,  2.48s/query]
Waiting for responses (attempt 1/3): 100%|██████████| 6/6 [00:00<00:00,  7.46query/s]

We can see that the responses are written to the output file, and we can also see them as the returned object. From running the experiment, we obtain prompt dicts where there is now a "response" key which contains the response(s) from the model.

For the case where the prompt is a list of strings, we see that the response is a list of strings where each string is the response to the corresponding prompt.

In [10]:

responses

responses

Out[10]:

[{'id': 0,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': 'How does technology impact us?',
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-24',
  'response': "Technology's impact on us is profound and multifaceted, affecting nearly every aspect of our lives.  Here's a breakdown of some key areas:\n\n**Positive Impacts:**\n\n* **Improved Communication:**  Instant communication across vast distances through email, messaging apps, video calls, and social media connects people globally.\n* **Increased Efficiency and Productivity:** Automation, software, and tools streamline tasks in various fields, boosting productivity in work and daily life.\n* **Access to Information:** The internet provides unparalleled access to information, education, and diverse perspectives, empowering individuals and fostering learning.\n* **Advancements in Healthcare:** Medical technology leads to earlier diagnosis, better treatments, minimally invasive surgeries, and improved overall healthcare outcomes.\n* **Economic Growth:** Technological innovation drives economic growth by creating new industries, jobs, and opportunities.\n* **Enhanced Entertainment and Leisure:** Streaming services, gaming, virtual reality, and other technologies offer diverse and engaging entertainment options.\n* **Accessibility for People with Disabilities:** Assistive technologies enhance the lives of individuals with disabilities, providing greater independence and participation.\n* **Environmental Monitoring and Conservation:** Technology helps monitor environmental conditions, track pollution, and develop sustainable solutions.\n\n\n**Negative Impacts:**\n\n* **Job Displacement:** Automation can lead to job losses in certain sectors, requiring workforce retraining and adaptation.\n* **Privacy Concerns:** Data collection and surveillance technologies raise concerns about privacy violation and potential misuse of personal information.\n* **Digital Divide:** Unequal access to technology creates a digital divide, exacerbating existing social and economic inequalities.\n* **Mental Health Issues:** Excessive social media use, cyberbullying, and online harassment can negatively impact mental health and wellbeing.\n* **Spread of Misinformation:** The rapid spread of misinformation and fake news online poses a threat to informed decision-making and social cohesion.\n* **Addiction and Dependence:** Technology can be addictive, leading to dependence and negative consequences for physical and mental health.\n* **Security Risks:** Cyberattacks, data breaches, and online fraud pose significant security risks to individuals and organizations.\n* **Environmental Impact:** The manufacturing and disposal of electronic devices contribute to pollution and resource depletion.\n\n\n**Overall:**\n\nTechnology's impact is neither inherently good nor bad. It's a powerful tool that can be used for positive or negative purposes.  Its effects depend on how we choose to develop, implement, and regulate it.  A responsible approach to technology development and usage, focusing on ethical considerations, inclusivity, and sustainability, is crucial to maximizing its benefits and mitigating its risks.\n",
  'safety_attributes': {'HARM_CATEGORY_HATE_SPEECH': '1',
   'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
   'HARM_CATEGORY_HARASSMENT': '1',
   'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
   'blocked': '[False, False, False, False]',
   'finish_reason': 'STOP'}},
 {'id': 3,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user',
    'parts': 'What efforts are being made to keep the hakka language alive?'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-30',
  'response': 'Efforts to preserve Hakka include language classes, media production (songs, films), online communities, and integrating Hakka into education.\n',
  'safety_attributes': {'HARM_CATEGORY_HATE_SPEECH': '1',
   'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
   'HARM_CATEGORY_HARASSMENT': '1',
   'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
   'blocked': '[False, False, False, False]',
   'finish_reason': 'STOP'}},
 {'id': 4,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'user',
    'parts': 'What efforts are being made to keep the hakka language alive?'},
   {'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-32',
  'response': "TypeError - if api == 'vertexai', then the prompt must be a str, list[str], or list[dict[str,str]] where the dictionary contains the keys 'role' and 'parts' only, and the values for 'role' must be one of 'user' or 'model', except for the first message in the list of dictionaries can be a system message with the key 'role' set to 'system'."},
 {'id': 2,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': ['How does international trade create jobs?',
   'I want a joke about that'],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-28',
  'response': ["International trade creates jobs in several ways, both directly and indirectly:\n\n**Direct Job Creation:**\n\n* **Export-oriented industries:**  Companies that produce goods or services for export directly hire workers in manufacturing, agriculture, technology, and services (e.g., logistics, marketing, design).  The demand for these goods and services from foreign markets fuels job creation within these sectors.  Think of a clothing manufacturer exporting to Europe, creating jobs for tailors, designers, and shipping personnel.\n* **Import-competing industries:** While often framed negatively, some domestic industries that compete with imports also create jobs.  These industries may adapt and innovate to remain competitive, leading to jobs in research and development, improved efficiency, and specialized production.  A domestic car manufacturer, facing competition from imports, might invest in creating more fuel-efficient vehicles, thus creating engineering and manufacturing jobs.\n* **Trade-related services:**  Many jobs are created in support of international trade. This includes transportation (shipping, trucking, air freight), logistics (warehousing, customs brokerage), finance (international banking, currency exchange), and legal services (international trade law).\n\n\n**Indirect Job Creation:**\n\n* **Increased productivity and efficiency:**  Trade allows countries to specialize in producing goods and services where they have a comparative advantage. This leads to increased productivity and efficiency, creating more output with the same or fewer inputs.  This freed-up labor and capital can then be used to create jobs in other sectors.  For instance, if a country becomes more efficient at producing agricultural goods due to international trade, it frees up workers to pursue jobs in manufacturing or technology.\n* **Lower input costs:** Access to cheaper raw materials, intermediate goods, and components from other countries lowers production costs for domestic firms.  This allows them to be more competitive, expand production, and hire more workers.  A furniture maker, for example, might use cheaper imported wood to lower its costs and expand its workforce.\n* **Increased consumer spending:** Cheaper imports increase consumer purchasing power. This increased demand stimulates economic activity, leading to more jobs in various sectors, from retail to entertainment.  If consumers have more money to spend due to cheaper imports, they might spend it on domestic services, creating jobs in the restaurant or tourism industries.\n* **Foreign direct investment (FDI):**  International trade attracts FDI, as foreign companies invest in establishing operations in countries with access to resources, markets, or skilled labor. This investment creates jobs directly through the establishment of new factories or offices and indirectly through supporting businesses.\n\n\n**However, it's important to note:**\n\n* **Job displacement:**  International trade can also lead to job losses in industries that struggle to compete with imports. This is often a concern that requires retraining and adjustment programs to help displaced workers transition to new jobs.\n* **Wage stagnation:**  Increased competition from low-wage countries can put downward pressure on wages in some sectors, especially for less-skilled workers.\n* **Uneven distribution of benefits:**  The benefits of international trade are not always evenly distributed, with some regions or sectors benefiting more than others.\n\nIn conclusion, while international trade can lead to job displacement in certain sectors, it generally creates more jobs than it destroys, especially when considering both direct and indirect effects.  The overall impact on employment depends on a variety of factors, including government policies, the structure of the economy, and the adaptability of the workforce.\n",
   'Why did the economist get fired from his job at the import/export company?  Because he kept saying, "It\'s a comparative advantage... it\'s a comparative advantage..." while laying off half the workforce!\n'],
  'safety_attributes': [{'HARM_CATEGORY_HATE_SPEECH': '1',
    'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
    'HARM_CATEGORY_HARASSMENT': '1',
    'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
    'blocked': '[False, False, False, False]',
    'finish_reason': 'STOP'},
   {'HARM_CATEGORY_HATE_SPEECH': '1',
    'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
    'HARM_CATEGORY_HARASSMENT': '1',
    'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
    'blocked': '[False, False, False, False]',
    'finish_reason': 'STOP'}]},
 {'id': 1,
  'api': 'vertexai',
  'model_name': 'gemini-1.0-pro-002',
  'prompt': 'How does technology impact us?',
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-26',
  'response': "## Technology's Impact on Humans\n\nTechnology has significantly impacted human lives in countless ways, shaping our society, culture, and individual experiences. Here's a brief overview of its impact on various aspects:\n\n**Positive Impacts:**\n\n* **Communication:** Technology has revolutionized communication, allowing us to connect with people across the globe instantly. From email and social media to video conferencing and messaging apps, staying in touch has never been easier.\n* **Information Access:** The internet provides access to a vast pool of information, enabling us to research, learn, and stay informed about various topics. Search engines, online libraries, and educational platforms empower individuals with knowledge and understanding.\n* **Healthcare:** Medical technology has made significant advancements, improving diagnoses, treatment options, and overall healthcare outcomes. From advanced imaging techniques to telemedicine and robotic surgery, technology is saving lives and enhancing the quality of life.\n* **Productivity:** Technology has automated many tasks, increasing efficiency and productivity across various industries. From manufacturing and agriculture to finance and healthcare, automation has streamlined processes, saving time and resources.\n* **Entertainment:** Technology offers endless entertainment options, from streaming services and video games to virtual reality experiences. It provides leisure and enjoyment for people of all ages and interests.\n* **Education:** Technology has transformed education, enabling access to online learning platforms, interactive tools, and personalized learning experiences. It empowers individuals to learn at their own pace and access resources beyond the classroom.\n\n**Negative Impacts:**\n\n* **Privacy Concerns:** With the increasing use of technology, concerns about data privacy and security are rising. Social media platforms, online tracking, and government surveillance raise questions about personal information and how it's used.\n* **Social Isolation:** While technology helps connect people, excessive use can lead to social isolation and loneliness. Social media interactions often lack the depth of face-to-face communication, and individuals may become disconnected from their communities.\n* **Cybersecurity Threats:** As technology advances, so do cyber threats. Hacking, malware, and data breaches pose significant risks to individuals and organizations, highlighting the need for strong cybersecurity measures.\n* **Addiction and Mental Health:** Technology addiction can negatively impact mental health, leading to anxiety, depression, and sleep disturbances. The constant stimulation and comparison-driven culture of social media can be detrimental to well-being.\n* **Environmental Impact:** The production and disposal of electronic devices contribute to environmental pollution and resource depletion. E-waste and energy consumption raise concerns about the sustainability of technological advancements.\n\n**Conclusion:**\n\nTechnology has a profound impact on human society, offering both benefits and challenges. Its impact is multifaceted, affecting how we communicate, learn, work, entertain ourselves, and interact with the world around us. It's crucial to understand both the positive and negative aspects of technology and utilize it responsibly to maximize its benefits while minimizing its potential harms.\n\n**Additionally:**\n\n* Technology has a significant impact on the economy, creating new jobs, industries, and economic opportunities. However, it also poses challenges, such as job displacement and the need for workforce reskilling.\n* Technology has also influenced political landscapes, enabling citizen engagement, information dissemination, and political activism. It raises concerns about privacy, censorship, and the spread of misinformation.\n\nIt's important to consider the ethical implications of technological advancements and ensure they align with human values and societal well-being. \n",
  'safety_attributes': {'HARM_CATEGORY_HATE_SPEECH': '1',
   'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
   'HARM_CATEGORY_HARASSMENT': '1',
   'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
   'blocked': '[False, False, False, False]',
   'finish_reason': 'STOP'}},
 {'id': 5,
  'api': 'vertexai',
  'model_name': 'gemini-1.5-flash-002',
  'prompt': [{'role': 'system',
    'parts': 'You are a helpful assistant designed to answer questions briefly.'},
   {'role': 'user', 'parts': "Hello, I'm Bob and I'm 6 years old"},
   {'role': 'model', 'parts': 'Hi Bob, how may I assist you?'},
   {'role': 'user', 'parts': 'How old will I be next year?'}],
  'parameters': {'candidate_count': 1,
   'temperature': 1,
   'max_output_tokens': 1000},
  'timestamp_sent': '18-10-2024-10-58-36',
  'response': 'You will be 7 years old next year.\n',
  'safety_attributes': {'HARM_CATEGORY_HATE_SPEECH': '1',
   'HARM_CATEGORY_DANGEROUS_CONTENT': '1',
   'HARM_CATEGORY_HARASSMENT': '1',
   'HARM_CATEGORY_SEXUALLY_EXPLICIT': '1',
   'blocked': '[False, False, False, False]',
   'finish_reason': 'STOP'}}]

Also notice how with the Vertex AI API, we record some additional information related to the safety attributes.

Running the experiment via the command line¶

We can also run the experiment via the command line. The command is as follows (assuming that your working directory is the current directory of this notebook, i.e. examples/vertexai):

prompto_run_experiment --file data/input/vertexai-example.jsonl --max-queries 30