Generating the prompts for experiments¶
In the following notebooks, we will compare the performance of prompto
against a traditional synchronous approach to querying LLM endpoints. Before doing this, we need a small sample of prompts to send to each model or API. In this notebook, we will generate a sample of prompts from the Stanford Alpaca project [1].
Downloading and sampling the data¶
First download the alpaca_data.json
from the tatsu-lab/stanford_alpaca
Github repo and save it in the local directory to regenerate a sample from it.
import json
import random
def sample_alpaca_data(n: int, seed: int):
# set seed for sampling
random.seed(seed)
# read in alpaca.json file
with open("./alpaca_data.json") as f:
alpaca_data = json.load(f)
# randomly sample n prompts
sampled_prompts = random.sample(alpaca_data, n)
return sampled_prompts
We generate 100 with a seed of 42:
sample_alpaca = sample_alpaca_data(n=100, seed=42)
sample_alpaca[:5]
[{'instruction': 'What are the advantages of using a Scrum Agile methodology?', 'input': '', 'output': 'The advantages of using a Scrum Agile methodology include improved communication, increased accountability, faster project completion, better quality projects, greater customer satisfaction, and better team morale. Additionally, using Scrum Agile allows teams to more easily pivot and respond to changing requirements, which is often necessary in software development.'}, {'instruction': 'Design an advertisement for a home security product.', 'input': '', 'output': 'Stay secure and monitored with [Product Name]! Our cutting-edge home security system keeps you connected no matter where you are in the world. With built-in sensors and 24/7 monitoring, you can rest easy knowing that your home is safe and sound. Get [Product Name] today and take complete control over your home security.'}, {'instruction': 'Cite a relevant quote that reflects the message of the message.', 'input': 'Message: Never give up on your goals.', 'output': '"Our greatest weakness lies in giving up. The most certain way to succeed is always to try just one more time." - Thomas A. Edison'}, {'instruction': 'Generate two similar sounding but semantically different words to contrast this word.', 'input': 'Light', 'output': 'Bright and Dim.'}, {'instruction': 'Explain the engineering design process.', 'input': '', 'output': 'The engineering design process is an iterative problem-solving approach used to develop engineered products. It involves defining the problem, devising a solution, constructing a prototype, testing the prototype, and making modifications as needed. This process is repeatedly done until the desired results are achieved. It is an important tool for any engineer as it helps improve products, reduce time-to-market, and reduce costs.'}]
Using the prompt templates¶
We use the prompt templates outlined in the README of the tatsu-lab/stanford_alpaca
Github repo to create the prompts for the model.
prompt_template_input = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input:
{input}
### Response:
"""
prompt_template_no_input = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
"""
We use a list compherension to decide whether or not to use the prompt_template_input
template or prompt_template_no_input
depending on the input
key is present in the sample:
sample_prompts = [
(
prompt_template_input.format(
instruction=prompt["instruction"], input=prompt["input"]
)
if prompt["input"] != ""
else prompt_template_no_input.format(instruction=prompt["instruction"])
)
for prompt in sample_alpaca
]
sample_prompts
['Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat are the advantages of using a Scrum Agile methodology?\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDesign an advertisement for a home security product.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCite a relevant quote that reflects the message of the message.\n\n### Input:\nMessage: Never give up on your goals.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate two similar sounding but semantically different words to contrast this word.\n\n### Input:\nLight\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nExplain the engineering design process.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive an example of a web service that could be automated using artificial intelligence.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a block of text, come up with a catchy headline that best represents the text.\n\n### Input:\nThe new coaching program will help you become a better leader and achieve higher professional success.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nEdit the text to make the meaning more precise.\n\n### Input:\nThe President announced an ambitious plan to cut taxes for all Americans.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nAnswer this question: "What are the advantages of using AI in healthcare?"\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a list of elements, find how many possible combinations of the elements can be made.\n\n### Input:\nH, O, C, N\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nAssign a category to the given list of words.\n\n### Input:\nfood, furniture, books\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a sentence, make it more concise while keeping its meaning intact.\n\n### Input:\nThe house was situated in an area where it was surrounded by trees on all sides.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a 5-sentence story about a dog who wants a special treat.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nFind out the capital of Canada. Output the name of the city.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a sentence that illustrates the meaning of the word "capricious".\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nList 5 common bacteria species.\n\n### Response:\n', "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a children's story that includes the character of a friendly lion.\n\n### Response:\n", 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nConvert 100 grams to pounds.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a new design for a watch based on the given input.\n\n### Input:\nGraphic: A cartoon character\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nExplain the concept of descriptive writing.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat is the most common type of conflict in literature?\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName five principles of sustainable development.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompare and contrast two ways of making coffee.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDescribe the impact of Covid-19 on small businesses.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a cautionary tale about a person who ignored safety regulations.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite the general formula for the perimeter of a right triangle.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a URL using a given text-string\n\n### Input:\ndiversity in tech\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a short slogan for the following business.\n\n### Input:\nGlobex Inc, a technical consulting services provider\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName some advantages of using AI in business\n\n### Response:\n', "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDefine 'artificial neural network'.\n\n### Response:\n", "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat does the phrase 'give-and-take' mean?\n\n### Response:\n", 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven the sentence, "The cafe serves some of the best coffee in town," generate a slogan for the cafe.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat is the average airline ticket price from Los Angeles to San Francisco?\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nSuggest a comic strip to express the given idea.\n\n### Input:\nThe joy of helping others.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate code to take user input and print it.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive me an example of a conflict resolution technique.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCome up with a rhymable phrase for "space race".\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat does the following proverb mean: "The grass is always greener on the other side"?\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a list of 3 potential strategies to reduce pollution in an urban area.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nDevelop the following ideas into a short story.\n\n### Input:\nA vampire cursed to never get satisfaction from drinking blood.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName a sport that requires good hand-eye coordination\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven two countries, find out one change they have made over time to better the environment.\n\n### Input:\nCountries: Germany and Chile\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nHow does exercise affect the body?\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nReformulate the following sentence to express an opinion.\n\n### Input:\nElectric cars are the future of smart transportation.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nFill in the blank "Computers can help us to _____ information more efficiently."\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven the sentence, output a sentence using a simliar syntax.\n\n### Input:\nThe dog barked loudly.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a recipe for an appetizer using only a few items from the pantry.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nSuggest two other activities to do with this given activity.\n\n### Input:\nVisit a museum\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompose a Haiku poem based on the following prompt: swimming.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName three cities in the United States with population over 1 million.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nFind the main concept expressed in the following sentence.\n\n### Input:\nHe acted in a way that went against all social norms.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nOutline the major points of the US Constitution.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nAnnotate the text by categorizing it into the five senses.\n\n### Input:\nI can’t wait to taste the warm, fruity muffin I just made.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nConvert the following Roman numeral to its equivalent value.\n\n### Input:\nXXVII\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompare a the customer service experience of two different restaurants.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a resumé for a data scientist.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nDesign a unique logo for a pet store using the provided name.\n\n### Input:\nPet Paradise\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDescribe a potential risk of using artificial intelligence.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nIdentify the subject and the action of the sentence.\n\n### Input:\nThe dog barked loudly.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven the input, explain how the design of the website integrates with the brand.\n\n### Input:\nThe website for the Cold Creek Cafe features muted green and brown colors, reflecting its rustic cabin-like aesthetic.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCategorize the following passage as either fiction or non-fiction.\n\n### Input:\nThe world of Agartha is a mysterious underground realm. It is a hidden land filled with a variety of bizarre creatures, both benevolent and malicious. Legends tell us that Agartha is a place of utopia, where knowledge and technology are plentiful and advanced beyond our understanding.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nPick the correct Spanish translation of “Hello”.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a headline for an article discussing the relationship between plant-based diets and reduced cardiovascular risks.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a play by play summary of the baseball game between the Chicago Cubs and the Los Angeles Dodgers\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a story using these three words: life, creation, surprise\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a chart showing the ratios of different planets to each other.\n\n### Input:\nThe radius of Jupiter is 69,911 km, and the radius of Earth is 6,371 km.\n\n### Response:\n', "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nExplain the differences between Darwin and Lamarck's theories of evolution\n\n### Response:\n", 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nMake a list of the key benefits to using artificial intelligence\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nIdentify a theme in William Wordsworth’s poem, “Daffodils”\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate an example of personification.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDesign a poster about endangered species.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a list of dates, provide the exact year for each.\n\n### Input:\nMay 31,\nApril 15,\nJune 8,\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDesign a mobile app interface created to track activity goals\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate an article discussing the pros and cons of eating organic food.\n\n### Response:\n', "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGive two pieces of advice based on the scenario.\n\n### Input:\nYou just graduated from college and don't know what to do next.\n\n### Response:\n", 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate an algorithm for converting an integer to a string.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate five example questions and answers related to psychology.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nConstruct a Big-O notation to explain the complexity of a certain algorithm\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhy do we need sleep?\n\n### Response:\n', "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a product, categorize four ways it can improve the user's life\n\n### Input:\nProduct: Alarm Clock\n\n### Response:\n", 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a process for reducing water waste\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a list of 5 tips for how to maintain work-life balance.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName one important consequence of deforestation.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nDescribe the character traits of Harry Potter\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a sample data set which contains 5 movie titles\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWhat were some of the major advances in technology during the Industrial Revolution?\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nClassify this email as an Inquiry or Suggestion\n\n### Input:\nHello,\n\nI am writing to suggest that the store stock more organic produce.\n\nSincerely,\n\n john\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nConvert the following times into 12 hours format.\n\n### Input:\n13:10\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nReferring to the given source link, explain the significance of the story in one sentence.\n\n### Input:\nhttps://www.nytimes.com/interactive/2020/05/25/magazine/white-house-covid-task-force.html\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive a detailed description of the differences between Marxism and Capitalism.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nWrite a web page about the birds of North America.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCome up with a name for a software that helps people identify possible investment opportunities.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nRewrite the given sentence so that it includes an example of one of the five senses.\n\n### Input:\nI felt a chill in the air.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nConstruct a Regex statement to extract the links from given text.\n\n### Input:\nThe following page has a list of useful links: www.example.com/links\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGenerate a story about a girl who discovers a magical item.\n\n### Response:\n', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nSummarize the following text into fewer than 100 words.\n\n### Input:\nPandas are native to China, and are the world’s most iconic endangered species. They live in temperate forest, mountain grasslands, and bamboo forests. Pandas feed mainly on bamboo but also eat other vegetation, insects and small animals. They face many threats from humans, from habitat loss to climate change.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nExplain what an embedding layer is and its purpose in Machine Learning.\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a funny story that uses metaphors\n\n### Response:\n', 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nChoose a title for a blog post about effective communication.\n\n### Response:\n', "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nGiven a product's features, write an introduction to the product.\n\n### Input:\nThe product is an app that helps people identify and manage their sleep habits. It can track sleep quality, circadian rhythm, and average sleep time.\n\n### Response:\n"]
We write to a sample_prompts.json
file for loading into different notebooks:
# write prompts to json
with open("./sample_prompts.json", "w") as f:
json.dump(sample_prompts, f, indent=4)
References¶
[1]: Stanford Alpaca: An Instruction-following LLaMA model. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto. 2023. https://github.com/tatsu-lab/stanford_alpaca.