Become a Prompt Engineer: Develop the Skill of Shot Prompting

Welcome back to already the fifth part of our Prompt Engineering series! At Tilburg.ai, we believe that the best way to learn is through examples, and here’s a little secret: language models are great at that too. Some of you might already know what we’re inferring about shot prompting.

In this part, we will dive into zero/one or few-shot prompting techniques and their applications.

By the end of this part, you will:

• Obtained the skill of using zero-shot, one-shot, and few-shot prompting techniques and their applications.

Let’s give it your best shot!

Shot Prompting

As we implicitly mentioned in the introduction, the name of this technique depends on the number of examples we provide. When no examples are given, we use zero-shot prompting. This is the technique we have used in our series so far. Specifying how the answer should look and structuring the output format is similar to the goal of shot prompting but is called template prompting. If we use one example, we call it one-shot prompting, and with more than one example, we use few-shot prompting. Let’s explore each of these shot-prompting techniques in more detail.

Zero-shot Prompting

In zero-shot prompting, we give a prompt to the model without explicit examples. The model is expected to generate a response based on its existing knowledge. This technique is ideal for quick and uncomplicated tasks. For example, we use zero-shot prompting when we ask the model to define prompt engineering without providing examples to guide the output.

Zero-shot Prompt: "Write out helpful tips for a comment to a student’s report?"

One-shot Prompting

One-shot prompting is a strategy where the model is given a single task-specific example before being presented with the actual prompt. Instead of asking the model to perform a task it hasn’t been trained on at all, you provide the model with a single example of the task you want it to perform. The model then uses this single example to extrapolate and generate text accordingly. This approach differs from zero-shot prompting in that it provides a sample or ‘shot‘ as a reference for the model to understand the expected structure or context of the response. This technique is ideal when we want the model’s output to follow a specific and consistent format or style.

Write me a comment for a student’s report using their name, final grade in the course, strength and an area for improvement that I will provide you delimited by backticks. The comment should be complimentary, professional, and look ahead to the next year. The comment should be addressed to the student and parents. 

Here is an example of what it should look like
```Input: Student 1: Byron, final grade: 92%, strength: great attention to detail, area for improvement: misses class frequently
Report card comment for Student 1:```    

Output: Congratulations, Byron, on your impressive final grade of 92% this year! Your exceptional attention to detail has been a driving force behind your success. As we look ahead to the next year, let's focus on improving attendance to ensure you have every opportunity to excel and continue your remarkable academic journey. Keep up the great work!

```Input: Student 2: Matthijs, final grade 98%, strength: great attention to detail, area for improvement: too eager, less preparation time would fulfil its purpose``

And the AI’s response:

Report card comment for Student 2:
Congratulations, Matthijs, on achieving an outstanding final grade of 98% this year! Your meticulous attention to detail has truly set you apart. Moving forward, let's work on finding a balance and allowing for adequate preparation time. Remember, a little less eagerness can lead to even greater success in the future. Keep up the fantastic work!

Few-shot Prompting

Finally, in few-shot prompting, the model is given multiple task-specific examples before presenting the actual prompt. Unlike zero-shot and one-shot prompting, this method provides several samples or ‘shots’ that act as references for the model on the expected structure or context of the response. The idea is to enable the model to generalize better over the task from these few examples. Hence, the model infers from these multiple examples to perform the actual task.

Using the Chatbot Function inside the API

We can use this few-shot prompting technique within the chat_completion.create function of the API by providing examples like previous conversations in the message list. In this case, you’re sending the model multiple messages. Rather than just showing it how it should respond, you have gone ahead and actually created responses. All these messages will be sent at once, giving the model an even better understanding of how it should respond. Here’s how you can use this technique using a classification example:

1. Start with providing Examples within the Message List: In the chat_completion.create function, include previous conversations as examples. Each example consists of a user message and an assistant message. For sentiment analysis, for instance:

response = client.chat.completions.create(
  model="gpt-4o",
  messages = [
    {"role": "user", "content": "I love this product!"},
    {"role": "assistant", "content": "positive"},
    {"role": "user", "content": "This is the worst service ever."},
    {"role": "assistant", "content": "negative"}
]) 

In each example, the user message represents the text we want to classify (the prompt from the user). The assistant message represents the classification response (the model’s output). This helps the model learn the expected output format from the given examples.

Next to classify new text, send it as a user message without an accompanying assistant message. This means that we want the model to provide the classification for this new text.

response = client.chat.completions.create(
  model="gpt-4o",
  messages = [
    {"role": "user", "content": "I love this product!"},
    {"role": "assistant", "content": "positive"},
    {"role": "user", "content": "This is the worst service ever."},
    {"role": "assistant", "content": "negative"},
    {"role": "user", "content": "The experience was quite pleasant."}
]) 

Models Response: positive

Observe that the model replies using the word “positive,” exactly as the assistant role in our examples specifies. This illustrates the efficacy of few-shot prompting: we direct the model to generate accurate and consistent responses by giving succinct examples. This approach improves the model’s performance in classification tasks and guarantees that the response format is followed.

Practical Example Case Few-shot Prompting Chatbot

Let’s circle back to the scoring card example. In the previous one-shot example, we provided a prompt around it. However, in this few-shot prompt example, we are just giving examples and letting the model learn from them. We are not specifying a prompt here. Note that for this specific example, it would be wise to include both positive and negative feedback on scoring cards in our example, as the model needs to understand what makes a “good” student and a “less performing” student.

response = client.chat.completions.create(
  model = "gpt-4o",
  
  # Provide the examples as previous conversations
  
  ## Example 1 ~ Positive
  messages = [{"role": "user", 
              "content": "Input: Student 1: Byron, final grade: 92%, strength: great attention to detail, area for improvement:  misses class frequently"},
              {"role": "assistant", 
               "content": "Congratulations, Byron, on your impressive final grade of 92% this year! Your exceptional attention to detail has been a driving force behind your success. As we look ahead to the next year, let's focus on improving attendance to ensure you have every opportunity to excel and continue your remarkable academic journey. Keep up the great work!
"},

## Example 2 ~ Negative
              {"role": "user", 
              "content": "Student 2: Matthijs, final grade 58%, strenght: Thinking outside the box, area for improvement: A thicker basis knowledge layer, by studying some additional hours"},
              {"role": "assistant", 
              "content": "I am sorry to disappoint, by stating you have failed the exam this year, Matthijs. Your strength in thinking outside the box has been evident throughout the term. In order to improve your final grade next year, let's focus on building a stronger foundation of knowledge by dedicating some additional hours to studying. With this added commitment, I have no doubt you will achieve great success in the upcoming year."},

# Provide another the student for the model to classify
              {"role": "user", "content": "Student 3: Rolf, final grade 78%, strength: Passionate about the exam material, area for improvement: More focus within the class"}
             ],
)

Models Response: Well done, Rolf, on achieving a final grade of 78%! Your passion for the exam material is truly admirable and has undoubtedly played a crucial role in your success. Going forward, let's aim to enhance your focus during class sessions. By maintaining your enthusiasm and concentrating more during lessons, you can reach even greater heights. Keep up the excellent effort!

Download the full Python file here:

Considerations of Shot Prompting 

Using shot prompting offers many possibilities, but it is not perfect. As with everything related to prompts, garbage in equals garbage out. Thus, the biggest limitation of shot-prompting is the dependence on the quality and variety of the given examples.

Does this mean you need to provide an endless list of examples? Certainly not. Sometimes the examples can even worsen performance or steer the model in the wrong direction. There is also the risk of overfitting, where the model is unable to generalize the examples and creates outputs that are too similar to the existing examples.

Additionally, there are some biases to watch out for:

• Majority Label Bias: The model tends to favor answers that occur more frequently in the prompt. For example, in our scoreboard task, if our prompt contains more positive than negative examples, the model may be inclined to predict a positive sentiment. The extent of this bias varies depending on the model.
• Recency Bias: LLMs tend to favor the most recent information they receive. Looking back at our scoreboard task, if the last few examples in a prompt are negative, the model may be more inclined to predict a negative sentiment.
• Restricted to the Provided Examples: We have only provided the model with “positive” and “negative” examples for our scoring card example. This implies that the model cannot return “neutral” feedback, for instance.

Conclusion

This was it again, lesson 5 of our series. In which we have flown through the world of shot prompting. Fortunately, it’s not a one-shot deal here; you can constantly refine your prompts. Either way, you have the tools you require for performing shot prompting, and we have secretly already made a start on creating a chatbot!