Creating Image-Based Chatbot with ChatGPT API and Azure

Creating a chatbot has never been easier, thanks to the advancements in Natural Language Processing (NLP) and the availability of various APIs and platforms. In this article, we will explore how to create an image-based chatbot using the ChatGPT API and Azure.

Prerequisites

Before we begin, here are the prerequisites for creating an image-based chatbot using ChatGPT API and Azure:

1. Azure Account
2. Python 3.6 or above installed
3. ChatGPT API key
4. Azure Blob Storage account

Step 1: Setting up Azure Blob Storage

Azure Blob Storage is used to store the images that will be used by our chatbot. Follow these steps to create an Azure Blob Storage account:

1. Log in to the Azure portal.
2. Click on the “Create a resource” button and search for “Storage Account.”
3. Choose a unique name for the account, select the appropriate subscription, and resource group.
4. Select the “StorageV2 (general purpose v2)” performance tier and choose the appropriate replication option.
5. Leave the other settings to their default values and click on the “Review + create” button.
6. Review the settings and click on the “Create” button to create the storage account.

Once the storage account is created, navigate to it and create a container to store the images.

Step 2: Collecting and Storing Images

Before we can start building the chatbot, we need to collect and store images in the Azure Blob Storage container. For this tutorial, we will be using the CIFAR-10 dataset, which consists of 60,000 32×32 color images in 10 classes.

Follow these steps to store the CIFAR-10 images in Azure Blob Storage:

1. Download the CIFAR-10 dataset from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
2. Extract the dataset and install the azure-storage-blob package using pip.
3. Create a Python script to upload the images to Azure Blob Storage:

import os
import pickle
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient

# Replace with your own connection string and container name
CONNECTION_STRING = "<connection_string>"
CONTAINER_NAME = "<container_name>"

def upload_images():
blob_service_client = BlobServiceClient.from_connection_string(CONNECTION_STRING)
container_client = blob_service_client.get_container_client(CONTAINER_NAME)
for i in range(1, 6):
with open(f"../cifar-10-batches-py/data_batch_{i}", "rb") as f:
data = pickle.load(f, encoding="bytes")
for j in range(len(data[b"data"])):
blob_client = container_client.get_blob_client(f"{i}_{j}.png")
blob_client.upload_blob(data[b"data"][j])

1. Replace the CONNECTION_STRING and CONTAINER_NAME variables with your own values.
2. Run the upload_images() function to upload the images to Azure Blob Storage.

Step 3: Setting up the ChatGPT API

The ChatGPT API is a pre-trained language model that can be used to generate text responses to user inputs. To use the ChatGPT API, you will need an API key, which can be obtained by signing up for a free account at https://app.chatgpt.com/signup.

Once you have obtained the API key, you can use the following code to interact with the ChatGPT API:

import openai
openai.api_key = "<your_api_key>"

def generate_response(prompt):
    completions = openai.Completion.create(
       

Now that you have obtained the API key, you can start interacting with the ChatGPT API. The first step is to install the openai Python module, which provides a simple interface to communicate with the OpenAI API services. You can install it using pip by running the following command:
pip install openai

Once you have installed the openai module, you can use the following code to interact with the ChatGPT API:

import openai_secret_manager
import openai

# Load API key from the environment variable
assert "openai" in openai_secret_manager.get_services()
secrets = openai_secret_manager.get_secret("openai")

# Initialize the OpenAI API client
openai.api_key = secrets["api_key"]

# Set the engine and parameters for the API request
engine = "text-davinci-002"
prompt = "Hello, how are you?"

# Send the request to the API and retrieve the response
response = openai.Completion.create(
    engine=engine,
    prompt=prompt,
    max_tokens=1024,
    n=1,
    stop=None,
    temperature=0.7,
)

# Print the generated text
print(response.choices[0].text.strip())

In this code, we first import the openai_secret_manager and openai modules. We then load the API key from the environment variable using the openai_secret_manager.get_secret function, which retrieves the secret from the Secrets Manager. We initialize the OpenAI API client by setting the api_key attribute of the openai module to the value of the API key. We then set the engine and prompt parameters for the API request. The engine parameter specifies which GPT model to use for generating text, and the prompt parameter is the input prompt that we want the model to complete. In this example, we are asking the model to complete the prompt “Hello, how are you?”

We then send the API request to the OpenAI API using the openai.Completion.create function, which takes the engine, prompt, and other parameters as input. The function returns a Completion object, which contains a list of Choice objects. Each Choice object represents a possible completion of the prompt, with its own text, score, and other attributes. We retrieve the first Choice object from the list using response.choices[0] and print its text attribute, which contains the generated text.

Building the Image-Based Chatbot

With the ChatGPT API set up, we can now start building the image-based chatbot. The first step is to collect some image data and create a dataset for training the chatbot. In this example, we will be creating a chatbot that can generate responses to images of cats and dogs. We will use the Microsoft Azure Cognitive Services Computer Vision API to analyze the images and extract information about the animals in them.

Collecting Image Data

To collect the image data, we will be using the Bing Image Search API, which is part of the Microsoft Azure Cognitive Services suite. The API allows us to search for images using keywords and retrieve their URLs and other metadata.

To use the Bing Image Search API, we first need to create a Bing Search resource in our Azure subscription. To do this, follow these steps:

1. Go to the Azure portal and log in with your account.
2. Click on “Keys and Endpoint” to obtain the endpoint URL for the API.

Setting up Azure Functions

Now that we have obtained the necessary credentials, we can start setting up the Azure Functions. We will be creating two functions – one for handling user input and generating a response from the ChatGPT API, and another for sending the response to the user via the messaging platform.

Creating the HTTP Trigger Function

The first step is to create an HTTP trigger function, which will be responsible for receiving user input and generating a response. To create a new function, follow the steps below:

1. In the Azure portal, navigate to your Function App and click on the “Functions” tab.
2. Click on the “New Function” button and select “HTTP trigger” as the template.
3. Give your function a name and configure the input and output bindings as shown below:

Input:

Output:

4. Click on “Create” to create the function.

Adding Code to the Function

Now that we have created the function, we can add the code to handle user input and generate a response using the ChatGPT API. Here is an example code snippet that you can use as a starting point:

import os
import requests

endpoint = os.environ['ChatGPT_endpoint']
api_key = os.environ['ChatGPT_api_key']

def main(req: func.HttpRequest) -> func.HttpResponse:
    try:
        req_body = req.get_json()
        message = req_body.get('message')

        response = get_response_from_chatgpt(message)

        return func.HttpResponse(response)

    except Exception as e:
        return func.HttpResponse(f"Error: {str(e)}")

def get_response_from_chatgpt(message):
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {api_key}'
    }

    data = {
        'inputs': message,
        'parameters': {
            'temperature': 0.5,
            'max_tokens': 50
        }
    }

    response = requests.post(endpoint, headers=headers, json=data)

    return response.json()['generated_text']

The get_response_from_chatgpt function sends a POST request to the ChatGPT API with the user’s message as the input. The temperature and max_tokens parameters control the creativity and length of the generated response. The function returns the generated response as a string.

The main function is responsible for handling the HTTP request and calling the get_response_from_chatgpt function. It returns the generated response as an HTTP response object.

Creating the Message Send Function

The second function we need to create is responsible for sending the response generated by the ChatGPT API to the user via the messaging platform. This function will be triggered by a message event from the messaging platform.

To create a new function, follow the steps below:

1. In the Azure portal, navigate to your Function App and click on the “Functions” tab.
2. Click on the “New Function” button and select “Event Grid trigger” as the template.
3. Give your function a name and configure the input binding as shown below:

Input:

4. Click on “Create” to create the function.

Adding Code:

Now, we can add the code to create our image-based chatbot using ChatGPT API and Azure. First, we need to import the required libraries:

def image_to_base64(image_path):
    with open(image_path, 'rb') as f:
        img_bytes = f.read()
    return base64.b64encode(img_bytes).decode('utf-8')

Next, we need to define a function to convert the image into base64 encoded string, as required by the ChatGPT API:

def image_to_base64(image_path):
    with open(image_path, 'rb') as f:
        img_bytes = f.read()
    return base64.b64encode(img_bytes).decode('utf-8')

This function takes the image path as input, reads the image bytes, encodes them in base64, and returns the base64 encoded string.

Next, we need to define a function to send the image to the ChatGPT API and get the response:

def chatbot(image_path, model):
    url = 'https://api.openai.com/v1/images/gpt3-chatbot'
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {API_KEY}'
    }
    data = {
        'image': image_to_base64(image_path),
        'model': model,
        'prompt': ''
    }
    response = requests.post(url, headers=headers, data=json.dumps(data))
    return json.loads(response.text)

This function takes the image path, the model (which can be ‘davinci’, ‘curie’, or ‘babbage’), and the prompt as input. It sends a POST request to the ChatGPT API with the base64 encoded image, the model, and the prompt, and returns the response as a JSON object.

Finally, we can define a function to process the response and extract the chatbot’s answer:

def get_chatbot_answer(response):
    chatbot_output = response['choices'][0]['text']
    return chatbot_output

This function takes the response from the ChatGPT API as input, extracts the chatbot’s answer from the JSON object, and returns it as a string.

Putting it all together, the complete code for our image-based chatbot using ChatGPT API and Azure looks like this:

import requests
import json
import base64
import io
from PIL import Image

API_KEY = 'YOUR_API_KEY'

def image_to_base64(image_path):
    with open(image_path, 'rb') as f:
        img_bytes = f.read()
    return base64.b64encode(img_bytes).decode('utf-8')

def chatbot(image_path, model):
    url = 'https://api.openai.com/v1/images/gpt3-chatbot'
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {API_KEY}'
    }
    data = {
        'image': image_to_base64(image_path),
        'model': model,
        'prompt': ''
    }
    response = requests.post(url, headers=headers, data=json.dumps(data))
    return json.loads(response.text)

def get_chatbot_answer(response):
    chatbot_output = response['choices'][0]['text']
    return chatbot_output

image_path = 'YOUR_IMAGE_PATH'
model = 'davinci'

response = chatbot(image_path, model)
chatbot_answer = get_chatbot_answer(response)

print(chatbot_answer)

Note that you need to replace ‘YOUR_API_KEY’ and ‘YOUR_IMAGE_PATH’ with your actual API key and image path, respectively. Also, make sure to choose the appropriate model for your application. In this example, we used the ‘davinci’ model, which is the most powerful and capable one

After deciding on the chatbot engine, the next step is to integrate it with a chat interface. For this tutorial, we will be using the Flask framework, which is a lightweight Python web framework that is perfect for building web applications.

Setting up Flask

First, we need to install Flask using pip, which is the package installer for Python. Open up a terminal and run the following command:

pip install Flask

Once Flask is installed, we can create a new Flask application by creating a new Python file and adding the following code:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/')
def home():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()

This code sets up a basic Flask application that has a single route / that returns the string Hello World!. To run the application, save the file as app.py and run the following command in the terminal:

pip install requests

import requests
from flask import Flask, request, jsonify

app = Flask(__name__)

CHATGPT_API_KEY = 'your-api-key-here'

@app.route('/')
def home():
    return 'Hello World!'

@app.route('/chat', methods=['POST'])
def chat():
    message = request.json['message']
    response = requests.post('https://api.openai.com/v1/engine/davinci-codex/completions', json={
        'prompt': f'chatbot: {message}\nuser:',
        'max_tokens': 50,
        'temperature': 0.7,
    }, headers={
        'Authorization': f'Bearer {CHATGPT_API_KEY}',
        'Content-Type': 'application/json',
    }).json()

    return jsonify({'response': response['choices'][0]['text']})

if __name__ == '__main__':
    app.run()

This code adds a new route /chat that accepts POST requests with a JSON payload that contains a message key. The code then sends a request to the ChatGPT API using the requests module, passing in the user’s message as the prompt. The response from the API is then returned as a JSON object containing a response key that contains the chatbot’s response.

Deploying to Azure

Now that we have our Flask application integrated with the ChatGPT API, we can deploy it to the cloud using Azure. Azure is a cloud computing platform that provides a wide range of services, including web app hosting.

To deploy our Flask application to Azure, we will use the Azure CLI. First, we need to install the Azure CLI by following the instructions here.

Once the Azure CLI is installed, we can create a new web app by running the following command in the terminal:

{
    "input_text": "Hello, Chatbot!"
}

Improving Chatbot Performance

One way to improve your chatbot’s performance is to fine-tune the ChatGPT model on a domain-specific dataset. This involves training the model on a dataset that is specific to the topic of your chatbot. For example, if your chatbot is designed to help users find recipes, you could fine-tune the model on a dataset of recipes. Fine-tuning allows the model to learn more about the specific domain and produce better responses.

Another way to improve chatbot performance is to incorporate user feedback. By analyzing user feedback, you can identify areas where the chatbot is struggling and make improvements accordingly. This could involve updating the chatbot’s knowledge base or adjusting the way it handles certain types of requests.

Adding Functionality

In addition to improving performance, you can also add new functionality to your chatbot. For example, you could add support for voice interactions or integrate your chatbot with third-party services such as payment gateways or weather APIs.

To add functionality to your chatbot, you will need to write additional code that interacts with the chatbot API. This code can be hosted on Azure or another cloud provider, depending on your preference.

Last Updated on April 29, 2023 by admin