Create Custom Chatbot with Azure OpenAI and Azure AI Search

Introduction

Nowadays, we hear a lot about generative AI (GenAI) and large language models (LLMs) from various providers for different use cases. Among them, OpenAI models such as GPT-4o and GPT-3.5 Turbo are some of the most widely used. 

In this article, I will show you how to create a personalized chatbot that integrates with your own data source using OpenAI models in Azure Cloud. 

Topics 

1. Overview of Azure AI services 
2. Introduction to Azure AI Search and Azure OpenAI
3. The retrieval-augmented generation (RAG) solution and its advantages
4. Architecture diagram
5. Create Azure resources and deploy the GPT-4o model with Terraform 
6. Adding knowledge base data to Azure AI Search for indexing 
7. Integrating external data sources with the LLM model 
8. Accessing the prompt and querying an external data source 
9. Deploying the OpenAI GPT model on Azure App Service with a custom domain

1. Overview of Azure AI services: 

Azure offers a wide range of AI services that developers can use to build intelligent applications. These services enable you to add capabilities like vision, speech, language, and decision-making to your apps through APIs and SDKs.

• Vision: Provides image and video analysis services
• Speech: Includes capabilities for speech recognition, translation and synthesis
• Language: Supports natural language processing (NLP) for understanding and analyzing text
• Decision: Offers AI services for anomaly detection and decision-making

2. Introduction to Azure AI Search and Azure OpenAI

Azure AI Search is a powerful search platform that enables you to create a rich search experience and combine it with large language models (LLMs) to build generative AI applications using your enterprise data. Azure OpenAI provides access to models like o1-preview, o1-mini, and GPT-4o mini/turbo/audio through REST APIs, enabling content generation, summarization, semantic search, code generation, and more.

Available models in Azure OpenAI Studio include:

• GPT-4o: Improves upon its predecessor, GPT-3.5
• Text-embedding-3-large/small: Converts text into numerical vectors for vector search
• DALL-E: Generates original images from natural language prompts
• Whisper: Transcribes and translates speech to text
• Text to Speech: Synthesizes speech from text

3. What is RAG (Retrieval-Augmented Generation)? 

RAG is an AI solution that combines search-based retrieval (like Azure AI Search) with generative models (like OpenAI) to create more accurate and context-aware responses. 

Imagine a customer support chatbot that can search your company's knowledge base through Azure AI Search and generate a response using Azure OpenAI. This helps create precise answers that are both accurate and contextually rich. 

Advantages of RAG: 

• Accurate responses: OpenAI generates up-to-date answers using data retrieved by Azure AI Search.
• Contextual information: Azure AI Search retrieves the most relevant documents, helping OpenAI generate responses tailored to the user's query.
• Reduced hallucinations: RAG works with a base of knowledge instead of relying solely on training data, which reduces hallucinations.
• Enhanced personalization: RAG models offer personalized responses based on user preferences, past interactions, and historical data.
• Improved user experience: By merging search-based retrieval with generative AI, RAG delivers comprehensive answers, preventing users from sifting through multiple sources.

4. Architecture diagram
The architecture diagram illustrates a workflow involving Azure Cognitive Search and Azure OpenAI.

5. Creating Azure resources and deploying the GPT-4o model with Terraform

You need to create the following resources using Terraform or from the Azure portal:

• Create a resource group: Organize your resources in Azure
• Create an Azure OpenAI service: Set up GPT-4o or GPT-3.5 models
• Create an Azure AI Search service: Index and retrieve documents
• Create a storage account and container: For uploading documents

I created the Terraform code in my GitHub repository to deploy these resources; you can clone or fork it from here.

Go to the Azure Portal to see the resources deployed under Resource Group “RG-openAI-demo" as below screenshot: 

6. Adding knowledge base data to Azure AI Search for indexing

Go to the Azure AI Search resource. 

Click on the "Import Data" option and select either a sample dataset (I used the sample dataset “hotels-sample” in this blog) or your data.

• Import your own data from Azure Blob Storage. 

 The following data files are uploaded in the Azure Blob Storage container named "democontainer" as knowledge base. 

• Follow the prompts to configure and create an index and an indexer.

• Go to Azure AI Search resource > Search Management > Indexers to verify the status. 

7. Integrating external data sources with the LLM model:  

• Now open Azure OpenAI resource “openai-rxt-demo" and click on Azure OpenAI Studio to access the Azure OpenAI playground. 

• In Azure OpenAI studio, click on “Deployments” to access the LLM models. 

• Choose the GPT model deployed GPT-4o and navigate to the Playground. 
• In the Chat Playground, you can start interacting with the model. 

• Configure important settings such as the system message (instructions for the model) and parameters (e.g., temperature, token limit) to customize how the chatbot responds.

• To integrate external data, select "Add Your Data”.

• Choose Azure AI Search as the data source. 

• Once you submit the request, the data source will be configured as Azure AI Search.

• You will need to request a quota increase for the GPT-4o model. There is no need to request a rate limit increase for GA versions. Such as GPT-3.5/turbo. Once Microsoft approves your request, you will be able to use GPT-4o with your own data. 

• If you would like to further increase the default rate limit, go here.

8. Accessing the prompt and querying the external data source: 

• In the Chat Playground, you can now ask questions that the model will answer based on the external data source.  
• For example, I asked the chatbot to "find me the best hotels in New York and visiting places" using the indexed knowledge base.

• I also asked a question outside the scope of the data source, but the model was unable to provide an answer, as it can only retrieve information from the specified dataset.

• When I unchecked "Limit response to your data content" in Advanced settings to allow the model to respond to queries beyond the provided data source, like I asked “tell me about the Rackspace Technology”, it responded from Internet.

9. Deploying the OpenAI GPT model on Azure App Service with a custom domain

You can deploy the model to a web app for custom domain access. To do this, follow the steps shown in the screenshot to link it to Azure App Service.

 Note: Enabling chat history will create a Cosmos DB database to store your chat records, which may increase your costs.

 After deployment, you can access the chatbot via a custom URL.

I hope this article helps you create a custom chatbot using Azure OpenAI and Azure AI Search. Feel free to clone the Terraform code and share any feedback or suggestions.