In today’s discussions about AI capabilities, there’s a significant emphasis on improving outcomes through refined models, achieved by training them with specific datasets to respond effectively to prompts.
Experts stress the importance of accurate datasets in significantly reducing AI errors.
Until recently, my enthusiasm was primarily centered on how these AI capabilities enhance Intelligent Document Processing (IDP) solutions, streamlining document processing and saving time for professional services teams to address errors. However, what if we begin leveraging IDP solutions to enhance AI outcomes? This entails orchestrating prompts, improving the accuracy of input sources, and minimizing manual labeling tasks. Let’s explore this potential synergy between AI and IDP.
In a scenario where you aim to develop an AI-based question-and-answer bot or an end-to-end conversational assistant using Conversational Language Understanding (CLU), the quality of the input source becomes paramount. This input source generates indexes that facilitate responses to all received prompts or utterances. The accuracy of these answers directly correlates with the quality of the input source, ultimately determining the level of satisfaction your customers will experience.
What is CLU?
Conversational language understanding (CLU) enables users to build custom natural language understanding models to predict the overall intention of an incoming utterance and extract important information from it. CLU only provides the intelligence to understand the input text for the client application and doesn’t perform any actions.
source: https://learn.microsoft.com/en-us/azure/ai-services/language-service/conversational-language-understanding/overview
The same happens in a use case involving Retrieval Augmented Generation (RAG) where to enhance the quality of prompt responses in generative AI models, the input source’s quality is equally crucial.
What is RAG?
Retrieval Augmented Generation (RAG) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. Adding an information retrieval system gives you control over grounding data used by an LLM when it formulates a response.
source: https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview
Regarding the vast array of document assets that a company may have, these serve as a valuable input source for AI solutions aimed at addressing questions, prompts, and utterances within the context of the company. Utilizing this internal content allows for tailored responses, rather than relying on a generic approach. However, achieving accurate results hinges on inputting this content in an appropriate manner, and this is where Intelligent Document Processing (IDP) solutions prove to be very useful.
To illustrate how this can work, let’s consider an imaginary scenario where an IDP platform is configured to gather documents from various locations within the company’s digital infrastructure. These documents are then processed through standard IDP features such as image conversion, OCR/ICR, classification, and extraction. The results of these activities are utilized to determine the appropriate path for each document to ensure accurate indexing.
For instance, some documents may undergo analysis by an Azure AI Search service, while others are processed by a generic large language model (LLM), and yet others by a specific small language model (SLM), and so forth. The objective here is to gather as much information as possible about each document, creating a knowledge store that can be leveraged by your AI solution.
What is Azure AI Search?
Azure AI Search, an AI-powered information retrieval platform, helps developers build rich search experiences and generative AI apps that combine large language models with enterprise data.
source: https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search
In this scenario, the IDP platform not only collects documents and processes them through standard built-in features but also orchestrates the utilization of multiple AI models to enhance the quality of results obtained. Additionally, the designed process can incorporate data rules and include humans in the loop (HITL) to ensure data accuracy before indexing in a knowledge store to be consumed by the AI solution.
Of course, implementing the proposed scenarios entails a considerable amount of activity on both the IDP and AI fronts. However, what’s truly remarkable is that these endeavors are not only feasible but also supported by numerous existent platforms that can facilitate their execution in an appropriate manner.
“Building advanced AI is like launching a rocket. The first challenge is to maximize acceleration, but once it starts picking up speed, you also need to focus on steering.” Jaan Tallinn
In my personal opinion, I perceive only advantages in embracing these emerging technologies. Naturally, it’s crucial to comprehend their workings, prioritize Responsible AI practices, and conduct extensive testing before deployment.
What is Responsible AI?
Responsible Artificial Intelligence (Responsible AI) is an approach to developing, assessing, and deploying AI systems in a safe, trustworthy, and ethical way.
source: https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai?view=azureml-api-2
Having a well-structured project in the hands of IDP and AI experts is the way to go forward. Use IDP orchestration and process capabilities to ensure that each document you will include in the knowledge store is properly indexed and also ensure that each AI service has access to only the information it needs, reducing the amount of unnecessary information it will also reduce hallucinations in responses.
AI journey is only in the beginning stages and there is no turning back.
This article is rooted in personal experience gathered over the years, encompassing perspectives as both an AI enthusiast and an automation expert specializing in financial processes.
Acronyms used in the article:
- AI – Artificial Intelligence
- IDP – Intelligent Document Processing
- CLU – Conversational Language Understanding
- OCR – Optical Character Recognition
- ICR – Intelligent Character Recognition
- LLM – Large Language Model
- SLM – Small Language Model
- RAG – Retrieval Augmented Generation
- HITL – Human in the Loop