When ChatGPT first came out the most common complaint was: “It is trained on data from 2021!” In response to this concern, ChatGPT introduced plugins and the browser plugin was aimed directly at this criticism. The browser plugin integrated live web searches into ChatGPT. It did this by running web searches with Bing and putting the results into the LLM prompt as a work-around for not having access to real-time information.
Very quickly, we realized the possibility of taking knowledge, data and content from our own repositories and mimicking what this plugin was doing. But there was a snag: the context size was capped at about 50 pages of text. Since context builds with each interaction, we practically had much less than 50 pages. Not only was the context size a hurdle, but the costs of managing a huge context could skyrocket. I mean, no one’s going to upload the entire Library of Congress into a prompt—not when it could run you over $40 million!
Enter stage right: RAG (you can dive deeper into RAG on our blog). RAG tucks away knowledge into a vector database and hunts through it to find content that may help answer your question. The bits of information it pulls are way less than the total knowledge stored, making it a cost-effective option. It sends these snippets to a large language model (LLM) to generate an answer. RAG does introduce a layer (search) which may not perform as well as pure LLMs do, but it remains a promising method and even though it’s in its infancy, you are sure to see it offered by the service desk solutions.
Integrating RAG into the enterprise realm is a natural progression of the browser plugin. By tapping into an organization’s wealth of resources, like policy documents or knowledge base articles, RAG can tailor its responses to reflect your company or school guidelines. However, the leap into the enterprise space isn’t without its challenges. High standards for data security and a nuanced understanding of different roles are paramount—after all, enterprise solutions aren’t quite like their consumer counterparts.
Consider the scenario where different unions offer distinct benefits and perks. It’s crucial that responses from a bot are precisely aligned with the specific union’s contract. Additionally, there are layers of information accessibility, such as certain details that should only be visible to managers. These intricacies certainly add a layer of complexity to implementing enterprise RAG effectively.
For this blog post, we conducted a small experiment using OpenAI’s assistants, which feature built-in RAG capabilities. We uploaded two documents into the RAG’s database: one was a manager’s guide to interviewing candidates, and the other, a guide for employees or students preparing for interviews. These documents present a unique challenge because they cover the same topic but cater to different audiences.
We began with the question, “How do you prepare for an interview?”
The RAG model had content addressing both the interviewer’s and the interviewee’s perspectives. We anticipated a response that would reflect both viewpoints, but unfortunately, that wasn’t the case:
The response exclusively addressed interviewees. Given that language models operate based on probabilities, it’s more likely for someone to be an interviewee than an interviewer. However, it was surprising and a bit concerning that the model completely overlooked one of the documents and perspectives.
Next, we asked, “What tips do you have on interviews?” Expecting an interviewee-focused reply, the result surprised us again:
Our answer here is clearly for interviewers! The bot knows about both documents and has indexed their content, but why and how it chooses one path over another is a mystery (as with most things LLM).
Thinking it might help to specify our role, we told the bot we were managers to see if it would alter its responses:
These instructions had no effect on the output of the previous questions. Ok, as a final test, we will put information about the user’s role directly in the question itself.
The results were the oddest yet blending together content for both interviewers and interviewees.
This exercise shows that any deployment of an LLM with RAG in an enterprise setting needs to be carefully thought out, planned and tested!
Ida is designed with an enterprise-first mindset. From the outset, we’ve prioritized security and system integration, making these features the core of Ida’s DNA. Ida excels in using RAG to quickly enhance capabilities and deepen understanding. Equally, it recognizes when personalization and role sensitivity are crucial, adeptly managing both simultaneously through Ida’s Multi-level AI system.
Our strategy allows client administrators to tailor Ida for personalized and predictable results in situations that demand specific attention, while still offering more flexible, ChatGPT-style interactions when possible. With Ida, you’re not confined to a single approach; instead, you can customize your chatbot experience to fit your needs.
This flexibility is achievable because Ida operates on dedicated customer tenancies and utilizes bespoke AI models, ensuring your data and insights remain secure and protected.
If you are working on RAG or similar challenges and want to have a chat about your use case, contact us below.
Gideon Taylor is thrilled to announce the latest release of our enterprise-grade chatbot, Ida 24.01, marking a significant leap forward in Ida’s AI-driven communication. This update introduces a suite of enhancements designed to refine user experience, streamline administrative tasks, and, most importantly, unveil our groundbreaking T2 Ida algorithm. Let’s dive into the highlights and features of this exciting new release.
At the core of Ida 24.01 is the beta release of the T2 Ida algorithm, a game-changing enhancement that sets a new standard for AI chatbot performance. This algorithm is not just an upgrade; it’s a foundation for the future, offering:
Ida 24.01 is packed with features and improvements that build upon our commitment to delivering exceptional AI chatbot solutions. Here’s what you can expect in this latest release:
The release of Ida 24.01 is a step in Gideon Taylor’s vision for giving your enterprise a voice with innovation and excellence in AI chatbot technology. With the introduction of the T2 algorithm and a host of enhancements, we’re setting new benchmarks for what enterprises can expect from their AI interactions. We’re excited for our clients to experience these advancements and look forward to their feedback as we continue to refine and evolve Ida’s capabilities.
The latest release of Ida, version 23.04, brings a suite of enhancements and new features to our cutting-edge digital assistant platform and is now generally available. This release focuses on improving scale with Generative AI and expanding the utility of Ida in diverse operational environments.
Here’s a summarized list of all the features, enhancements, and fixes included in the Ida 23.04 release:
This release reaffirms our commitment to leveraging Generative AI to deliver a sophisticated and user-friendly digital assistant that evolves continuously to meet the dynamic needs of our clients. Stay tuned for more updates and improvements as we continue to push the boundaries of digital assistance technology.
Part II of our series on Generative AI (“GenAI”) expands on Part I: Top 7 Generative AI Misconceptions. In the last year, GenAI has become a focal point of both fascination and skepticism. To fully appreciate its impact, it’s crucial to understand how it diverges from Traditional AI. This post delves into those distinctions, addressing seven common inquiries from clients implementing conversational AI—ranging from chatbots to sophisticated digital assistants.
Armed with these definitions, let’s explore how GenAI stands in contrast to its predecessor, Traditional AI, through the lens of practical applications and client-driven concerns.
Traditional AI is a trained model for a specific task or domain thereby sometimes called “Narrow AI”. Every output is known and approved in Traditional AI so the AI will not answer questions out of its domain. Generative AI (GenAI) is different in that it is not trained by the customer, rather it is trained by a vendor on a public data set and is meant to answer broadly a much larger set of questions. While Traditional AI has curated outcomes, GenAI uses word sequence probabilities to generate answers. Generative is a bit misleading in that it doesn’t create completely new outputs, it simply takes previous examples and shuffles them around.
| Traditional AI | Generative AI | |
| Implementation Effort | ✅ | |
| Lower Software Cost | ✅ | |
| Analytics | ✅ | |
| Control | ✅ | |
| Breadth & Scale | ✅ | |
| Ongoing Effort | ✅ | ✅ |
| Predictability | ✅ | |
| Security | ✅ | |
| Run-time Speed | ✅ |
Ultimately, a hybrid approach may be the most pragmatic path forward, combining the reliability of Traditional AI for core, sensitive tasks with the adaptive prowess of Generative AI for more general inquiries.
There is great curiosity over which AI to use in which situation. Since our readership is made up of enterprise use cases, we will focus on those and specifically around user support chatbots in enterprises, higher education institutions and public sector organizations. This post aims to compare Traditional AI vs. Generative AI using the key questions our clients are asking us.
Understanding the data requirements for AI systems is key to implementing effective chatbots and digital assistants. Here’s a breakdown of the types of data needed for Traditional AI versus Gen AI:
Training Data
Traditional AI operates on a foundation of examples, where inputs (like user questions) are directly linked to their intents. For enterprise-level chatbots, this means preparing a dataset with a mixture of synthetic data, crafted by data scientists, and authentic interactions from users. These inputs are then mapped to predefined, curated answers, with the AI learning to recognize and respond to variations of queries within the trained domain.
Domain Data
The responses in Traditional AI must be crafted and validated by humans. This guarantees that the chatbot responds correctly, following explicit guidelines. For example, in an HR service scenario, every potential query about hiring practices is paired with a corresponding, verified response.
This doesn’t mean answers cannot be automated, but the AI will have clearly defined instructions as to how it provides any given answer.
Training Data
Gen AI starts with an extensive understanding of language, courtesy of the LLM’s training on vast and varied public datasets. This eliminates the need for organizations to create training data. However, fine-tuning with organization-specific data is an option, albeit an expensive one. The trade-off with fine-tuning involves cost and the risk of exposing sensitive data. Instead, Gen AI tends to rely on enriched prompts, utilizing the RAG method to generate responses.
Domain Data
To tailor Gen AI to a particular domain, your unique organizational data is overlaid at the time of the prompt. Unlike Traditional AI, Gen AI doesn’t need an array of examples but does require high-quality, consistent, and current policy information and/or enterprise data to inform its responses. This means your effort is in curating policies which are timely, accurate and high-quality. There should be no contradictions in the data set.
Implementing AI for conversational interfaces varies significantly between Traditional AI and Generative AI (Gen AI), each with its own set of procedures and challenges.
The process begins with identifying the scope of queries the bot should answer, often derived from FAQs, customer support tickets, and data analytics. The next step involves creating training data for each identified question. This data synthesizes real and synthetic examples, which the AI uses to learn. Finally, answers are crafted and curated. Each of these steps can require domain experts and/or data scientists to ensure accuracy and relevance.
Gen AI deployment in the enterprise is centered on the Retrieve, Augment, Generate (RAG) approach. It starts with gathering domain-specific knowledge and ensuring that this information is accurate, up-to-date, and of high quality. This curated knowledge is then used to enrich prompts that guide the Gen AI’s responses. Unlike the discrete answer curation in Traditional AI, Gen AI relies on an extensive base of documents and policies to draw from, highlighting the importance of the adage “garbage in equals garbage out.” Similarly, domain experts are needed to manage this knowledge.
The security of AI-driven responses is a critical aspect, especially when sensitive information is involved.
In Traditional AI, security is implemented through predefined rules that determine the sourcing and sharing of answers. Because each response is based on explicit instructions, there’s a high level of confidence in the system’s security. For instance, in environments with varying information access, like different labor unions or educational levels, Traditional AI can provide persona-specific answers. Administrators have the advantage of complete visibility and control, ensuring responses are securely tailored and compliant with organizational protocols.
Security in Gen AI is more complex due to the dynamic nature of response generation. While it can incorporate persona-specific policies, there’s less certainty about the precise output, as it’s generated in real-time. Despite attempts to tag policies for specific personas, Gen AI might not always adhere to these when generating responses. This has raised valid concerns regarding the use of Gen AI in contexts where strict security filtering is crucial. For example, tests have shown that Gen AI may not consistently reference the intended policy documents, potentially leading to breaches in confidentiality or incorrect information dissemination.
Addressing questions outside the trained expertise is a common challenge for AI systems, handled quite differently by Traditional AI versus Gen AI.
Traditional AI operates within a defined set of parameters. When faced with out-of-domain queries, it typically lacks the confidence to provide an accurate answer and is programmed to offer a polite deflection, such as an apology or a redirection to human assistance. This conservative approach minimizes the risk of providing incorrect information.
In contrast, Gen AI will venture a response based on the LLM’s expansive understanding of language, regardless of the domain. This can be beneficial for general inquiries but poses a risk when the question falls outside the organization’s policies or the AI’s trained expertise. In such cases, Gen AI might produce responses that are distracting, potentially inaccurate, or even in violation of workplace policies.
NOTE: Some LLMs allow you to instruct the bot to not attempt to answer out-of-domain questions and perform more like traditional AI. In our early testing at the time of this post, that setting only sometimes works.
In order for your AI to learn and get better over time an understanding of the different learning mechanisms is required.
Traditional AI learns with new training data. This data can improve understanding or even teach the AI about new knowledge, like a policy. The training data comes by way of humans or autonomous software decisions. The AI doesn’t get better unless its data and corpus of outcomes gets better.
In the case of our own Ida product, most clients allow the software to autonomously supplement training data while the team is actively filling knowledge gaps. Our clients will train the model every 2-4 weeks so that the AI is constantly getting better.
Gen AI learns in a few ways, the first being retraining of the LLM that exists beneath the Gen AI. This is very expensive, but the cost is bore by the LLM provider and usually is retrained a few times a year. Because training happens infrequently, the LLMs understanding is based on older information.
While not traditional machine learning, the RAG method can improve over time as the knowledge it sources improves. When inconsistencies and inaccuracies are reduced in the source data or content, the RAG element will produce better prompts thereby producing answers that feel better, even if not done by way of AI learning.
Monitoring usage and effectiveness of the AI is important to continual improvement.
Imagine a well-organized filing cabinet. Each outcome by traditional AI is cataloged, allowing a clear understanding of how the AI reached its conclusion. This transparency allows for in-depth analysis of user queries, AI confidence levels, AI reasoning and the decision trail.
Analytics extend beyond individual interactions to include insights into data such as:
In contrast, generative AI operates more like a magician’s hat. While it delivers impressive results, the inner workings remain largely shrouded in mystery. We see the user’s query and the AI’s response, but the process of generating the response remains a black box.
This lack of transparency presents several challenges for analytics:
The maintenance of AI systems is critical to their effectiveness and accuracy, with practices differing significantly between Traditional AI and Generative AI (Gen AI).
The upkeep of Traditional AI involves regular analysis of user interactions to identify and address knowledge or comprehension gaps. This means continually feeding the system new data, which includes questions, answers, and enriched training data to improve its understanding and response accuracy. Retraining with this updated information helps the AI expand its capabilities and adapt to changes in user behavior and language. Additionally, it’s essential to periodically review and update existing answers to reflect any changes in policies or information.
Since the training of Gen AI’s LLMs is not typically handled by individual users or organizations (unless fine-tuning is applied), maintenance focuses on enhancing the prompts used in the RAG method. To address new topics or refine understanding, the knowledge base used for prompt enrichment must be updated. Maintenance for Gen AI also includes ensuring the information used for responses remains current and precise, necessitating regular content reviews and updates.
Traditional AI offers a controlled environment with predictable outcomes and robust security, making it suitable for scenarios where precision and reliability are paramount. Its maintenance and learning processes, though requiring more effort, provide a level of transparency and control that can be critical for sensitive or highly regulated domains.
Generative AI, on the other hand, offers versatility and the ability to handle a broader range of queries. Its ability to generate responses from a vast corpus of language data can provide advantages in terms of answering more questions with less effort. However, the challenges associated with its black-box nature, potential security concerns, and the need for high-quality domain data for accurate responses must be carefully considered.
Below we will illustrate major features of any AI and whether Traditional or Generative AI has an advantage.
| Traditional AI | Generative AI | |
| Implementation Effort | ✅ | |
| Lower Software Cost | ✅ | |
| Analytics | ✅ | |
| Control | ✅ | |
| Breadth & Scale | ✅ | |
| Ongoing Effort | ✅ | ✅ |
| Predictability | ✅ | |
| Security | ✅ | |
| Run-time Speed | ✅ |
Organizations must weigh these factors against their specific use cases, resources, and risk profiles. Ultimately, a hybrid approach may be the most pragmatic path forward, combining the reliability of Traditional AI for core, sensitive tasks with the adaptive prowess of Generative AI for more general inquiries. This is exactly the path Ida is taking. Such a strategy can harness the strengths of both systems while mitigating their individual weaknesses, leading to a robust, dynamic, and efficient AI implementation. If you have any questions or need help in your AI project, don’t hesitate to contact us.
Gideon Taylor is pleased to announce our participation in The Tambellini Group’s Future Campus™ Summit in Newport Beach, CA January 17-18, 2024. We’re also excited to share that our CEO Paul Taylor and IntraSee co-founder and Managing Director Andrew Bediz will be discussing how to avoid “analysis paralysis” when implementing Artificial Intelligence in a Higher Ed environment during a plenary session at the Summit. Please join us!
