Evolving Trends in Prompt Engineering for Large Language Models (LLMs) with Built-in Responsible AI Practices

Editor’s note: Jayachandran Ramachandran and Rohit Sroch are speakers for ODSC APAC this August 22-23. Be sure to check out their talk, “Evolving Trends in Prompt Engineering for Large Language Models (LLMs) with Built-in Responsible AI Practices,” there!

The advent of Transformer Architecture has indeed revolutionized the field of Natural Language Processing (NLP) by introducing a design that efficiently harnesses both data and computing power. Furthermore, the self-supervised pretraining of Transformer models on extensive corpora has demonstrated remarkable capabilities in tackling a wide array of NLP tasks. As researchers delved deeper into the impact of model scaling on enhancing capacity, they explored pushing the parameter scale to even greater sizes. Intriguingly, upon surpassing a certain threshold in parameter scale, these enlarged language models not only achieve significant performance improvements but also exhibit enhanced reasoning abilities. This led to the era of In-context learning (ICL), enabling Large Language Models (LLMs) to showcase foundational capability. Rather than relying solely on task-specific fine-tuning, these models can now execute specific tasks with carefully engineered prompts.

The emergence of Large Language Models (LLMs) has inaugurated a new era in the realm of artificial intelligence, reshaping the possibilities for organizations across diverse sectors. LLMs such as GPT-4, PaLM-2, Llama-2, and others are propelling the surge of Generative AI, ushering in novel applications that are reshaping both technological and business landscapes. From enhancing enterprise searches to powering conversational bots and content generation, LLMs are enabling unique capabilities that were once considered distant. However, this transformative shift does come with its share of challenges.

In this blog post, our objective is to illuminate the constantly evolving research around the LLMs space, while also addressing key ethical considerations and trying to provide practical guidance to AI practitioners and clients with examples of our internal use cases, facilitating the responsible development of LLM applications. In essence, we explore the transformative potential and the evolving landscape by delving into the following four critical dimensions:

Fig1: An illustration of the tech stack for four critical dimensions for adopting LLMs to various business use cases

1. Prompt Engineering: The goal is to steer the LLMs through refined prompts for effective instruction understanding and execution. At the core of efficient LLM utilization lies the art of prompt engineering which involves crafting prompts that guide LLMs effectively, paving the way for reliable responses. Various prompting techniques, such as Zero/Few Shot, Chain-of-Thought (CoT)/Self-Consistency, ReAct, etc. are harnessed to channel LLMs output.

2. Evaluating Prompt Completion: The goal is to establish effective evaluation criteria to gauge LLMs’ performance across tasks and domains. Measuring the performance of LLMs presents a complex challenge that demands thorough evaluation criteria to gauge LLM effectiveness. We showcase the following criteria for evaluation and mechanism for feedback to steer the LLMs towards optimal performance and continuous improvement.

1. Auto Eval
2. Common Metric Eval
3. Human Eval
4. Custom Model Eval

3. LLM Optimization & Deployment: The goal is to enhance LLM accessibility, employ PEFT methods for efficient, cost-effective fine tuning and deployment. Parameter Efficient Fine Tuning (PEFT) methods along with quantization based on QLoRA are making LLMs even more accessible & feasible for task-specific adaptation. These methods ensure that LLMs are not only finetuned effectively but also deployed with minimal computation needs and cost thereby aligning with the resource constraints of real-world applications.

4. Responsible AI: The goal is to emphasize and address ethical considerations in LLMs while fostering trust among users of AI applications. As LLMs become integral to AI applications, ethical considerations take center stage. We showcase the following indispensable Responsible AI principles safeguarding sensitive information, enhancing trust, and detecting bias to foster consumer confidence and ensure that the AI-driven outcomes are aligned with societal values.

1. Fairness/Bias
2. Explainability
3. Privacy
4. Security

At Course5 AI Labs, we are driving advances in the field of Artificial Intelligence (AI) through cutting-edge applied research, innovation, and rapid experimentation. One of our AI-Powered Augmented Analytics solutions is Course5 Discovery, which allows business users to ask natural queries and consume descriptive, predictive, and prescriptive insights. Here is a process flow of how we are applying the above four dimensions to Course5 Discovery which can be generalized for Text to SQL applications.

Fig 2: An illustration of applying four dimensions to our Course5 Discovery for Text to SQL applications

For AI applications based on LLMs, it is recommended to initially employ a custom model that offers metadata as guidance to LLMs. This trainable custom model can then be progressively improved through a feedback loop as shown above.

Learn more at our upcoming talk at ODSC APAC 2023:

Large Language Models (LLMs) have enabled organizations to reimagine and reinvent the technology and business ecosystems. These models are helping create unique capabilities, whether it’s enterprise search, topic identification, summarization, conversational bots, content generation, and many more. Organizations are leveraging LLMs through various means, such as out-of-the-box applications, prompt engineering, and model fine-tuning. Although we are witnessing early success, there are challenges, and adopting LLMs for various business use cases is still an evolving space. In this talk, we delve into the cutting-edge aspects of LLMs, focusing on four critical dimensions: Prompt Engineering, Evaluation, Model Optimization & Deployment, and Responsible AI.

About the authors: