Align your models

Alignment is the process of managing the behavior of generative AI (GenAI) to ensure its outputs conform with your products needs and expectations. These efforts are an open and active research area, and you need to decide what it means for your model to be aligned with your product, and how you plan to enforce that. In this document, you can learn about two techniques—prompt templates and model tuning—and tools that enable prompt refactoring and debugging that you can employ to achieve your alignment objectives. For more on model alignment goals and approaches, see Artificial Intelligence, Values and Alignment.

Prompt templates

Prompt templates, also called system prompts, provide context around user input and model output, depending on your use case, as system instructions and few-shot examples that guide the model toward safer and higher quality outcomes. For example, if your objective is high quality summaries of technical scientific publications, you may find it helpful to use a prompt template like:

The following examples show an expert scientist summarizing the
key points of an article. Article: {{article}}
Summary:

Where {{article}} is a placeholder for the article being summarized.

Contextualized prompt templates can significantly improve the quality and safety of your model's output. However, writing prompt templates can be challenging and requires creativity, experience, and a significant amount of iteration. The Model Alignment library provides two methods for iteratively improving your prompt template design with the assistance of LLMs, such as Gemini. Additionally, there are many prompting guides available, including best practices for the Gemini API and Vertex AI.

Prompt templates typically provide less robust control over the model's output compared to tuning, and are more susceptible to unintended outcomes from adversarial inputs. To accurately understand how well a prompt template is performing toward specific safety objectives, it is important to use an evaluation dataset that wasn't also used in the development of the template. Prompt debugging tools can also be useful for understanding the specific interactions between system, user, and model content in the prompts your model sees. Specifically, they can connect portions of the generated output to the most relevant and influential content in the prompt.

Model tuning

Tuning a model starts from a checkpoint, a specific version of a model, and uses a dataset to refine the model's behavior. Gemma models, and other LLMs, are available in both Pretrained (PT) and Instruction Tuned (IT) variants. PT variants treat the prompt as a prefix to continue from, whereas IT tuned variants have been further tuned to treat the prompt as a set of instructions describing how to complete a task. In most cases, you should start with an IT variant, to inherit the basic instruction-following and safety benefits, but further tuning may be required to achieve your specific application objectives.

Tuning models for safety is delicate and challenging. If a model is over-tuned, it can lose other important capabilities. For an example, see the catastrophic interference issue. Moreover, safe behavior for a model is contextual. What is safe for one application may be unsafe for another. If you find that your model is not performing well to additional safety tuning, consider adopting safeguards that enforce your behavioral policies instead.

Two of the most well known approaches to tuning LLMs are supervised fine-tuning (SFT) and reinforcement learning (RL).

• Supervised fine-tuning (SFT) uses a dataset of examples that encodes the desired behavior of your application by labeling the data. To use SFT to tune your model for safety, you need to have a dataset with examples of safe and unsafe behaviors, so that the model can learn to distinguish the differences.
• Reinforcement learning from human preferences (RLHF) uses a reward model to score your LLMs generations based on their compliance with behavioral policies. As with SFT, the RLHF reward model needs to be trained on both safe and unsafe behaviors in order to generate an appropriate score. While more expensive, RLHF can result in better performance as the reward model has more flexibility to encode nuance into its score.

For both techniques, the final results largely depends on the quality of your tuning data. Once you have the right data, you can tune the Gemma model using KerasNLP.

Gemma Instruction Tuning Tutorial

Start Google Colab

Data requirements and characteristics

Research studies have shown that data quality is often more important than quantity, so you should invest time reviewing your training examples to ensure the quality of your data (Touvron, 2023; Zhou, 2023).

Two common approaches to curate training and evaluation data for tuning are to generate user input manually or with the help of LLMs. Human data generation is often higher quality and requires less oversight review than LLM generated data, but it is harder to scale and often more costly.

Supervised fine-tuning data

Data for safety fine-tuning is largely reliant on examples of user queries likely to elicit a model response that violate your content policies. During the SFT stage the model is typically fine-tuned on pairs of adversarial queries with safe responses. There are two kinds of adversarial queries. Well-intended queries that might inadvertently result in unintended application behavior and queries designed by adversarial users who intend to make your system act in unintended ways. For more details, see the Evaluate model and system for safety guide.

Example of safety fine-tuning datapoint

RL data

Reinforcement Learning is a two-stage process. The first stage is to train the reward model, and the second is to use the reward model to train your application's model. To train the reward model, user queries are needed and typically include two possible responses and a label indicating which of the two responses should be preferred. A typical data point looks as follows:

Example of safety datapoint for RLHF

For the second step of RLHF, you only need a set of user queries. In the safety-specific case, you want to carefully balance your dataset with adversarial and non-adversarial queries. More on adversarial data can be found in the section Evaluate model and system for safety.

Best practices for model tuning

As mentioned earlier, the quality of your tuning set conditions the quality of the final model. You should consider the following factors when creating your tuning dataset:

• Data coverage: Your dataset must cover all your content policies for each of your product use cases (e.g., question answering, summarization, and reasoning).
• Data diversity: The diversity of your dataset is key to ensuring that your model is tuned properly and spans across many characteristics. It may need to cover queries of various lengths, formulations (affirmative, questions, etc.), tones, topics, levels of complexity as well as terms related to identities and demographic considerations.
• Deduplication: Just like for pre-training data, removing duplicated data reduces the risk that tuning data will be memorized and also reduces the size of your tuning set.
• Contamination with evaluation sets: Data used for evaluation should be removed from the tuning data.
• Responsible data practices go beyond filtering: Mis-labeled data is a common source of model errors. Offer clear instruction to the people in charge of labeling your data, either your team or external raters if you are using crowd-rating platforms, and aim for diversity in your rater pools to avoid unfair bias.

Developer resources

• High quality tuning datasets, including safety-related data:
  • Anthropic's dataset for RLHF
  • Safety-Tuned LLaMAs datasets
• Google's People + AI Guidebook offers a deep insight into a responsible approach to data collection and preparation.
• LIT website