> ## Documentation Index
> Fetch the complete documentation index at: https://ragopt.aboneda.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 3. Generation Metrics

Generation metrics evaluate the quality of your LLM's responses using an LLM-as-a-judge approach.

<Note>
  **Note:** All generation metrics are Based on paper [Faster, Cheaper, Better
  Multi-Objective Hyperparameter Optimization for LLM and RAG
  Systems](https://arxiv.org/abs/2502.18635).
</Note>

## Quick Reference

| Metric                | Measures             | Key Question                 | Score Range |
| --------------------- | -------------------- | ---------------------------- | ----------- |
| **SafetyMetric**      | Factual grounding    | Is it true?                  | 0-1         |
| **AlignmentMetric**   | Usefulness & clarity | Is it helpful?               | 0-1         |
| **ResponseRelevancy** | Query-answer match   | Does it answer the question? | 0-1         |

***

## SafetyMetric (Faithfulness)

**Measures:** Whether responses are grounded in retrieved contexts, preventing hallucinations.

### Basic Usage

```python theme={null}
from rag_opt.eval import SafetyMetric,RAGEvaluator
from rag_opt import init_chat_model

llm = init_chat_model(model="gpt-3.5-turbo",
                    model_provider="openai",
                    api_key=OPENAI_API_KEY)

metric = SafetyMetric(llm=llm)
evaluator = RAGEvaluator(metrics=[metric])
eval_dataset = ... # rag.get_batch_answers(train_dataset)
results = evaluator.evaluate(dataset)

print(f"Safety Score: {results[0].value:.3f}")
```

### Configuration

```python theme={null}
from rag_opt.dataset import EvaluationDataset,EvaluationDatasetItem,GroundTruth
from rag_opt.eval import SafetyMetric,RAGEvaluator
from rag_opt import init_chat_model

llm = init_chat_model(model="gpt-3.5-turbo",
                    model_provider="openai",
                    api_key=OPENAI_API_KEY)

# Custom prompt (must include {contexts}, {question}, {answer})
custom_prompt = """
Verify if this answer is fully supported by the contexts.

Contexts: {contexts}
Question: {question}
Answer: {answer}

Rate faithfulness (0-100):
- 100: Completely grounded
- 50: Partially grounded
- 0: Fabricated or contradicts contexts

Return only a number.
"""
metric = SafetyMetric(llm=llm, prompt=custom_prompt)
evaluator = RAGEvaluator(metrics=[metric])
eval_dataset = EvaluationDataset(
    items=[
        EvaluationDatasetItem(
        question="what is my name",
        answer="John",
        contexts=["My name is John"],
        ground_truth=GroundTruth(answer="John", contexts=["My name is John"]),
    )]
)
results = evaluator.evaluate(eval_dataset,return_tensor=False)
for _,result in results.items():
    print(f" Score: {result.value:.3f}")
```

***

## AlignmentMetric (Helpfulness)

**Measures:** Whether responses are useful, detailed, and clear.

### Basic Usage

```python theme={null}
from rag_opt.eval import AlignmentMetric,RAGEvaluator

llm = ...
metric = AlignmentMetric(llm=llm)
evaluator = RAGEvaluator(metrics=[metric])
results = evaluator.evaluate(eval_dataset,return_tensor=False)

print(f"results: {results}")
```

***

## ResponseRelevancy

**Measures:** Whether the response actually answers the question asked.

### Basic Usage

```python theme={null}
from rag_opt.eval import ResponseRelevancy,RAGEvaluator

metric = ResponseRelevancy(llm=llm)
evaluator = RAGEvaluator(metrics=[metric])
results = evaluator.evaluate(eval_dataset,return_tensor=False)

print(f"results: {results}")
```

***

### When to Use Each Metric

**1. SafetyMetric**

* Factual accuracy is critical
* Compliance/legal requirements apply
* Working in high-stakes domains

**2. AlignmentMetric**

* User experience is priority
* Optimizing for helpfulness
* Balancing detail vs brevity

**3. ResponseRelevancy**

* Ensuring on-topic responses
* Building Q\&A systems
* Quality assurance for chatbots