LLM Evaluation Metrics: Core

Generation & Fluency Metrics are essential for evaluating how well an LLM generates text that is both accurate and coherent. These metrics assess various aspects of the generated text, such as its predictability, fluency, and diversity.

Below are metrics that can be used to measure these characteristics

2.1. Perplexity

Perplexity measures how confident a LLM is in generating text. It comes from probability theory and is commonly used in language modeling. A lower perplexity indicates the model is more confident in its predictions, suggesting that the generated text closely follows typical language patterns. On the other hand, a higher perplexity suggests more uncertainty, meaning the model is struggling to predict the next word accurately.

Perplexity ≈ 1: The model is very confident in its predictions, meaning it almost always knows the next word. This is ideal but rare. Highly Confident

Perplexity between 10-100: The model has moderate confidence, making good predictions most of the time. This range is typical for well-trained models. Moderate Confidence

Perplexity > 1000: The model is highly uncertain, struggling to predict the next word accurately. This indicates poor performance. Highly Uncertain

P("The") = 0.1, P("cat") = 0.2, P("sat") = 0.15, P("on") = 0.2, P("the") = 0.1, P("mat") = 0.25

P("The") = 0.1, P("cat") = 0.2, P("is") = 0.05, P("sitting") = 0.05, P("on") = 0.2, P("a") = 0.1, P("mat") = 0.25

Likelihood = P("The") * P("cat") * P("is") * P("sitting") * P("on") * P("a") * P("mat")

Likelihood = 0.1 * 0.2 * 0.05 * 0.05 * 0.2 * 0.1 * 0.25 = 0.00000025

Perplexity = exp(-1/N * sum(log(P(w_i))))

Perplexity = exp(-1/7 * log(0.00000025)) ≈ 42.64

Why Perplexity is Useful

Perplexity is useful for evaluating language model training. A model with low perplexity on a dataset has learned patterns well. However, if perplexity is too low, it might mean the model is overfitting, memorizing instead of generalizing.

2.2. BLEU (Bilingual Evaluation Understudy)

BLEU measures how similar the generated text is to a reference text by comparing n-grams (sequences of words). It is widely used in machine translation and summarization to evaluate accuracy, where a higher BLEU score indicates better similarity. BLEU works best for structured tasks like translation, where exact phrasing matters more than meaning.

BLEU Score ≈ 1.0: The generated text is nearly identical to the reference text. This is rare but ideal. Highly Similar

BLEU Score 0.5 - 0.9: The generated text has strong similarity but may contain slight variations. Moderate Similarity

BLEU Score < 0.5: The generated text differs significantly from the reference, indicating poor accuracy. Low Similarity

Simplified version of the BLEU formula:

Brevity Penalty: This adjusts the score if the generated text is shorter than the reference (to avoid short translations).

reference = "The cat sat on the mat"
generated = "The cat sat"
BLEU_SCORE = high precision but penalized for brevity

Average Precision of N-grams: This looks at how many words (or pairs of words) in the generated text match with the reference. Higher matches mean a better score.

{"The", "cat", "sat", "on", "the", "mat"}

{"The", "cat", "is", "sitting", "on", "a", "mat"}

{"The", "cat", "on", "mat"}

Precision = (matching unigrams) / (generated unigrams) = 4 / 7 = 0.57

BLEU = 1 * exp(log(0.57)) = 0.57

Why BLEU is Useful

BLEU is useful for quick and automatic evaluation of machine translation, ensuring the model follows the expected structure. It is fast and scalable for comparing large datasets.

2.3. METEOR (Metric for Evaluation of Translation with Explicit ORdering)

METEOR improves upon BLEU by considering synonyms, stemming, and word order. This makes it better at evaluating semantic similarity, meaning it can recognize different ways of saying the same thing.

High METEOR Score: The generated text conveys a meaning very close to the reference text, even with different wording. High Semantic Match

Moderate METEOR Score: Some words or structures differ, but the core meaning is preserved. Moderate Semantic Match

Low METEOR Score: The generated text is significantly different in meaning or structure. Low Semantic Match

Why METEOR is Useful

METEOR is useful for more flexible text evaluation, as it accounts for synonyms and grammatical variations. This makes it a better choice for assessing human-like text quality.

2.4. Diversity Metrics

Diversity Metrics evaluate the variety of generated outputs, ensuring responses are not repetitive and exhibit richness in language.

High Diversity Score (0.8 - 1.0): The generated text uses a wide range of vocabulary and sentence structures, avoiding repetition. High Linguistic Variety

Example: “The sunset painted the sky in hues of crimson, tangerine, and violet, casting long shadows over the quiet streets.”

Moderate Diversity Score (0.4 - 0.7): Some repetition is present, but the text still shows a reasonable level of variation. Moderate Linguistic Variety.

Example: “The sunset was beautiful, with shades of orange and pink. The streets were quiet as the sun went down.”

Low Diversity Score (0.0 - 0.3): The generated text is highly repetitive or lacks variety in word choice and structure. Low Linguistic Variety.

Example: “The sunset was nice. The sunset was colorful. The sunset was in the sky. The sunset was beautiful.”

Type-Token Ratio (TTR)

High TTR (0.8 - 1.0): Rich vocabulary with minimal repetition.

Example: “The cat observed the fluttering butterfly, then leaped gracefully onto the windowsill to bask in the golden sunlight.”

Moderate TTR (0.4 - 0.7): Some words are repeated, but the text still maintains a reasonable level of variety.

Example: “The cat sat on the mat, then walked to the door. The mat was soft, and the cat enjoyed sitting on it.”

Low TTR (0.0 - 0.3): The text contains frequent repetition, with limited vocabulary variety.

Example: “The cat, the cat, the cat sat. The cat sat on the mat. The mat, the mat, the mat.”

Distinct-N

High Distinct-1 & Distinct-2: The text uses a wide range of unique words and phrase combinations, indicating high lexical and phrase diversity.

Example: “The gentle breeze rustled through the autumn leaves as birds chirped melodiously in the distance.”

Moderate Distinct-1 & Distinct-2: Some words and phrases are repeated, but the text still maintains a reasonable level of variety.

Example: “The wind blew through the trees. The leaves swayed in the breeze as the birds chirped.”

Low Distinct-1 & Distinct-2: The text contains frequent repetition, leading to low lexical and phrase diversity.

Example: “Hello hello hello. The wind, the wind, the wind.”

Why Diversity Metrics are Useful

Diversity Metrics help ensure that AI-generated responses remain engaging, avoiding repetitive or overly formulaic outputs. This is especially important in creative applications like storytelling and dialogue generation.

Precision & Structure-Based Metrics help evaluate how well a generated text preserves key information and meaning. These metrics compare the output to a reference text, assessing aspects like precision, recall, and semantic similarity.

To measure these characteristics, we use structured metrics that compare the generated text to a reference. Two commonly used metrics for this are ROUGE-N and BERTScore

3.1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Measures how well a generated text matches a reference text. It is commonly used in summarization and text generation tasks. ROUGE scores evaluate overlap between generated and human-written texts, helping assess content quality.

ROUGE-N: Counting Matching Words

ROUGE-N measures the overlap of N-grams (sequences of N words) between the generated and reference texts. Higher scores indicate better content similarity. An n-gram is a sequence of N words. For example, in the sentence “The cat sat”, the bigrams (2-grams) are “The cat” and “cat sat”.

ROUGE-1: Measures overlap of unigrams (single words). Useful for evaluating lexical similarity. Basic Word Overlap

ROUGE-2: Measures overlap of bigrams (two-word sequences), capturing more contextual accuracy. Phrase-Level Similarity

{"The cat", "cat sat", "sat on", "on the", "the mat"}

{"The cat", "cat is", "is sitting", "sitting on", "on a", "a mat"}

{"The cat"}

ROUGE-2 score = (matching bigrams) / (reference bigrams)

ROUGE-2 score = 1 / 5 = 0.20 (20%)

ROUGE-S: Skip-Bigram Overlap

ROUGE-S (ROUGE-Skip) accounts for skip-bigrams—pairs of words appearing in the same order but not necessarily adjacent. It helps evaluate semantic similarity even if the exact word sequence differs.

Higher ROUGE-S: Suggests strong similarity in meaning, even with slight word rearrangements. Flexible Similarity Matching

Lower ROUGE-S: Indicates significant structural differences, reducing semantic alignment. Weak Content Overlap

("The cat", "cat sat", "sat on", "on the", "the mat", "The sat", "cat on", "sat the", "on mat")

(“The cat”, “cat is”, “is sitting”, “sitting on”, “on a”, “a mat”, “The is”, “cat sitting”, “is on”, “sitting a”, “on mat”)

ROUGE-S score = (matching skip-bigrams) / (reference skip-bigrams)

ROUGE-S score = 2 / 9 = 0.22 (22%)

ROUGE-L: Longest Common Subsequence

ROUGE-L measures the longest common subsequence (LCS) between the generated and reference texts. Unlike ROUGE-N, it considers word order and structure.

Higher ROUGE-L: Indicates better structural alignment, meaning the generated text follows a similar word order as the reference. Strong Structural Match

Lower ROUGE-L: Suggests significant deviations in sentence structure, reducing fluency. Weak Structural Match

["The", "cat", "sat", "on", "the", "mat"]

["The", "cat", "is", "sitting", "on", "a", "mat"]

["The", "cat", "on", "mat"]

ROUGE-L = (LCS Length) / (Length of Reference Text)

ROUGE-L = 4 / 6 = 0.67 (67%)

3.2. BERTScore

Measures how similar two texts are using word embeddings. Unlike ROUGE, which checks exact word matches, BERTScore understands meaning by comparing words in a semantic space.

How BERTScore Works

Imagine each word is a point in space. BERTScore finds how close these points are between the generated and reference text. Closer points mean higher similarity.

High BERTScore: The generated text is semantically close to the reference, even if words are different. Strong Meaning Match Example: Reference: “The cat is sitting on the mat.” Generated: “A cat is resting on the carpet.” BERTScore: 0.92

Medium BERTScore: The generated text has some semantic overlap with the reference, but not a perfect match. Partial Meaning Match Example: Reference: “The cat is sitting on the mat.” Generated: “A dog is lying on the rug.” BERTScore: 0.78

Low BERTScore: The generated text has different meaning, even if some words overlap. Weak Meaning Match Example: Reference: “The cat is sitting on the mat.” Generated: “The weather is rainy and cold.” BERTScore: 0.45

Why BERTScore is Useful

BERTScore is great for evaluating summaries and translations, since it looks beyond exact words and focuses on meaning. However, it requires pre-trained models, making it more complex than ROUGE.

["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]

["A", "fast", "brown", "fox", "leaps", "over", "the", "lazy", "dog"]

  Cosine similarity("quick", "fast") = 0.88
  Cosine similarity("jumps", "leaps") = 0.92
  Cosine similarity("the", "the") = 1.00
  Cosine similarity("lazy", "lazy") = 1.00
  Cosine similarity("dog", "dog") = 1.00

BERTScore = 1 / N * Σ cosine_similarity

BERTScore = (0.88 + 0.92 + 1.00 + 1.00 + 1.00) / 5 = 0.96

4. Compare different metrics

METEOR vs BERTScore

Both METEOR and BERTScore evaluate semantic similarity, but differ in approach:

Matching approach: METEOR uses synonym matching, stemming, and word order for surface-level matches. BERTScore uses contextualized embeddings for deeper semantic understanding.

Handling abstract meaning: METEOR works with straightforward content but struggles with figurative language. BERTScore better captures meaning but has limitations with creative content.

Computation complexity: METEOR is simpler and less resource-intensive. BERTScore requires pre-trained models, making it slower but more accurate.

Context understanding: METEOR considers word order but misses contextual meanings. BERTScore better understands context and multiple word meanings.

ROUGE vs BLEU

Both ROUGE and BLEU are evaluation metrics for text generation, but they approach measurement differently:

Measurement focus: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) prioritizes recall, measuring how much of the reference text appears in the generated text. BLEU (Bilingual Evaluation Understudy) emphasizes precision, measuring how much of the generated text appears in the reference.

Application domain: ROUGE was designed for summarization tasks, making it better for evaluating condensed content. BLEU was created for machine translation and works best when evaluating texts of similar length to their references.

N-gram handling: ROUGE offers variants (ROUGE-N, ROUGE-L, ROUGE-S) to capture different linguistic aspects. BLEU uses a weighted average of n-gram precisions with a brevity penalty to prevent very short outputs from scoring highly.

Limitations: ROUGE may reward verbose outputs that include reference content among filler. BLEU may penalize valid paraphrases that use different wording than references. Neither fully captures semantic meaning or fluency.