Test Cross-Lingual Transfer

Your model nails English prompts but stumbles on Spanish, French, or Hindi. That's the hidden cost of fine-tuning on a single language: it silently erodes the multilingual skills of a pretrained model. In this lesson, you'll learn to test cross-lingual transfer with a multilingual model — a critical check to prevent your fine-tuned model from becoming monolingual and to produce a model that generalizes across languages for users worldwide.

The problem this lesson solves

Fine-tuning a multilingual LLM on English-only data can cause catastrophic forgetting of other languages. The model's weights adjust to optimize English performance, degrading non-English representations. This is a real and costly issue: a customer support model trained solely on English tickets fails on queries in Spanish, even though the base model originally handled both. Without testing cross-lingual transfer, you ship a model that silently breaks for a significant portion of your audience.

Ignoring cross-lingual behavior leads to poor user experience, support overload, and reputation damage. The solution isn't to train on every language but to add a systematic test that measures transfer—so you can decide whether to fine-tune multilingually or invest in language-specific datasets.

Core concept / mental model

Think of cross-lingual transfer as a bridge: a multilingual base model is a vast network of roads connecting many cities (languages). Fine-tuning on one language is like building a highway to one city—those road improvements may incidentally benefit nearby cities, but they can also reroute traffic away from others. Cross-lingual transfer tests measure how much of the improvement from training on language A "spills over" to language B.

Formally, you compare performance on a target language before and after fine-tuning on a source language. A positive transfer means the fine-tuned model performs better on the target language than the base model. A negative transfer means performance degrades—a red flag.

Key metrics to watch:

• Accuracy or F1 on a target-language test set
• Perplexity (lower is better) when generating in the target language
• Entity F1 for tasks like NER to see if models recognize names in all languages

The goal isn't just to see if transfer happens; it's to understand when it works and when it fails, so you can make informed training decisions.

How it works step by step

Here's the workflow to test cross-lingual transfer with a multilingual model:

1. Select a multilingual base model — e.g., bert-base-multilingual-cased or mT5-small.
2. Split your data by language — prepare a source-language training set and a target-language test set (e.g., train on English, test on Spanish).
3. Fine-tune the model on the source-language data using your preferred method (full fine-tuning, LoRA, etc.).
4. Evaluate on both languages — use the same test set and metrics for the source and target languages.
5. Compare against the base model — measure relative performance to quantify transfer.
6. Reason about the results — positive transfer? negative? neutral? Decide next steps.

A critical detail: use identical evaluation pipelines for both languages to ensure fairness. Also, watch out for data leakage—if your target-language test set appears in the source-training data, results are meaningless.

Hands-on walkthrough

Let's implement a practical test using Hugging Face Transformers and datasets. We'll use a small multilingual subset to keep it fast.

Step 1: Load a multilingual dataset

First, load a dataset with parallel or multi-language annotations. We'll use xtreme and sample English (en) and Spanish (es) subsets.

from datasets import load_dataset

ds = load_dataset("xtreme", "xnli", split="validation")

# Filter for English and Spanish
ds_en = ds.filter(lambda x: x["lang"] == "en").select(range(200))
ds_es = ds.filter(lambda x: x["lang"] == "es").select(range(200))
print("EN size:", len(ds_en), "| ES size:", len(ds_es))

Expected output:

EN size: 200 | ES size: 200

Step 2: Fine-tune on the source language (English)

We'll use a small model and a simple classification token to keep the run short. In practice, you'd add a task head for YOUR task (e.g., NER, sentiment). Here, we fine-tune on a binary label (simulate a binary classification).

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-multilingual-cased", num_labels=2)

def tokenize(batch):
    return tokenizer(batch["premise"], batch["hypothesis"], truncation=True, padding="max_length", max_length=128)

# Prepare training data (use premise text as input, label from 'label')
ds_en = ds_en.map(tokenize, batched=True)
ds_en = ds_en.rename_column("label", "labels")
ds_en = ds_en.remove_columns(["premise", "hypothesis", "lang", "idx"])
ds_en = ds_en.map(lambda x: {"labels": x["labels"] % 2})  # binary for demo

training_args = TrainingArguments(
    output_dir="./en-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    evaluation_strategy="steps",
    eval_steps=50,
    save_total_limit=1,
    logging_steps=50,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=ds_en,
    eval_dataset=ds_en.select(range(50)),  # quick eval
)

trainer.train()

Expected output: training loss decreases over epochs (e.g., 0.69 → 0.45).

Step 3: Evaluate on English and Spanish

Evaluate the fine-tuned model on both test sets to measure transfer.

import numpy as np

# Prepare evaluation sets (same tokenization)
test_en = ds_en.select(range(100))
test_es = ds_es.map(tokenize, batched=True)
test_es = test_es.rename_column("label", "labels")
test_es = test_es.remove_columns(["premise", "hypothesis", "lang", "idx"])
test_es = test_es.map(lambda x: {"labels": x["labels"] % 2})

def evaluate_split(dataset):
    preds = trainer.predict(dataset)
    acc = (preds.predictions.argmax(-1) == dataset["labels"]).mean()
    return acc

acc_en = evaluate_split(test_en)
acc_es = evaluate_split(test_es)
print(f"Fine-tuned: EN accuracy = {acc_en:.2f}, ES accuracy = {acc_es:.2f}")

# Also evaluate the base model for comparison
from transformers import pipeline
pipe = pipeline("text-classification", model="bert-base-multilingual-cased", tokenizer=tokenizer)

# Quick pseudo-eval (simplified — see step 4 for proper method)
print("Base model: EN acc = 0.50, ES acc = 0.50 (random baseline)")

Expected output (example): Fine-tuned: EN accuracy = 0.62, ES accuracy = 0.61 (positive transfer!) or ES accuracy = 0.45 (negative transfer).

Step 4: Compare with base model and compute transfer score

Train a base model evaluation using the same metrics (we'll approximate here).

# Simulated base model performance (replace with actual evaluation)
base_acc_en = 0.55
base_acc_es = 0.56

# Calculate relative transfer
finetuned_en = acc_en
finetuned_es = acc_es

transfer_en = (finetuned_en - base_acc_en) / base_acc_en
transfer_es = (finetuned_es - base_acc_es) / base_acc_es

print(f"Transfer EN: {transfer_en:+.2%}, Transfer ES: {transfer_es:+.2%}")

if transfer_es < 0:
    print("Negative transfer detected — consider multilingual fine-tuning.")
else:
    print("Positive/neutral transfer — model benefits cross-lingually.")

Expected output: Transfer EN: +12.7%, Transfer ES: +8.9% or Transfer ES: -19.6%.

Compare options / when to choose what

Depending on your situation, several strategies exist to handle cross-lingual transfer:

For most finetuning projects, start with source-only fine-tuning and add target-language data only if your transfer tests show degradation. This saves data and compute.

Troubleshooting & edge cases

• Model outputs gibberish in target language after fine-tuning. Solution: Check tokenizer vocabulary—your fine-tuning may have overfit to source language-specific tokens; consider adding target-language examples.
• Positive transfer on accuracy but poor generation quality. Accuracy metrics may not capture fluency; evaluate with BLEU or perplexity on generation tasks.
• Data leakage in target test set. Ensure your target-language test texts aren't in the source training data; deduplicate before splitting.
• Unbalanced class distribution across languages. Use stratified splits; uneven labels can inflate or deflate accuracy metrics.
• Model architecture doesn't use multilingual embeddings. bert-base-uncased is not multilingual—verify your base model's language coverage before drawing conclusions.

What you learned & what's next

You now know how to test cross-lingual transfer with a multilingual model — from designing the experiment (source/target language split) to interpreting transfer scores, and when to pivot to multilingual fine-tuning. You can explain the core idea, apply it in a practical exercise, and connect it to your overall finetuning workflow.

Next, you'll learn how to mitigate negative transfer by incorporating target-language data during fine-tuning—a natural next step in the LLM Finetuning track. You'll also explore advanced techniques like language-aware adapters to efficiently support many languages.