The first comprehensive embedding benchmark for Slovak — and two open models that run locally yet match proprietary APIs.
A 45M-parameter Slovak model reaches 91% of the best 560M multilingual model — and runs right here in your browser↓
Slovak — a West Slavic language with ~5 million speakers — had no embedding benchmark. MMTEB covers it with just 8 tasks, mostly multilingual subsets. SkMTEB collates 31 datasets spanning news, government, medical, social-media, and encyclopedic Slovak (2000–2025), including 7 datasets created specifically for this work. Only 6 overlap with MMTEB, so the two are complementary.
We adapt Multilingual E5 by vocabulary trimming (250K → 60K tokens, kept by Slovak frequency) followed by fine-tuning on curated skLEP data. Training takes under one GPU-hour per model on a single H100. Both are open-weight and locally deployable — no API, no data leaving the machine.
On par with text-embedding-3-small (70.48) at a fraction of the footprint. Small enough to run directly in a web browser.
Comparable to text-embedding-3-large (75.07); TOST equivalence within ±2 points. An open model you can host yourself for Slovak semantic search and RAG.
No server, no API key. Load the 45M-parameter model once (~45 MB, then cached) and it runs entirely on your device. Ask a question in Slovak and it ranks the snippets by meaning, even when none of the words match.
Downloaded once and cached · nothing leaves your device · powered by transformers.js
Each snippet is prefixed with passage:, the query with query:, then ranked by cosine similarity — the same recipe used to score the model on SkMTEB.
Average SkMTEB score vs. parameter count (log scale). The Pareto frontier connects models that no smaller model beats. Both Slovak models sit on or near the frontier — and beyond ~600M, scale stops paying off for Slovak.
All 31 models on SkMTEB (percent). All = mean over 31 datasets; Type = unweighted mean over task types. Click any header to sort; best per column is highlighted. Multilingual E5-large-instruct leads overall, with Gemini close behind.
Vocabulary trimming removes tokens that Slovak never uses. It cuts size by 35–62% while quality holds steady or improves slightly, and transfer to English and Czech stays within 1 F1 point.
| Variant | VT | FT | Size | Avg | Δ |
|---|---|---|---|---|---|
| mE5-small | 118M | 70.32 | — | ||
| + VT | ✓ | 45M | 70.45 | +0.13 | |
| + VT + FT | ✓ | ✓ | 45M | 70.56 | +0.24 |
| + VT + FT + prompt | ✓ | ✓ | 45M | 71.07 | +0.75 |
| mE5-large | 560M | 74.25 | — | ||
| + VT | ✓ | 365M | 74.56 | +0.31 | |
| + VT + FT | ✓ | ✓ | 365M | 74.70 | +0.45 |
| + VT + FT + prompt | ✓ | ✓ | 365M | 74.72 | +0.47 |
Ablation: vocabulary trimming (VT), fine-tuning (FT), and query:/passage: prompts. Δ is vs. the untrimmed baseline.
| Bitext pair | E5-sm orig | trim | E5-lg orig | trim |
|---|---|---|---|---|
| Flores ces–slk | 99.87 | 99.74 | 99.87 | 99.87 |
| Flores eng–slk | 95.58 | 95.72 | 100.0 | 100.0 |
| NTREX ces–slk | 98.08 | 98.11 | 99.22 | 99.27 |
| NTREX eng–slk | 94.64 | 95.56 | 98.80 | 98.66 |
| Tatoeba slk–eng | 82.69 | 82.56 | 93.22 | 93.22 |
| Opus slk–eng | 77.08 | 76.95 | 86.21 | 86.27 |
Cross-lingual transfer is preserved: mean change 0.25 F1 (small) and 0.04 F1 (large) after trimming.
SkMTEB ships through the standard MTEB library. Evaluate any model in a few lines.
# pip install mteb import mteb model = mteb.get_model("slovak-nlp/e5-sk-small") tasks = mteb.get_benchmark("SkMTEB") results = mteb.MTEB(tasks=tasks).run(model, output_folder="results/")
@inproceedings{suppa-etal-2026-skmteb,
title = {{SkMTEB}: {S}lovak Massive Text Embedding Benchmark and Model Adaptation},
author = {{\v{S}}uppa, Marek and Ridzik, Andrej and Hl{\'a}dek, Daniel and
Kn{\v{a}}{\v{z}}ekov{\'a}, Nat{\'a}lia and Ondrejov{\'a}, Vikt{\'o}ria},
booktitle = {Proceedings of ACL},
year = {2026},
eprint = {2606.13647},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.13647},
}