Aleph Alpha

Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.

How does the community evaluate multilingual capabilities of LLMs? Mostly by translating evaluation datasets, since most of them are predominantly in English. How does translation change what the eval actually measures?

Foreign cultural context is the harder problem. Questions about another country’s laws still test that country’s culture, but in German. A translation often implicitly decides what that cultural context is, influencing what that eval measures.

As model builders for German-speaking users, we believe this choice should be ours to make. Read more in our new blog by Alejandro, Raziye, Prabhu, @frankstefansch1, and Martin:

Scaling RL environments and their infrastructure comes with several challenges. We documented our experiences while building Kolibri: stalling GPUs, images filling the disks, crashed sandboxes scored as wrong answers, and reward hacks.

Blog post by @gitlostmurali, @LukasBluebaum, and Max Hammer: Tech report:

Kolibri 1 now runs natively in Transformers 5.20! We are still overwhelmed by all the community feedback. You can now spin up our Kolibri models in just a few lines of code using 🤗transformers. Special thanks go to Luca Motz (@lucamotz3141) and Harry Mellor (@hmellor_) for pushing this.

The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.

Given a prompt that ends with a space, we find that Olmo 3 still generates an answer with a space and is semantically correct 90% of the time.

Our recommendation is to split the prompt and the answer so that the tokens match what the model is trained with. This can be checked before any inference, with just a tokenizer: tokenizing the prompt and answer separately should give the same tokens as tokenizing them together. Read the full blog here:

Expert load balancing is key to training Kolibri efficiently at scale. We developed two new methods: Exact Quantile Balancing (EQB) for global balance and Load-Error Injection (LEI) for local balance. @JesseLiJiaze and @PitNeitemeier explain them in a new, visual blog post.

Blog post: Paper:

Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵

Across 62 base checkpoints from 5 model families, PriorBPB reduces mean |correlation| with answer length from 0.68 to 0.18. It also tracks accuracy along capability ladders better than mean-of-ratios BPB on both base and instruct checkpoints.

tl;dr: We suggest to not use mean-of-ratios or corpus BPB alone and use PriorBPB with a stated shared prior for cross-task comparisons. Check out the blog post, paper, and Apache 2.0 code by @tfburns and the team here: