Bryan Li

In a globalized world, every event has ripple effects. When LLMs turn to the open web, how do they reflect state media? We release the WorldStateMedia dataset: 118M articles from 136 countries in 114 languages, and study their prevalence, narratives, and impact on LLMs. 1/6🧵

What happens when LLMs retrieve from state media on territorial disputes? RAG over state media sources decreases factuality vs. RAG over Wikipedia. And for multilingual queries, RAG over English sources increases divergent responses vs. over in-language sources. 5/6🧵

So glad to get this final project out the door, a year post-PhD graduation! Huge thanks to co-authors @samarhdr @_grace_kim @fionalluo @peterljq @lmgAboveFog @duncanjwatts @vidal_rene and Chris. 📜 🤗 6/6🧵

Given a request, AI systems generate long-form reports. Report evaluation relies on nuggets: essential & granular facts. Prior work on nugget generation treats them as short statements. Our #ICTIR2026 oral proposes DoGMaTiQ, an automated pipeline for Q&A nuggets. 1/6🧵

DoGMaTIQ scores for reports also best correlate with manual human scores, at system-level and at topic-level. More experiments are in our paper: circularity of LLMs for report generation + evaluation, as well as robustness to underperforming systems. 5/6🧵

Thanks to collaborators @pybeebee @houyu0930 @lawrie_dawn @EYangTW @lauradietz99 + more! At SIGIR #ICTIR2026 in Melbourne 🇦🇺 @EYangTW will present on July 24 at 14:00, or catch him at the poster session! 📜 📊 6/6🧵

In a world of geopolitical conflicts, how can AI help us navigate? Our #ACL2025-F work studies RAG robustness across 49 languages. TL;DR: 📈 boost robustness w/ multilingual RAG, 🤔 take care w/ low-resource citations 📜 🤗 1/4 🧵

We study cross-lingual robustness over 4 LLMs and 2 IR models. We find A) multilingual RAG performs best; B) LLM’s citations varies widely across langs. Our further experiments investigate aspects of cross-lingual RAG from IR to LLM explanations. 3/4 🧵

This is the final paper of my PhD! Thanks to my many @upennnlp collaborators: @samarhdr, Chris, and the 7 wonderful students who I was fortunate to mentor. Please look out for our poster at ACL 2025 in Vienna. 4/4 🧵

RT @laurenbjiang: 🚀 How well can LLMs know you and personalize your response? Turns out, not so much! Introducing the PersonaMem Benchmark -- 👩🏻‍💻Evaluate LLM's ability to understand evolving persona from 180+ multi-session user-chatbot conversation history 🎯Latest models (GPT-4.1, GPT-4.5, o4-mini, Llama-4, Gemini 2.0, Deepseek-R1, Claude-3.7) all struggle in personalization! 🎨7 personalization skills tested in 15 scenarios 🌟Realistic long-context evaluation up to 1M tokens 👇 Check out what we discovered… (1/6)

Externally retrieving knowledge empowers LLMs for domain-adapted MT ⚖️🩺. But how is knowledge best represented, and how viable is generating it from an LLM itself? Our @GoogleAI paper investigates these questions through a careful experimental setup 📜.

TL;DR - translation pairs > bilingual terminologies, generation especially boosts translations for small LLMs Our ablations highlight the need for more challenging domain-adapted MT datasets with modern LLMs. Thanks to collaborators Jiaming, @ebriakou & @ColinCherry!

RT @YueYangAI: We share Code-Guided Synthetic Data Generation: using LLM-generated code to create multimodal datasets for text-rich images, such as charts📊, documents📄, etc., to enhance Vision-Language Models. Website: Dataset: Paper: Code: