English
Vals AI
Report

Vals AI✓

@ValsAI

Public LLM Evaluation // https://t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder

280 Following29K Followers2.4K Posts
  1. Vals AI✓@ValsAI
    For one weekend, we are benchmaxxing on purpose! Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build. Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors. Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs. Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
    Download
    96678.3K
  2. Vals AI✓@ValsAI
    About half of U.S. adults now use AI chatbots, and questions about visas, work permits and immigration status are among the most consequential they ask. Can the models be trusted? With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to test whether frontier models are able to answer these high-stakes questions.
    Download
    65984.2K
  3. Vals AI✓@ValsAI
    Posts @ValsAI
    For immigration attorneys, AI can help with research and drafting, but cannot replace advice tailored to a client’s situation. For AI labs, improving reliability means working with immigration experts on current, practice-grounded data.
    104467
  4. Vals AI✓@ValsAI
    Posts @ValsAI
    This benchmark is a first step. AI use in immigration shows potential, and we want it used with a clear view of where it falls short and what is at stake. Immigration Bench is a joint effort with the Talent Mobility Fund. We thank the immigration experts who wrote the questions and cross-reviewed every rubric. Results and methodology: Full report:
    005477
  5. Vals AI✓@ValsAI
    Ten Claude agents proved in Lean that seven electrons on a sphere settle into a pentagonal bipyramid. We’ve now written up the underlying method and suggest some values of N to tackle next.
    Download
    431305.5K
  6. Vals AI✓@ValsAI
    Posts @ValsAI
    Which N come next? We expect N=10, then N=9. N=32, an icosahedron nested inside its dual dodecahedron, is the hard one.
    Download
    106761
  7. Vals AI✓@ValsAI
    Posts @ValsAI
    For the full methodology, visit and
    004708
  8. Vals AI✓@ValsAI
    Vals AI
    RT @nikilravi: @aozerv and I looked through our eval traces, found the behavior that @cohere reported, and did some further digging!
    Vals AI @ValsAI
    We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
    010326
  9. Vals AI✓@ValsAI
    We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
    Download
    Cohere @cohere
    While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report:
    182922016K
  10. Vals AI✓@ValsAI
    Vals AI
    RT @aozerv: We found some half-concerning, half-endearing phone-home behaviors in Qwen3.8 models. Crazy what RL can do to someone!
    Vals AI @ValsAI
    We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
    020314
  11. Cohere✓@cohere
    While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report:
    9701.3K11K3M
  12. Vals AI✓@ValsAI
    Posts @ValsAI
    Qwen3.8 models have this behavior, and Qwen3.7 models don’t. We found that Qwen3.8 models hallucinated dates in these web requests that realistically coincide with when they might have been in RL training.
    Download
    21205K
  13. Vals AI✓@ValsAI
    Posts @ValsAI
    Read more in our blog post:
    01101.6K