v0.25.0: new Harbor features to mitigate reward hacking
31310921.3K
- "'Evaluator' as a role has the potential to be as prevalent as 'software developer'"Onur Solmaz @onusozHaving 6.5k stars on @harborframework repo is crazy The group of people doing evals is still so small compared to developers as a whole But we know literally every company who uses AI will be running their own private evals at some point... "Evaluator" as a role has the potential to be as prevalent as "software developer"02221.7K
Having 6.5k stars on @harborframework repo is crazy The group of people doing evals is still so small compared to developers as a whole But we know literally every company who uses AI will be running their own private evals at some point... "Evaluator" as a role has the potential to be as prevalent as "software developer"
30284.5K- MMNE (make model names easy)Kobe @kobe0938Which model names (-m) can you use in Harbor? Now there's one answer: LiteLLM's.006716
- Harbor 🤝 Docker SandboxesDocker @DockerAgents that can edit files and run commands can also game their own evals. Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs.15451.8K
- Posts @harborframework𝚞𝚟 𝚝𝚘𝚘𝚕 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 "𝚑𝚊𝚛𝚋𝚘𝚛[𝚜𝚋𝚡]" 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚎 𝚜𝚋𝚡 -𝚝 <𝚝𝚊𝚜𝚔> to get started!001221
- Harbor FrameworkDocker @DockerAgents that can edit files and run commands can also game their own evals. Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs.01024
- 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚛𝚎𝚏𝚛𝚎𝚜𝚑/𝚋𝚕𝚎𝚗𝚍𝚎𝚛𝚋𝚎𝚗𝚌𝚑-𝟸𝟶Refresh @refresh_devIntroducing Blender Bench v1 (BBv1): a benchmark for frontier LLMs on 20 realistic 3D production workflows including animation, modeling, rigging, cloth & surfacing. GPT-6 Astra leads overall; Claude Opus 5.5 wins modeling & cloth simulation.102536
- results need receipts! 𝚑𝚊𝚛𝚋𝚘𝚛 𝚞𝚙𝚕𝚘𝚊𝚍 to share your trials on the hubUnreal Labs @unreallabsaiFull results, harbor traces and implementation details in our blog -005767
Posts @unreallabsaiFull results, harbor traces and implementation details in our blog -11201.9K- 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚕𝚘𝚘𝚗𝚒-𝚕𝚊𝚋/𝚜𝚖𝚍𝚍-𝚋𝚎𝚗𝚌𝚑Suresh @SureshRaghu07New blog: Is human taste overrated in harness engineering? An agent harness is the system around a model. It decides which tools the model can call, what evidence it sees, and what it remembers between steps. Most harnesses are still designed by human taste. Run the agent, read where it failed, change the harness, try again. That loop is increasingly being automated, with a strong model rewriting the harness itself. So we asked how far that can go. We ran Qwen on three drug design tasks from SMDD-Bench and compared a manually redesigned harness against one produced by automated harness search with Claude in the loop. No single approach won everywhere. The manual harness pulled clearly ahead when the fix was changing what the agent sees and remembers. Automated search did better once that was in place and the remaining problem was how to search the chemistry itself. The hard part was rarely implementing the fix. It was diagnosing what kind of failure we were looking at, and that is where human taste still mattered. Work with @KevinH1119568 , @aviral_kumar2 and @niloofar_mire Read more 👇115803
New blog: Is human taste overrated in harness engineering? An agent harness is the system around a model. It decides which tools the model can call, what evidence it sees, and what it remembers between steps. Most harnesses are still designed by human taste. Run the agent, read where it failed, change the harness, try again. That loop is increasingly being automated, with a strong model rewriting the harness itself. So we asked how far that can go. We ran Qwen on three drug design tasks from SMDD-Bench and compared a manually redesigned harness against one produced by automated harness search with Claude in the loop. No single approach won everywhere. The manual harness pulled clearly ahead when the fix was changing what the agent sees and remembers. Automated search did better once that was in place and the remaining problem was how to search the chemistry itself. The hard part was rarely implementing the fix. It was diagnosing what kind of failure we were looking at, and that is where human taste still mattered. Work with @KevinH1119568 , @aviral_kumar2 and @niloofar_mire Read more 👇662511.5K- The Harbor team will be at Runtime tomorrow! @alexgshaw will be hosting a Harbor genius bar @ryan_marten will be presenting “Lessons from Terminal-Bench” Come with your questions and feature requests and say hi!16498.1K
- Harbor FrameworkRT @LaudeVentures: With hundreds of contributors and thousands of participants on Discord and GitHub, the @terminalbench team ships frequent updates in the open so everyone can learn alongside them. And with @harborframework, they've built the infrastructure for anyone to benchmark the use cases they care about. Great to host the community at Laude Lab last week and hear what's coming next! @alexgshaw @ryan_marten @StevenDillmann @Mike_A_Merrill @andykonwinski04018
- Harbor FrameworkRT @ishanxnagpal: hey @harborframework community i just ported these 2,698 tasks to harbor what model should i post-train with these?elie @eliebakouchxiaomi MiMo oss RL environments + training code02022
- Harbor FrameworkRT @alexgshaw: Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union010013
- Harbor FrameworkRT @ryan_marten: The gold standard for science is sharing reproducible and auditable experimental data. When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial. We built harbor to make this easy. Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!Unreal Labs @unreallabsaiIntroducing Unreal Agent: An open-source harness with state-of-the-art cost efficiency 39% cheaper than Codex+Astra on Terminal-Bench 4.0 while maintaining performance04017
- Introducing Unreal Agent: An open-source harness with state-of-the-art cost efficiency 39% cheaper than Codex+Astra on Terminal-Bench 4.0 while maintaining performance
2163154.4K763K - Harbor FrameworkRT @raphaelee: Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command. Full writeup:01024
- Harbor FrameworkRT @benediktstroebl: @vercel’s harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework02021
- Jev-as-a-Judge in Harbor Rewardkit!Benedikt Stroebl @benediktstroeblJev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.11353.5K
Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.
26245.9K
