I think adversarial error correction of LLM output in a loop with LLMs is underrated.
Peer review is the OG human based error correction loop! Even with all it's flaws. And with agents we can have machine based peer review of peer reviews of peer reviews of peer reviews and so on.
And depending on the setup, models, verifiability of the domain, exact output that is being fact checked, etc., the amount of errors in this error correcting loop can either converge to some constant or to some range, or diverge, or do some other weird oscillating.
I find this question very interesting recently. So recently I tried a quick very simple experiment to evaluate it more scientifically. I tried to evaluate iterative agentic fact checking error correction in a loop, grounded in web search, with two prompts. And here it basically plateaus after one fact check.
But this is very limited methodology. It's simplest version of this experiment. It has another LLM as a judge which has access to ground truth, which can still hallucinate, which is also weak model. And the dataset is just a single paragraph of STEM text with explicitly injected errors by LLM. And I used Codex with the weakest model as the fact checking agent. I should try this evaluation for facts that are exactly verifiable by exact symbolic code, but the fact checking agents will probably write code themselves to fix the errors. Or maybe the fact checking agent could write Lean and be instructed to never compile it the Lean code, but judge could compile it. I should also try this evaluation on something much bigger than a paragraph of text, like on the output of a whole agentic pipeline for generating whole research project with code, experiments, paper and so on. I really wonder what are the dynamics of all of this in various domains with current and future models.
Also I think there will be heavy tailed distribution, something similar to a power law, governing the difficulty of verifying the accuracy of verifiable statements for the various statements. And better models will be able to catch the tail better and introduce less additional errors in the tail in the process of correcting, relative to worse models. Also, nuance and simplifications are lovely, so accuracy is on a spectrum for various statements. And when you switch from easily variable scientific facts like most dogs having 4 limbs and try to do more formal statements, then stuff from theoretical compsci like stuff similar to proof complexity, average-case complexity, etc. becomes relevant. Also some statements cant be verified at all, or not yet.
