Multi-Agent LLM Research Framework
A research framework for language-model agents playing a social deduction game. I developed game logic, agent reasoning modules, and an evaluation pipeline to compare how agents use incomplete information.
View source on GitHubThe question
Can language-model agents reason about what other players know, defend a claim, and change a group’s decision? One-Night Ultimate Werewolf provided a bounded setting: players have hidden roles, limited information, and a shared decision to make.
The system
The framework combines a game engine with agents ranging from a random baseline to memory-enhanced and LoRA fine-tuned models. The engine handles phases, voting, and role reveals, while the agents produce actions and dialogue within those rules.
The reasoning modules track claims and accusations, update beliefs about player roles, and check for contradictions. A separate evaluation pipeline records persuasion, consensus, and rule compliance, allowing comparisons beyond the final winner.
What the experiments showed
The project report covers 499 games across five configurations. These are results within this game and evaluation setup, not measures of general reasoning ability.
Villager win rate stayed at 99–100% across all configurations, including random agents. That made it unhelpful for distinguishing agent quality. Persuasion and consensus exposed differences that the outcome metric concealed.
Persuasion score
Group consensus
A more reproducible successor
In NewMafia, I separated the game engine from the agent infrastructure and made capabilities such as memory and reflection independently configurable. Seeded runs, experiment manifests, and tests for state and voting make it easier to repeat a run and isolate what changed.