AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace
Shiqi Chen, Daniel Madrigal Diaz, Wei-Ning Chen, Gbola Afonja et al. · Microsoft / arXiv · 9 Nov 2026
Source finding
Of 4,943 executions of previously discovered tasks, 49.8 percent received full credit. The remaining 50.2 percent included clarification requests, incomplete content, wrong targets and other deviations.
Our interpretation
Recognising tasks and executing them correctly require separate tests. A list of discovered work does not establish a completed result.
Method and evidence quality
One simulated organisation with 44 employees over six months, 94 actor-days and 120 tasks. Seventeen models ran in the same agent three times each. Execution was graded against actual changes and delivered content. Medium evidence for this test.
Limitations and interests
One synthetic organisation, not a field test. The 4,943 executions repeat tasks across models and runs. Not a general error rate; clarification can be sensible in real work. Unreviewed preprint by Microsoft researchers with a vendor interest.
What businesses can do
Test separately which tasks the agent finds and what it actually changes. Check recipients, records and content before granting it more permissions.
Where to start
Section 3.2 on grading, 5.1 on setup and 5.3 with Figure 4b on execution failures.