MC1518085 - Microsoft Copilot Studio - Improvements to agent Evaluations experience
Microsoft 365 Message Center announcement MC1518085.
MS Message Center's Summary
Microsoft is rolling out enhancements to the agent evaluations experience in Copilot Studio. The update brings richer outcome explanations, reasoning traces, run comparisons, and support for larger datasets up to one hundred test cases. Makers can also generate datasets directly from knowledge sources. This feature is available now for awareness with no mandatory administrator action required.
- Administrator impact
- No direct administrator action is required.
- End user impact
- Makers will experience improved troubleshooting tools and automated dataset generation capabilities when building and testing agents in Copilot Studio.
- Importance
- 2/10: The update introduces minor feature improvements to maker experiences with no administrative configuration or service disruption risk.
- Message Center ID
- MC1518085
- Category
- stay Informed
- Severity
- normal
- Services
- Microsoft Copilot (Power Platform)
- Tags
- New feature
- Published
- 2026-10-21
- Last updated
- 2026-10-21
- Expires
- 2026-12-21
We are enhancing the Evaluations experience to help makers better understand agent quality and behavior in Microsoft Copilot Studio. This feature is now available. How does this affect me? The improvements will bring new capabilities that help makers identify quality issues more quickly, understand why evaluations succeed or fail, compare results across runs, and create higher quality evaluation datasets with less manual effort. Additional validation and guidance will help makers resolve configuration issues before running evaluations. New capabilities include the following: Richer explanations of evaluation outcomes will help makers understand why evaluations succeed or fail. Reasoning traces and cited knowledge sources will help makers understand agent behavior and grounding. Evaluation run comparison will help identify improvements and regressions across runs. Evaluations will support datasets with up to 100 test cases. Makers will be able to use customizable dataset and question generation. Makers will be able to generate datasets from configured knowledge sources. Single-response datasets will be supported. Improved validation, connection prerequisite checks, credit-limit enforcement, and actionable error messages will help makers resolve configuration issues before running evaluations. These enhancements are designed to reduce manual effort, improve troubleshooting, and help makers evaluate and optimize agents more effectively. What action do I need to take? This message is for awareness, and no action is required.