Most agile practitioners use AI at work. Most think they understand it well enough to guide their teams. The data suggests otherwise.
A new assessment from AI4Agile, made freely available to raise baseline competency across the profession, reveals a troubling pattern: 43% of experienced practitioners failed the test, even after retakes.
The short version
- First cohort: 30 participants, all with context and motivation
- Average score: 75%, but pass rate only 57%
- Wide range: scores from 23% to 93%
- Risks and ethics section had the lowest pass rate
- Assessment is free to address a systemic confidence problem
The Numbers That Should Worry You
AI4Agile released results from their initial cohort of 30 assessment takers. These weren't cold attempts by strangers. They came with context, motivation, and time to prepare. They were already using AI in their work.
Average score landed at 75%. Pass rate came in at 57%. That means nearly half failed, even after some retook it.
The range is what gets me. Scores from 23% to 93% span almost the entire possible range. Seniority didn't predict performance. Fresh perspective didn't predict performance. Something about the assessment catches people regardless of background.
The data comes via age-of-product.com, which hosts the assessment and published the full results.
Why Confidence Outpaces Competence
Here's the uncomfortable part. These practitioners aren't avoiding AI. They're using it confidently. They write prompts, get structured responses, and move forward. The problem is that confidence and competence aren't the same thing here.
AI produces output that looks polished. It structures responses like an expert would. It uses confident language. These cues bypass our skepticism and trigger "this is good" instincts—even when the reasoning underneath is thin.
Think about how you review AI output in your sprint retrospectives or when drafting stakeholder updates. How often do you catch the flaws versus accepting the framing?
Pattern matching looks like analysis when you aren't specifically watching for the difference. Confident wrong answers look like confident right answers. Without practice testing where your judgment is reliable, you won't notice when AI leads you astray.
Where the Gaps Hurt Most
The risks, ethics, and organizational change section has the lowest pass rate. This area tests whether practitioners can spot where the obvious AI decision is actually wrong—accounting for privacy implications, liability concerns, and the organizational dynamics that kill AI adoption.
Most AI tutorials skip these topics entirely. Most practitioners learn AI by writing prompts and iterating. Nobody teaches the failure modes.
This creates a blind spot that matters. When a Scrum Master suggests using AI to analyze customer feedback data, they need to know whether the output represents real insight or sophisticated pattern matching. When a product manager uses AI to draft stakeholder communications, they need to know when the output contains confident nonsense.
The people advising teams on AI adoption often have the least visibility into these specific failure modes.
A Practical Scenario
Say you're a Scrum Master running a retrospective. Your team used a new tool this sprint, and you want to understand why velocity dropped. You paste three months of ticket data into an AI tool and ask for patterns.
The AI returns a clean list: "communication gaps on Thursdays," "overlapping dependencies," "unclear acceptance criteria." It looks useful. Your instinct says this is good analysis.
But what did it actually do? It found words that appear together frequently. It didn't verify causality. It didn't check whether those patterns existed before the new tool. It didn't account for the fact that your team changed priorities mid-sprint for unrelated reasons.
A practitioner with calibrated AI judgment would push back on this output. They'd ask what the tool actually analyzed versus what it assumed. They'd recognize that frequency of terms isn't the same as explanation of causes.
The assessment tests exactly this kind of critical evaluation.
Why Free Access Matters
The assessment remains free. The creator's position: the profession has a false confidence problem that affects team decisions every day. Too many practitioners advise on AI adoption without a reliable baseline of what good judgment looks like.
Raising the floor across the profession matters more than restricting access behind a paywall. If you fail, you get something more useful than a certificate—a clear signal about where your judgment needs work before you confidently advise others.
This isn't a framework certification. It's a practical benchmark for using AI in agile work.
Bottom Line
If you're coaching teams on AI adoption, running sprints, or making product decisions, you need to know where your judgment is reliable and where it's not. The assessment takes about 40 minutes. The cost is nothing. The signal it provides is worth the investment.
Watch whether organizations begin treating AI judgment validation like Scrum certification—required evidence of competency before advising teams. That shift would validate what these early numbers suggest: current confidence levels aren't matching the stakes.
Key takeaways
- 43% of experienced agile practitioners fail AI judgment assessments
- Casual AI use builds confidence faster than calibration
- Risks, ethics, and organizational dynamics have the lowest pass rate
- Free access aims to raise the profession's baseline
- Wide score range suggests no reliable predictor of performance
Frequently asked questions
What's the pass threshold for the AI4Agile assessment?
The assessment targets 80% as the pass threshold, though the actual first-attempt pass rate in the initial cohort was 57%.
Who should take this assessment?
Any agile coach, product manager, Scrum Master, or delivery professional who uses AI at work or advises teams on AI adoption.
What does the assessment actually test?
It tests practical judgment: distinguishing reliable AI output from confident noise, diagnosing why prompts produce generic results, using AI for data analysis without fooling yourself, and recognizing organizational risks in AI deployment.
Why do risks and ethics questions have the lowest pass rate?
These questions require understanding privacy implications, liability concerns, and organizational dynamics that kill AI adoption—topics most AI tutorials and training programs don't cover.
How long does the assessment take?
Approximately 40 minutes to complete the 40 scenario-based questions.