Example scenario. Two managers hear the same sales conversation and give different scores. One values that the adviser asked three questions. The other notices that the adviser never checked the answer. Before the team writes feedback, it needs to define what it is rating.
From research to rubric design
The 2026 JMIR simulated patient study identifies expert review of scenario fidelity as a priority for future work. Apply a comparable quality check in a workplace pilot by asking experienced reviewers to assess sample conversations.
A workplace example
Define three levels for needs discovery. At the initial level the employee recommends a solution based on assumptions. At the developing level they ask a question but omit a summary. At the expected level they ask an open question, summarise the response and check their understanding.
What decision should the rubric support?
A sales adviser asks three questions and quickly offers a proposal. One manager considers that effective discovery. Another points out that the adviser never checked what the customer meant. The managers are rating different behaviours, which makes their scores hard to compare.
Start with the purpose of assessment. A few behaviours may be enough to give useful feedback before the next attempt. If scores will be compared across the organisation, check how different reviewers rate the same examples.
Sample scoring rubric
| Criterion | 1 · Initial | 2 · Developing | 3 · Expected |
|---|---|---|---|
| Open question | Starts with a proposed solution | Asks a general question | Follows up using details from the context |
| Checks understanding | Does not confirm the answer | Summarises it | Summarises and asks for confirmation |
| Next step | Leaves it unclear | Suggests an action | Specifies action, owner and timing |
- Define a behaviour
- Write an example at each level
- Score the same conversation independently
- Update disputed criteria
Turn a score into useful feedback
“Level 2” offers little direction by itself. Give the learner an example from the conversation. “Your two questions surfaced the need. Before proposing an option, try summarising the customer’s priority in one sentence and checking it with them.” The learner can act on this in the next conversation.
If scores differ, examine the wording first. Separate or prioritise behaviours when a single level bundles several together. Repeat calibration using pilot conversations. The JMIR example concerns clinical communication, so use reviewers from your own workplace to check whether the criteria fit the job.
Check agreement between reviewers
For calibration, select a weak, an intermediate and a strong conversation. Have two reviewers score them independently before discussing the criteria on which they disagree. One person may treat repeating the customer’s words as confirmation while another expects an explicit check. Add an example phrase to the rubric to clarify the distinction.
If the scenario does not give the learner a chance to demonstrate a behaviour, record it as unobserved. For example, the character may end the conversation before the employee can verify a key detail. Treating missing evidence as low performance can send the learner towards the wrong development activity.
Document what changes in the rubric
Use the first version on three sample conversations. For each disputed rating, record which words the reviewer used as evidence. If disagreements concentrate on “summarises the need”, add an example that meets the standard and one that falls just short. Ambiguity in the criterion may explain more than the learner’s ability.
Use the revised rubric on comparable fresh examples in the second pilot. Record the changed criteria before comparing scores across versions. This helps avoid interpreting a scoring change as a skill gain. If the scenario did not allow a behaviour to be observed, raise it separately in the manager’s feedback conversation.
How to apply it with COBIDU
Use these behaviours as scenario-specific criteria in COBIDU AI Simulation. Ask two managers to rate the same samples independently and compare their ratings with the simulation report. Clarify disputed criteria with examples. For consequential decisions, consider the score alongside human assessment.
Once one behaviour is clearly defined, the same approach can be applied to a role. The next article turns observable behaviours into a skills map.
