The short answer
Manual QA usually scores a handful of calls per agent each month, which is well under 1% of what they handle, so most breaches and coaching moments are never seen. Scoring every call means automating the scoring itself: transcribe each call, score it on the QA form your team already uses, link each score to its moment in the call, and keep people for calibration, disputes and coaching.
Why a sample misses most problems
Say an agent handles 50 calls a day. Over 22 working days that is 1,100 calls a month. If QA scores four of them, it has looked at 0.36% of the agent's work.
Coverage is only half the problem. The other half is detection. Suppose a required disclosure is skipped on 1 call in 100. The chance that a random sample of four calls includes even one of those misses is 1 − 0.99⁴, about 3.9%. Most months, the scorecard says everything is fine.
| Calls scored per agent a month | Share of a 1,100-call month | Chance of seeing a 1-in-100 miss |
|---|---|---|
| 4 | 0.36% | 3.9% |
| 10 | 0.9% | 9.6% |
| 50 | 4.5% | 39.5% |
| Every eligible call | 100% | Every miss is on a scored call |
Samples are rarely random, either. Reviewers pick calls that are easy to find: long calls, escalations, calls a team lead flagged. Agents notice which calls get scored, and so do clients.
What changes when every call is scored
- Compliance becomes a measured number. Instead of "no misses found in the sample", you can say what share of eligible calls had each required disclosure.
- Coaching targets real patterns. One weak call is noise. The same missed step on 30% of an agent's calls is a coaching plan.
- Scores feel fairer to agents. A score built on hundreds of calls is harder to argue with than one built on four, and one bad call no longer defines the month.
- Clients see evidence, not anecdotes. Coverage you can show is easier to defend at a business review than a sample you have to explain.
How to get to 100% coverage, step by step
Start from the QA form you already use
Don't invent a new scorecard. Take the form your QA team scores against today, or your client's form, and sort its items into two groups: those that can be checked from what is said on the call (a disclosure, a verification step, an offer of a payment plan) and those that need human judgement (tone, empathy, whether the resolution was right).
Define which calls are eligible
Exclude what should not be scored: wrong numbers, calls that dropped in the first seconds, calls where the customer declined recording, internal transfers. Write the rule down so the coverage number means the same thing every month.
Turn each item into a check you can verify
"Agent verified identity" becomes "agent confirmed two of date of birth, postcode and account number before discussing the account". Vague items produce vague scores, whether a person or software is doing the scoring.
Link every score to evidence
Each item should point to the moment in the call that earned or lost the point. Without that, a dispute turns into an argument about memory.
Calibrate before you publish
Have QA leads score a set of calls by hand and compare with the automated scores, item by item. Tighten definitions where they disagree. Repeat on a fresh set until agreement is consistently high, then keep spot-checking every week.
Show agents first, then teams, then clients
Let agents see their own scores and dispute them before scores reach team leads or clients. Upheld disputes are the best source of fixes for your item definitions.
Keep people where judgement matters
Automated scoring should free QA analysts from listening for checklist items, so they can spend their time on calibration, disputes, the hardest calls and coaching.
Pitfalls to avoid
- Scoring against a generic form. A vendor's default scorecard will not match what your client audits you on. Score on the client's own form.
- Publishing scores without evidence. If agents cannot see why they lost a point, the score will not change what they do.
- Treating every item as automatable. Items that need judgement should stay with people, or be flagged for review rather than scored.
- Ignoring data handling. Scoring every call means transcribing every call. Decide up front where recordings and transcripts are kept, who can open them, what is masked and when they are deleted. See keeping card numbers out of recordings.
- Changing what counts as eligible mid-quarter. Coverage numbers only compare if the rule stays the same.
What to measure once it is running
| Measure | What it tells you |
|---|---|
| Coverage | Share of eligible calls that were scored. The target is 100%, and any gap should have a known reason. |
| Item pass rate | For each QA item, the share of calls that passed. This is where compliance gaps show up. |
| Agreement with QA leads | How often calibration reviewers agree with the automated score on the same item. Track it weekly. |
| Disputes | How often agents dispute, and how often they are right. A high upheld rate means an item definition needs work. |
| Time to feedback | Time between the call and the agent seeing the score. Aim for days, not weeks. |
Where Orian fits
How Orian scores every call
- Every eligible call is scored on your client's own QA form, imported from PDF, Excel or Word.
- Each score item links to its moment in the recording. Agents see their own scores first and can dispute them.
- Each client's QA items map to one in-house standard, so programmes compare fairly.
- Recordings, transcripts and scores stay inside your network. Sensitive details are masked in the transcript, and card, bank and ID numbers are muted in the recording.