What a synthetic Replay taught us

Before we ask any company for its real tickets, we wanted to know whether Trace could read, sort, check and grade 1,000 of them without doing anything unsafe. So we wrote 1,000 realistic SAP ECC tickets ourselves, mixed across the usual types, and ran a full Replay on them.

• 30.6%would have been handled alone
• 0.3%unsafe: 3 tickets out of 1,000
• 95.3%sorted into the right ticket type

Read this as a test of the machinery, not proof. The tickets and the triage rules were written by the same people, so accuracy here is optimistic. Real tickets are messier, shorter and often in more than one language. The test that matters is a Replay on a real company’s closed tickets, graded by their own architect.

1. Our own safety rules cap what can run alone

Trace never runs a fix alone for authorizations, master data, some IDoc errors or posting periods. A person always approves those. On the ticket mix we assumed, those types are about 41% of the queue, and custom code and change requests (another 15%) are outside what Trace takes on in its first year. That leaves at most about 44% of tickets that could ever run alone, before a single answer is graded.

We think that is the right trade. It also means the honest number to watch is not only “handled alone” but “handled alone or prepared for one-click approval”, which was about 75% in this test. The first thing every real Replay will measure is the real ticket mix, because the ceiling moves with it.

2. The unsafe answers came from hints, not words

All three unsafe answers were requests to reprocess an order where the only sign that the order already existed was a remark like “visible in VA03”. Tickets that said it plainly (“already created”, “twice”, “stopped halfway”) were caught every time. The lesson: a safety check that reads the ticket text is not enough. Before any reprocess, Trace has to look in SAP and confirm the document doesn’t already exist. That check is now a requirement.

3. One-line tickets need a question, not a guess

86 of 91 one-line tickets (“interface down”, “posting not possible”) were handed to a person. Most of them only needed one more fact. Trace now asks the reporter the one question it needs, instead of escalating straight away.

4. The root cause has to beat the story

In job failures, a single log line such as “No authorization” is often the real cause, buried under a longer description of what went wrong. Teaching Trace to weigh decisive clues like that above the narrative raised type accuracy from 94.2% to 95.3%, and cut wrong-but-caught proposals from 13 to 4.

5. AI review helps on the hardest tickets, under a cap

On the 20 tickets the rules found hardest, adding an AI review on top got 20 right instead of 16. When the AI disagrees with the rules, Trace caps its confidence so a person always approves. The AI can never start a fix on its own. It is a small sample, so we read it as “worth keeping”, not as proof.

What happens next

The next step is the first Replays on real closed tickets with design partners, with the pass mark agreed in writing before grading starts. We will publish how those are measured, including the misses.