TARK AI Research · Public readout · August 2026

Does telling an AI "you are an expert" still help?

Many prompts open with a role, like "You are an expert in marketing." We tested whether that helps when the model has to use tools and follow rules.

Paper · Code

It didn't. Adding a role did not improve performance in our test.

Giving the AI more time to think did. Success went from 88.5% in non-thinking mode to 96.9% in thinking mode with medium effort.

What did we test? ↓

01 · The study

When the AI has to do the work

We tested jobs where the model has to act. People call these agentic tasks. The model reads the request, uses tools, follows the booking rules, and answers from what the tools returned. Using a tool is tool calling. That's how the model queries another system.

A booking agent is a clear case. Before it can reply, it may need to check whether the company serves that ZIP, pull open appointment times, or book a slot.

We wrote 120 appliance-repair booking scenarios. For each one, the model had to read the request, use the right tools, follow the rules, and give the required reply.

120 test scenarios

3 prompt versions

2 thinking modes

3 repeats

2,160 test runs

Instructions only

No role was added to the prompt.

The prompt already explained the tools, booking rules, and required response.

What the AI already knew

Every prompt version used the same task instructions. A short excerpt:

Use the provided tools when operational steps require them.
For time filtering, valid slot means slot start_time > current_local_datetime.
Never call book_slot for a slot that is not strictly in the future.
When multiple valid future slots exist, choose the earliest by start_time (then include its slot_id).
customer_response must exactly match one canonical response.

1
Customer request

A messy message about an appliance, a ZIP code, and a booking need.
2
Call the tools

Check coverage, fetch times, and book only when the case allows it.
3
Follow the rules

Pick the right action and use what the tools returned.
4
Give the final response

We scored the full result against the correct answer.

02 · The result

Adding a role did not improve results

The three prompt versions scored about the same.

Instructions only 92.9%

Instructions + role 93.3%

Instructions + role + qualities 91.9%

No. All three versions scored about the same. Adding a role did not help in this test.

+8.4

point improvement with thinking

Success went from 88.5% in non-thinking mode to 96.9% in thinking mode with medium effort.

Non-thinking mode 88.5%

Thinking mode · medium effort 96.9%

03 · Real examples

What the AI had to get right

Three cases from the test. Try the first one, then read the right answer for the other two.

Choose the right time

"Quinn at my house in 10012. Dishwasher stops mid-cycle; neighbors keep asking about the water noise."

Rule: choose the earliest future appointment.

Pick one time to see why this case was hard.

Two more cases

"Hey, Mira here. Our building elevator was down all morning so I could not message earlier. Dishwasher in my apartment keeps stopping at rinse. We moved from 10021; current place is 10012."

Use the current ZIP

The right result was to identify Mira, ZIP 10012, a dishwasher, and a home appliance request. No tool calls were needed.

"Liam calling for Nora at apartment 10001. The washer is banging hard on spin and the hallway lights are flickering again."

Book for Nora, not Liam

The right sequence was check service, find times, then book the 11:30 appointment under Nora. The final response had to say "your booking is confirmed."

What usually went wrong

~52

Skipped a required tool. The model answered before finishing the allowed workflow.

~39

Used an extra or wrong tool. It crossed a boundary set by the case.

~17

Returned the wrong approved phrase. Close, but it did not match the required response.

04 · Weaker-model follow-up

Role prompts did not close the model gap

We ran the same prompt comparison on gpt-4.1-mini in non-thinking mode, then compared it with Luna under the same mode.

GPT-5.6 Luna

Non-thinking mode

Instructions only 89.2%

Instructions + role 89.4%

Instructions + role + qualities 86.9%

GPT-4.1 Mini

Non-thinking mode

Instructions only 63.1%

Instructions + role 66.9%

Instructions + role + qualities 62.8%

The model gap was much larger than the role effect. Luna scored 89.2% with instructions only. GPT-4.1 Mini scored 63.1%. On either model, adding a role did not clearly help.

We also tried a longer persona on GPT-4.1 Mini, separately. It scored 54.4%, the one clear drop in the follow-up.

The same test was harder for smaller models

A separate 40-scenario screen used instructions only, with thinking off. Luna's high score was not because the test was easy.

GPT-5 Nano ≈0%

GPT-5 Mini 20%

GPT-5.4 Nano ≈40%

GPT-5.4 Mini 75%

GPT-5.6 Luna 92.5%

This screen measured how hard the test is for each model, not whether a role helps. Results marked ≈ are approximate.

05 · What to do

What to change in the prompt

Write clear instructions. Say which tools to use, which rules to follow, and when to stop.
Patch the mistakes you actually see. Fix the failure. Don't add generic professional traits.
Use thinking mode when the decision is hard. Medium effort helped most with time filtering and picking the appointment.
Don't assume a role improves accuracy. It didn't in this test.

Methods, stats, prompts, and reproducibility notes are in Part B. Paper · Code