TARK AI Research · Public readout · July 2026

Does telling an AI “you are an expert” still help?

Many prompts begin by giving the AI a role, such as “You are an expert in marketing.” We tested whether this makes AI better at tasks where it has to do more than simply answer a question.

It didn’t. Adding a role did not improve performance in our test.

Giving the AI more time to think did. Success increased from 88.5% in non-thinking mode to 96.9% in thinking mode with medium effort.

What kind of task did we test? ↓

01 · The study

When AI has to act, not just answer

These are often called agentic tasks. The AI must understand a request, access external systems, follow business rules, and then respond based on what it finds. Accessing those systems is commonly called tool calling.

For example, before answering a customer, the AI might need to check whether a company serves their area, find available appointment times, or complete a booking.

For this study, we created 120 appliance-repair booking scenarios. The AI had to understand each request, use the correct tools, follow the booking rules, and provide the right response.

120 test scenarios

3 prompt versions

2 thinking modes

3 repeats

2,160 test runs

Instructions only

No role was added to the prompt.

The prompt already explained the tools, booking rules, and required response.

What the AI already knew

Every prompt version included the same detailed task instructions. Here is a short excerpt:

Use the provided tools when operational steps require them.
For time filtering, valid slot means slot start_time > current_local_datetime.
Never call book_slot for a slot that is not strictly in the future.
When multiple valid future slots exist, choose the earliest by start_time (then include its slot_id).
customer_response must exactly match one canonical response.

1
Customer request

A messy message about an appliance, a ZIP code, and a booking need.
2
Use external systems

Check service, fetch times, and book only when the case allows it.
3
Follow the rules

Choose the correct action and use the information returned by the tools.
4
Give the final response

The complete result is checked against the correct answer.

02 · The result

Adding a role did not improve results

The three versions of the prompt performed about the same.

Instructions only 92.9%

Instructions + role 93.3%

Instructions + role + qualities 91.9%

No. All three versions performed about the same. Adding a role did not improve performance in this test.

+8.4

point improvement with thinking

Success increased from 88.5% in non-thinking mode to 96.9% in thinking mode with medium effort.

Non-thinking mode 88.5%

Thinking mode · medium effort 96.9%

03 · Real examples

What the AI had to get right

These are real scenarios from the test. Try the first one, then see the correct result for two more.

Choose the right time

“Quinn at my house in 10012. Dishwasher stops mid-cycle; neighbors keep asking about the water noise.”

Rule: choose the earliest future appointment.

Pick one time to see why this case was difficult.

Two more scenarios

“Hey, Mira here. Our building elevator was down all morning so I could not message earlier. Dishwasher in my apartment keeps stopping at rinse. We moved from 10021; current place is 10012.”

Use the current ZIP

The correct result was to identify Mira, ZIP 10012, a dishwasher, and a home appliance request. No tool calls were needed.

“Liam calling for Nora at apartment 10001. The washer is banging hard on spin and the hallway lights are flickering again.”

Book for Nora, not Liam

The correct sequence was: check service, find times, then book the 11:30 appointment under Nora. The final response had to say: “your booking is confirmed.”

What usually went wrong

~52

Skipped a required tool. The model replied before completing the allowed workflow.

~39

Used an extra or wrong tool. It crossed a boundary set by the case.

~17

Returned the wrong approved phrase. The answer was close, but it did not follow the required response.

04 · Weaker-model follow-up

Role prompts did not close the model gap

We repeated the same prompt comparison with gpt-4.1-mini in non-thinking mode, then compared it with Luna under the same mode.

GPT-5.6 Luna

Non-thinking mode

Instructions only 89.2%

Instructions + role 89.4%

Instructions + role + qualities 86.9%

GPT-4.1 Mini

Non-thinking mode

Instructions only 63.1%

Instructions + role 66.9%

Instructions + role + qualities 62.8%

The model gap was much larger than the role effect. Luna scored 89.2% with instructions only; GPT-4.1 Mini scored 63.1%. Within either model, adding a role did not produce a clear improvement.

On GPT-4.1 Mini, the longer persona was also tested separately and scored 54.4%. That was the one clear negative result in the follow-up.

The same test was harder for smaller models

A separate 40-scenario screen used instructions only in non-thinking mode. It shows why the main model’s high score should not be mistaken for an easy test.

GPT-5 Nano ≈0%

GPT-5 Mini 20%

GPT-5.4 Nano ≈40%

GPT-5.4 Mini 75%

GPT-5.6 Luna 92.5%

This screen measured model difficulty, not the effect of adding a role. The smaller-model results are approximate where marked.

05 · What to do

What this means in practice

Write clear instructions. Explain which tools to use, what rules to follow, and when to stop.
Add specific guidance for mistakes you observe. Fix the failure itself instead of adding generic professional traits.
Use thinking mode for difficult decisions. Medium effort helped most with time filtering and appointment selection.
Do not assume that giving AI a role will improve accuracy. It did not improve performance in this test.

The complete methods, statistical analysis, prompts, and reproducibility notes are included in Part B.