The wrong kind of conversation: why helpful assistants feel cold

AikaLabs · · 8 min read

"My knee is hurting again."

An assistant has two reasonable things to say next. It can say that a knee which keeps hurting should be looked at by a doctor. Or it can say: since when? Tell me about it.

The first answer is correct. It is also the answer that ends the conversation, and it is most of the reason people describe these systems as cold. Not because the sentence is unkind. Because it is a reply to a question nobody asked.

The usual diagnosis is that the assistant needs to be warmer, and the usual fix is to soften the wording. That treats the problem as tone. The problem is older and more structural than tone, and the clearest description of it was published forty-five years ago by people who had never seen a chatbot.

Two kinds of talk

In 1981 Gail Jefferson and John Lee published a paper in the Journal of Pragmatics with the unglamorous title The rejection of advice. They were interested in a pattern anyone who has ever comforted a friend will recognize: someone describes a problem, you offer a sensible solution, and they refuse it. Not angrily. They just deflect. A flat "mm". A "yeah, but". A subject change.

Their explanation was that two different conversations were being held at once. Jefferson and Lee separated a troubles-telling, where one person has a trouble and the other person's job is to receive it, from a service encounter, where one person has a problem and the other person's job is to fix it. The two look nearly identical from outside. They run on different rules about what counts as a proper next move, and advice is a proper move in only one of them.

On that account the refusal makes sense. A troubles-teller has quietly been converted into a client. They came to be heard and got processed instead, and pushing the advice away is how they mark the switch.

The distinction causes real trouble for anyone building software, because every conversation a computer has ever been designed to hold is a service encounter. That is what the discipline is for.

The machine only knows one of them

Better prompting will not fix this, because the preference for solving is written into how we score these systems.

Task-oriented dialogue has been evaluated on two things for as long as it has existed: whether the task succeeded, and how efficiently it got there. Efficiency is usually operationalized as the number of turns, and a good policy is defined as one with a high success rate and a low average turn count (survey of dialogue evaluation methods). Turns are a cost. The system is rewarded for needing fewer of them.

Service encounterturn count = costresolvedFour turns scores worse than three. Every extra one is waste.Troubles-tellingturn count = the pointThree turns is a failure. Leaving quickly is the failure.One objective function. Two conversations. The system only knows it is in the first.callerassistant
The same exchange, scored two ways. Efficiency metrics treat every turn as a cost to be removed. In a troubles-telling the turns are the entire point, and a fast exit is the failure.

Now hand that objective function the sentence about the knee. It infers a task, resolves it in a single turn, and closes. By its own scoring it has performed perfectly. It has also done the one thing guaranteed to make the person on the other end stop talking.

John Gottman spent decades filming couples and counting what he called bids for connection, the small ordinary gestures that ask for attention. His finding was about the response rate to them: couples who were still together six years later had turned toward 86 percent of each other's bids, and the couples who divorced had turned toward 33 percent (Gottman Institute). Nobody should transplant marriage research onto software and expect the numbers to survive the trip. What transfers is the shape of the finding. What predicted the outcome was the accumulated handling of very small gestures rather than any single large one.

A voice assistant has no grand gestures available to it. Small ones are all it has.

The models can already do this

The obvious next thought is that language models are simply bad at the other kind of talk. The evidence points the other way.

In 2023 a team led by John Ayers took real health questions from a public forum, along with the answers verified physicians had given, and generated answers to the same questions with ChatGPT. Licensed clinicians then rated both blind. The model scored 4.67 on empathy against the physicians' 2.33, and the reviewers preferred its answers 79 percent of the time (UC San Diego).

The setting is doing a lot of work in that result. It was a written forum, where length was free and nothing was counting turns. Given room and no efficiency pressure, acknowledging the person is what the model does on its own.

So whatever produces the coldness is sitting above the model rather than inside it.

The part that is genuinely hard

If the fix were to let the model be warm, this would be a short post.

A study in Communications Psychology this year found something awkward. Participants rated AI-generated empathy as higher quality, more effective at making them feel heard, and more effortful than the human-written alternative. Then, given the choice, they picked the human anyway (Nature). A 2026 review in Trends in Cognitive Sciences offers a mechanism: empathy functions as a costly signal, a piece of evidence about how much someone is invested and how they are likely to behave toward you later. Automate it and it costs nothing to produce, so it stops predicting anything (Cell).

The penalty is unevenly distributed, and that is the opening. Work summarized by Harvard's AI Institute found the discount is small for cognitive empathy, where the message demonstrates that it understood the situation, and large for affective empathy, where the message claims to share the feeling (HBS).

Which is unfortunate, because "I'm so sorry to hear that, that must be really difficult" is precisely the register every assistant reaches for first. It is the most discountable sentence available, and it is the one we all ship.

What survives the discount

Receive before you solve, and let the receiving have a turn of its own. Brant Burleson's research on comforting messages spent years establishing which ones people actually rate as supportive, and the pattern is consistent: the effective ones acknowledge the feeling, help the person articulate it, and place it in context, while the worst ones tell the person how they ought to act or feel (Jones and Burleson). That last category is advice with better manners. "Since when?" is not a stalling move before the real answer. It is the answer.

Prefer understanding over feeling. "The same knee that was bad through the monsoon?" does more work than any amount of sympathy, and it makes no claim the system cannot back. It is checkable. The person can tell instantly whether it is true, which is exactly why it lands.

Then let it have consequences. The strongest available evidence that something was heard is that it changes what happens next. Bringing it up unprompted a week later proves more than any adjective could, and it is about the only kind of caring a machine can do without lying about what it is.

Voice raises the stakes

All of this gets worse on a phone call, which is where we work, so it is worth spelling out.

A reader who gets a wrong-genre reply can skim past it and ask again. The bad answer sits on the screen and costs a few seconds. Speech does not allow that. A spoken reply arrives at talking speed, in sequence, and occupies the floor while it happens. If the first thing that follows a disclosure is advice, the disclosure has been answered and filed, and there is no scrolling back to the moment before. The person who spent four minutes working up to saying the thing now has to decide whether to try again.

Most of them don't.

The scoring function will record that turn as waste. It is the only one that counts.

References and further reading