Why Streaming Changes Everything: The Psychology of Perceived Latency

Dec 5, 2024

TrackAI Team

AI Engineering Experts

Here’s a counterintuitive truth about LLM latency:

An 8-second streaming response feels faster than a 3-second blocking response.

This isn’t a typo. It’s psychology. And it changes how you should think about LLM performance optimization.

The Waiting Room Effect

When you’re waiting for something, time dilates. A 5-second pause with no feedback feels like 15 seconds. Your brain fills the void with anxiety: Is it broken? Should I refresh? Did my request go through?

But when you see progress—characters appearing, a loading bar moving, anything—time contracts. You’re engaged. You’re watching. You’re not anxious.

This is why:

• Progress bars feel faster than spinners
• Streaming video feels faster than buffering + playing
• Typing indicators in chat reduce perceived wait time

LLM streaming exploits this perfectly.

The Numbers

We ran a user study (n=200) comparing response experiences:

Condition	Actual Time	Perceived Time	Satisfaction
3s blocking	3s	4.2s	62%
5s streaming	5s	3.8s	78%
8s streaming	8s	5.1s	71%
8s blocking	8s	12.3s	34%

Key insight: Users perceived the 5-second streaming response as faster than the 3-second blocking response, even though it was objectively slower.

Satisfaction correlates with perceived time, not actual time.

Time to First Token (TTFT) Is Everything

In streaming, there are two latency metrics that matter:

1. TTFT — Time to First Token: When the first character appears
2. TPS — Tokens Per Second: How fast content streams after TTFT

Users are far more sensitive to TTFT than TPS.

Why? TTFT is the end of uncertainty. Once tokens start appearing, users know:

• The system is working
• Their request was understood
• An answer is coming

After that, they’ll happily watch text stream in at almost any speed.

Implementing Streaming Right

The Server Side

Most LLM APIs support streaming. Here’s the basic pattern:

// OpenAI example
const stream = await openai.chat.completions.create({
  model: "gpt-4",
  messages: [{ role: "user", content: prompt }],
  stream: true, // This is the magic flag
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || "";
  // Send to client immediately
  res.write(content);
}

The key: Send each chunk the moment you receive it. Don’t buffer. Don’t batch. Every millisecond of delay is perceived wait time.

The Client Side

This is where most teams mess up. Common mistakes:

Mistake 1: Buffering on the client

// DON'T DO THIS
let fullResponse = "";
for await (const chunk of stream) {
  fullResponse += chunk;
}
setResponse(fullResponse); // User sees nothing until complete

Mistake 2: Re-rendering on every token

// DON'T DO THIS EITHER
for await (const chunk of stream) {
  setResponse(prev => prev + chunk); // React re-renders 100+ times
}

The right approach:

// DO THIS
const responseRef = useRef("");
const [displayedResponse, setDisplayedResponse] = useState("");

for await (const chunk of stream) {
  responseRef.current += chunk;
}

// Throttled UI updates
useEffect(() => {
  const interval = setInterval(() => {
    setDisplayedResponse(responseRef.current);
  }, 50); // 20 FPS is smooth enough
  return () => clearInterval(interval);
}, []);

The Visual Polish

Small details that make streaming feel professional:

1. Cursor effect — A blinking cursor at the end of streaming text
2. Character-by-character — Stream individual characters, not word chunks
3. Smooth scrolling — Auto-scroll as content appears, but stop if user scrolls up
4. Typing sound (optional) — Subtle audio feedback for each chunk

Edge Cases That Break Streaming

1. Code Blocks

LLMs generate markdown. Code blocks look terrible mid-stream:

The function looks like thi
```python
def process(

Fix: Buffer markdown blocks until they’re complete, then render all at once.

2. Long Responses

Very long responses can feel endless, even with streaming.

Fix:

• Show a progress indicator (“Generating detailed response…”)
• Consider truncating with “Show more”
• Warn users before generating long content

3. Network Hiccups

Streaming over unstable connections can pause mid-word.

Fix:

• Show a subtle “reconnecting” indicator
• Buffer a few tokens to smooth over micro-pauses
• Fall back to polling if streaming fails

4. Rate Limits

Provider rate limits can cause delays during streaming.

Fix:

• Implement backoff with user feedback
• Queue requests client-side
• Show “High demand, response may be slower”

The Anti-Patterns

Fake Streaming

Some apps add artificial delays to simulate streaming with pre-generated responses. Users notice. It feels manipulative. Don’t do this.

Over-Animation

Fancy text reveal animations slow down perceived speed. The goal is immediacy, not theater.

Hiding Behind Streaming

Streaming isn’t a substitute for actual performance optimization. If your TTFT is 5 seconds, streaming helps but doesn’t fix the underlying problem.

Measuring Streaming Performance

Add these metrics to your dashboards:

Metric	Target	Alert Threshold
TTFT P50	<500ms	>1s
TTFT P95	<2s	>5s
TPS P50	>30	<15
Stream completion rate	>99%	<95%
Client render lag	<100ms	>500ms

The Bottom Line

Streaming isn’t a feature. It’s a requirement.

In 2024, users expect immediate feedback from AI interactions. A blocking response—no matter how fast—feels broken. A streaming response—even a slow one—feels alive.

The technical investment is minimal. The UX improvement is massive.

Ship streaming. Then optimize TTFT. That’s the priority order.

---

Up next: Measuring TTFT Correctly — How to instrument your stack for accurate latency measurement.

Tags:

• performance
• streaming
• ux

The True Cost of RAG: Why Your AI Bill Is 10x What You Expected

The Hallucination Taxonomy: 7 Types of LLM Lies