When the Market Goes to Therapy

Session 1: The Intake

Meet the Patient

34% Drop in 23 days

33 Years of data

5,884 Stocks analyzed

Initial Assessment Notes

Date: March 2020. Presenting symptoms: Severe mood swings. Patient experienced a 34% decline in just 23 trading days. Investors reported extreme bearish sentiment. Google searches for "market crash" and "recession" spiked to unprecedented levels.

Clinical question: Did the collective anxiety cause the crash, predict it, or simply react to it?

The Market personified as a patient on a therapy couch — The Market: our most complex patient

They call us when nobody else can figure it out. We're empanADA: leading experts in behavioral psychology. But nothing prepared us for this patient. The Market doesn't talk. It doesn't sit on our couch and tell us about its childhood. Instead, it communicates through price movements, volatility spikes, and regime shifts. To understand it, we must observe the people in its life: the ones who interact with it daily and claim to know what it's thinking.

First of all, let's check the pulse! For the market, that pulse is its daily returns — a continuous signal that reflects stress, calm, and sudden shocks. Before interpreting sentiment or regimes, we listen to this heartbeat.

Market returns over time — Daily NASDAQ returns - The heart beat of the market
↺ Click to view calculation details

Next: The Patient's Support Network

We can't interview The Market directly. But we can study the two groups who spend the most time with it:

The Inner Circle

AAII (a.k.a. American Association of Individual Investors) Weekly Sentiment Survey

Active investors who interact with the patient every week. They report their expectations directly: "Where do you think the market will be in 6 months?"

Informed
Explicit predictions
Weekly frequency
Since 1987

The General Public

Google Trends Search Behavior

Society at large: their collective search behavior reveals unconscious fears and hopes about the economy. They don't know they're being observed.

Spontaneous
Implicit sentiment
Real-time signals
Massive scale

THE CENTRAL CLINICAL QUESTION

Do investors’ predictions pull the market in a certain direction, or are they largely reactions to market movements that have already begun?

Are the millions of people trying to read the market actually destabilizing it? Is the patient's support system… toxic?

Two Competing Hypotheses

Before we begin our analysis, we must acknowledge two fundamentally different ways this relationship could work:

Sentiment Has Power

Hypothesis A

Investors look ahead. Their collective expectations shape capital flows, which move prices. Beliefs become self-fulfilling prophecies.

Implication: Sentiment should lead market movements. We'd see investors turn bearish before crashes.

Sentiment Is Just a Mirror

Hypothesis B

Markets move because of fundamentals, shocks, and institutional flows. Retail investors simply update their views afterward, explaining what already happened.

Implication: Sentiment should lag market movements. Investors turn bearish after seeing losses.

Session Notes

Only one of these hypotheses can dominate in the data. Armed with decades of price history, sentiment surveys, and search data (plus statistical tools sharper than any scalpel) we're going in.

Let's meet the witnesses.

Session 2: The Witness Profiles

Two Perspectives on the Patient

Observation Method

To understand our patient, we interview two key witness groups separately. First, the Inner Circle (AAII survey respondents): investors who explicitly report their expectations. Then, the General Public, whose unconscious fears and hopes are revealed through Google search behavior.

Part A: The Inner Circle Speaks

Every week since 1987, members of the American Association of Individual Investors have answered one simple question:

AAII Weekly Sentiment Survey

"Where do you think the market will be in the next 6 months?"

Whatever your choice, you've just joined millions of investors who make this call regularly. Now let's see what the data reveals about their collective psychology.

Finding 1: A Baseline of Optimism

Distribution of investor sentiment across all observations (1987-2020)

Clinical Observation

Across 33 years of data, bullish sentiment is the baseline state, accounting for 37.6% of all observations. Neutral and bearish views each represent about 31%.

This asymmetry reveals something fundamental about investor psychology: optimism is the default. Even accounting for recessions, crashes, and bear markets, investors expect growth more often than decline.

This isn't irrational: markets have historically trended upward. But it raises a question: when investors turn bearish, is it because they're seeing something, or because they just saw something?

The violin plots tell us more. Bullish sentiment centers around 35-40% with occasional surges above 60%. It's symmetric: optimism can be extreme in either direction.Bearish sentiment is different. It clusters around 25-30% but shows a right skew. This indicates that deep pessimism is rare, appearing only in acute stress episodes.Neutral sentiment is the most stable, consistently hovering near 30%.

Distribution of sentiment levels showing median, spread, and outliers

Finding 2: Beliefs Are Sticky

Levels tell us what investors believe. But to understand behavior, we need to study transitions, how sentiment changes from week to week.

Sentiment state transition matrix — Week-to-week sentiment transition probabilities

Reading the Transition Matrix

Each cell shows the probability of moving from one sentiment state (row) to another (column) in a single week:

77% Bullish → Bullish Optimism is highly persistent

64% Bearish → Bearish Pessimism is sticky too, but less so

23% Bearish → Bullish Recovery from pessimism is relatively fast

11% Bullish → Bearish Falls into pessimism are rare

Key Insight: Sentiment states are highly persistent. When investors are optimistic, they tend to stay optimistic. But crucially, pessimism is less sticky than optimism: bad moods pass faster than good ones. This asymmetry will matter when we study market regimes.

Finding 3: Critical Moments: When Beliefs Shift Suddenly

To better understand how investor expectations evolve over time, we analyze the sudden week-to-week changes in Bullish, Neutral, and Bearish sentiment. Examining sudden shifts allows us to identify moments when investors rapidly changed their outlook. This helps us assess whether shifts in sentiment tend to anticipate market developments or react to them.

Top sudden sentiment shifts: critical moments in market psychology

The interactive time-series plots reveal that sentiment typically evolves smoothly from week to week, but sharp discontinuities do occur. These sudden increases correspond to moments when investors rapidly changed their expectations, likely due to major market events or shifts in uncertainty. The difference in the density and magnitude of spikes across the three sentiment states suggests that Bearish sentiment is the most sensitive to shocks.

Part B: The General Public Under the Microscope

Methodology Note

Survey data tells us what investors say they believe. But beliefs aren't always articulated. Sometimes they show up through what people search when worried, curious, or excited. Google Trends provides this unconscious pulse.

Search Query Classification

Every week, millions of searches reveal what market participants are paying attention to. Unlike surveys, these searches are spontaneous, unfiltered, and often emotional. We classified Google search queries into three sentiment categories based on their economic meaning:

Bullish Indicators

"Investment"
"Stock market"
"Exchange-traded fund"

Growth-oriented, opportunity-seeking

Neutral Indicators

"Dow Jones"
"Nasdaq"
"S&P 500"

Information-seeking, monitoring

Bearish Indicators

"Recession"
"Unemployment"
"Inflation"

Fear-driven, risk-focused

Bull vs Bear Google Trends word cloud — Word clouds showing the language of optimism versus pessimism in search behavior

Search Patterns Across Market Cycles

Dominant search sentiment over time: the crowd's unconscious mood

Pattern Observed

Bullish searches build gradually during expansions; people slowly become more interested in investing as confidence grows.

Bearish searches spike sharply during stress; fear triggers immediate information-seeking. The asymmetry mirrors what we saw in AAII sentiment.

Correlation: Do Search Behaviors Move Together?

Correlation within and between Google Trends search categories — Correlation matrices showing relationships within and between Google Trends search categories

Understanding the Correlations

These correlation matrices reveal how Google search terms cluster together within and across sentiment categories. Red cells indicate strong positive correlations, while blue cells show weak or negative correlations. Click on a category above to explore specific search behavior patterns.

Key Findings

Neutral (0.71–0.97): Index searches are highly unified—users checking one index check them all.

Bearish (0.01): Inflation and Unemployment searches are uncorrelated—anxiety is issue-specific, not generalized.

Bullish (0.65): "Stock Market" and "ETF" are linked, but "Investment" remains broader.

Cross-category (0.82): Neutral and Bullish overlap strongly; Bearish stands apart.

Market optimism is broad and interconnected; market anxiety is isolated and threat-specific. When people feel bullish, their searches span indices, ETFs, and investments in unison. When fear strikes, attention fragments—users fixate on either inflation or unemployment, rarely both. This asymmetry suggests that confidence is a rising tide, while panic is a series of isolated waves.

Session 2 Summary: What the Witnesses Told Us

1

Optimism is the baseline. Investors are bullish more often than bearish.

2

Beliefs are sticky. Once in a sentiment state, investors tend to stay there, but pessimism fades faster than optimism.

3

Fear arrives suddenly. Bearish sentiment shows the largest and fastest changes during market stress.

4

Two signals, same psychology. AAII surveys and Google searches capture different but related aspects of investor sentiment.

Next Session Preview

We've profiled the witnesses. Now comes the critical question: Who moves first? Does sentiment lead market movements, or does it simply follow them?

Time for the timeline analysis.

Session 3: The Timeline

Who Moves First?

Critical Session

Everything we've learned leads here. We have two witnesses (AAII and Google) and one patient (The Market). Now we determine the direction of causality: Does sentiment lead, follow, or move with the market?

The Initial Evidence: Correlation

We begin with a simple but essential question: is investor sentiment related to returns at all?

Joint distribution of sentiment and returns — Joint distribution of investor sentiment and market returns

The left panel exhibits an upward tilt from bottom-left to top-right. Periods of negative past returns are associated with bearish sentiment, while positive past returns coincide with more bullish attitudes. This indicates a strong relationship between sentiment and past performance.

In contrast, the distribution linking sentiment to future returns appears more symmetric and dispersed. No clear directional pattern emerges: bullish investors do not systematically earn higher future returns, nor do bearish investors consistently experience lower ones.

Sentiment metric

Investor sentiment is summarized by a single index, S = %Bullish − %Bearish, capturing net optimism on a scale from −100 (fully bearish) to +100 (fully bullish).

Correlation with market returns

Past 6-month returns (Pearson) ρ ≈ 0.32 p < 0.001

Sentiment is strongly and positively correlated with recent market performance. Investors tend to become more optimistic following periods of rising prices, consistent with trend-chasing behavior.

Future 6-month returns (Pearson) ρ_s ≈ −0.09 p < 0.001

Looking forward, the relationship weakens and reverses sign. High levels of optimism are followed by slightly lower future returns, suggesting that excessive bullishness may precede periods of underperformance.

Overall interpretation. Sentiment appears to respond strongly to past returns but provides limited predictive power for future performance. Although both correlations are statistically significant, their modest magnitude indicates that sentiment explains only a small share of market return variation.

Initial finding. Higher optimism is associated with higher future returns, and the relationship is statistically significant and robust to outliers.

Nevertheless, correlation alone cannot establish causality: do shifts in sentiment drive markets, or do markets primarily shape investor sentiment?

The Critical Test: Granger Causality

Statistical Method

Granger Causality asks a specific question: Does adding past values of Variable X reduce the prediction error of Variable Y, compared to predicting Y using only its own history?

The Formal Test

To test if Sentiment (S) causes Returns (R), we compare two models using an F-test:

1. Restricted Model (Null Hypothesis)

We try to predict Returns using only past Returns.

R_t = α + ∑ β_i R_t-i + ε_t

2. Unrestricted Model (Alternative Hypothesis)

We add past Sentiment to see if predictions improve.

R_t = α + ∑ β_i R_t-i + ∑ γ_i S_t-i + ε_t

The Test: We check if the coefficients γ (the effect of sentiment) are jointly different from zero.
If γ ≠ 0, then Sentiment "Granger-causes" Returns.

Evolution of Causal Strength: Blue (Returns→Sentiment) vs. Red (Sentiment→Returns)

The results tell a story of complete asymmetry. By looking at the plot above, we can trace the flow of information:

The Red Line (Right Side): Sentiment fails to predict

This tests the Unrestricted Model above. The coefficient sum (∑ γ) is effectively zero. Knowing past sentiment adds no predictive power to the model. The line stays flat near zero, far below the dashed significance threshold.

The Blue Line (Left Side): Returns strongly drive Sentiment

Here we flip the equation: attempting to predict Sentiment using past Returns. The blue line skyrockets, reaching huge significance levels (p < 0.01). This confirms that yesterday's market performance is the primary driver of today's investor mood.

The Conclusion: This confirms Extrapolative Expectations. Investors assume that "what just happened" is "what will happen." When markets rise, they become bullish. When markets fall, they become bearish. They are not forecasting; they are reacting.

Session 3 Summary: The Timeline Verdict

1

Correlation exists but is misleading. Sentiment and returns are correlated (ρ ≈ 0.33), but this doesn't mean sentiment predicts returns.

2

Causality is asymmetric. Returns massively affect sentiment; sentiment barely affects returns.

Next Session Preview

We've established the direction of causality. Now let's examine the patient's mood cycles: distinct behavioral states and how sentiment affects transitions between them.

Time for the mood analysis.

Session 4: The Mood Cycles

Identifying States

Behavioral Analysis

Patients don't exist in a single mood, they cycle through distinct psychological states. Our patient, The Market, is no different. This session examines these mood cycles using Markov regime models and tests whether investor sentiment affects state transitions.

First, Define the Patient's Moods

Before analyzing mood cycles, we need an objective way to classify the market's own states. We let the data speak for itself.

We use a Gaussian Mixture Model (GMM) with 3 components to identify distinct market regimes. Unlike simple thresholds, GMM provides soft membership probabilities and handles the fuzzy boundaries between real market states. Features include 6-month forward returns, realized volatility, and market momentum.

GMM clustering of market regimes — GMM clustering reveals three distinct market regimes. Each color represents a different regime, with the model capturing natural boundaries between bullish, neutral, and bearish states.

Bullish Regime

Higher returns, moderate volatility. The market is trending upward with confidence.

Neutral Regime

Mixed performance, sideways movement. The market is indecisive.

Bearish Regime

Lower/negative returns, heightened volatility. Stress and drawdowns.

Regime Transitions

A Markov model assumes the market exists in one of several discrete states, and the probability of transitioning to the next state depends only on the current state, not the entire history. The market's "mood" today tells us something about tomorrow's mood.

Unconditional Transitions

First, we examine how the market transitions between states without considering external factors like sentiment.

Unconditional Market Regime Transition Matrix — Market regime transition probabilities

Regime Persistence Patterns

Each cell shows the probability of moving from one market state (row) to another (column):

84% Bull → Bull Bull markets are self-reinforcing

90% Neutral → Neutral Neutral states are most persistent

88% Bear → Bear Downturns are sticky

Key Pattern: All three regimes show high persistence (84-90%), meaning once established, they tend to continue. Interestingly, neutral regimes are the most persistent at 90%, while bull markets persist at 84%. This suggests the market often stays in "wait-and-see" mode before committing to a direction.

Next: Where Does the Market Spend Its Time?

A key property of Markov chains is that, under certain conditions, they converge to a stationary distribution — a stable long-run probability distribution over states that no longer changes with additional transitions. Why does this matter for our analysis? If the empirical (observed) distribution of market regimes closely matches the theoretical stationary distribution, it suggests: 1. The market has been observed long enough to reflect its equilibrium behavior 2. The Markov model provides a reasonable description of regime dynamics 3. The regime frequencies we observe are representative of long-term expectations

Empirical vs Stationary Distribution — Empirical vs. theoretical distribution

What Are We Comparing?

Stationary (theoretical): Given the transition probabilities, math tells us how often the market should be in each state if we ran the process forever.

Empirical (observed): How often the market actually was in each state. In our data: Neutral (829 weeks), Bullish (413 weeks), Bearish (412 weeks) across 1,654 total observations.

Why it matters: If these match, our Markov model is a good fit. The market's mood cycling has been consistent over time.

Finding: Close match. The market spends roughly half its time in neutral territory, with bullish and bearish periods roughly equal—consistent with the theoretical prediction.