X For You Feed Algorithm
A Deep Dive from Scratch

Big Picture#

Your For You feed is built by three co-operating services, all orchestrated by a central Home Mixer. The pipeline runs entirely in Rust (core pipeline + Thunder) and Python/JAX (Phoenix ML models), with gRPC connecting each service. The diagram below shows the real-time data flow on every feed request.

The Four Core Components#

Before diving into each pipeline stage, here is a quick map of which service does what:

Pipeline Walkthrough#

The CandidatePipeline trait in Rust defines the exact execution order across eight distinct stages. Use the interactive explorer below to walk through each step — you will see the source code, the logic, and why each stage exists.

// candidate-pipeline/candidate_pipeline.rs
async fn execute(&self, query: Q) -> PipelineResult<Q, C> {
    let hydrated_query = self.hydrate_query(query).await;
    let candidates    = self.fetch_candidates(&hydrated_query).await;
    let hydrated      = self.hydrate(&hydrated_query, candidates).await;
    let (kept, _)     = self.filter(&hydrated_query, hydrated);
    let scored        = self.score(&hydrated_query, kept).await;
    let selected      = self.select(&hydrated_query, scored);
    // ... post-selection hydration, filters, side effects
}

let hydrate_futures = hydrators.iter().map(|h| h.run(&query));
let results = join_all(hydrate_futures).await; // parallel fetch
for (hydrator, result) in hydrators.iter().zip(results) {
    if let Ok(hydrated) = result {
        hydrator.update(&mut hydrated_query, hydrated);
    }
}

The Phoenix ML Model#

Phoenix is the recommendation engine responsible for discovering out-of-network content. It has two components: a retrieval model to find candidates, and a ranking model (a full transformer) to score them.

Retrieval — Two-Tower Architecture#

The retrieval model uses a classic two-tower neural network:

• →User Tower — encodes your engagement history (post hashes + action types + product surface + dwell time) into a single dense vector.
• →Candidate Tower — encodes each post in the global corpus into a vector, normalized to the unit sphere for stable dot products.
• →Similarity Search — dot product between your user vector and all candidate vectors; top-K are returned to the ranking model.

# phoenix/recsys_retrieval_model.py
class CandidateTower(hk.Module):
    """Two modes: MLP projection (default) or mean-pool → L2 norm."""

    def __call__(self, post_author_embedding: jax.Array) -> jax.Array:
        # Mean-pool across hash embeddings, then L2-normalize to unit sphere
        candidate_representation = jnp.mean(post_author_embedding, axis=-2)
        candidate_norm_sq = jnp.sum(candidate_representation**2,
                                    axis=-1, keepdims=True)
        # EPS clamp prevents division-by-zero for zero embeddings
        candidate_norm = jnp.sqrt(jnp.maximum(candidate_norm_sq, EPS))
        candidate_representation = candidate_representation / candidate_norm
        return candidate_representation.astype(post_author_embedding.dtype)

Ranking — Grok-Based Transformer#

The ranking model is a Grok-based transformer (ported from the Grok-1 open-source release). It takes three types of input tokens, concatenated into a single sequence:

• 1. Your user embedding (1 token)
• 2. Your engagement history (up to 128 tokens)
• 3. Candidate posts (up to 32 tokens each)

# phoenix/recsys_model.py
embeddings = jnp.concatenate(
    [user_embeddings, history_embeddings, candidate_embeddings], axis=1
)

# Candidates CANNOT attend to each other — only to user context.
# This makes scores consistent regardless of what else is in the batch.
model_output = self.model(
    embeddings,
    padding_mask,
    candidate_start_offset=candidate_start_offset,
)

Scoring Formula#

The RankingScorer computes the weighted score from 22 predicted signals (20 discrete action probabilities + 2 continuous predictions). These are weighted and summed, then author diversity and OON adjustment are applied in the same scorer pass:

Drag the sliders to see how adjusting each predicted signal moves the final score. Key additions vs. earlier descriptions: retweet (not "repost"), vqv (video quality view), share_via_dm, quoted_vqv, and not_dwelled as a negative signal.

Post age is also a factor — bucketed into 1-hour bins up to 80 hours, giving fresher posts an advantage:

# phoenix/recsys_model.py
POST_AGE_MAX_MINUTES = 4800  # 80 hours

def compute_post_age_bucket(impr_ts_sec, post_creation_ts_sec,
                             granularity_mins=60):
    post_age_minutes = (impr_ts_sec - post_creation_ts_sec) // 60
    bucket = (post_age_minutes // granularity_mins) + 1
    return jnp.clip(bucket, 0, overflow_bucket)

The Grox Content Pipeline#

Grox is a separate content-understanding service that runs classifiers against posts before they enter the main ranking pipeline. It uses Grok (the LLM) to make decisions.

# grox/classifiers/content/spam.py
class SpamEapiLowFollowerClassifier(ContentClassifier):
    async def _classify(self, post: Post) -> list[ContentCategoryResult]:
        convo = await self._to_convo(post)
        result = await self._sample(convo)   # calls Grok
        return await self._parse(post, result)

# grox/engine.py — processes tasks from an async queue
async def _run(self, started_event: Event):
    await self._init_run()
    while not self._is_shutdown() or not self._task_queue.empty():
        task = await self._poll_task()
        asyncio.create_task(self._run_task(task))

What the Algorithm Rewards#

Based on the scoring formula and pipeline structure, here is what genuinely moves posts higher in feeds:

What the Algorithm Penalizes#

Key Design Decisions#

model_output = self.model(
    embeddings,
    padding_mask,
    candidate_start_offset=candidate_start_offset,
)

@dataclass
class HashConfig:
    num_user_hashes:   int = 2
    num_item_hashes:   int = 2
    num_author_hashes: int = 2
    num_ip_hashes:     int = 0

pub trait CandidatePipeline<Q, C> {
    fn query_hydrators(&self)        -> &[Box<dyn QueryHydrator<Q>>];
    fn sources(&self)                -> &[Box<dyn Source<Q, C>>];
    fn hydrators(&self)              -> &[Box<dyn Hydrator<Q, C>>];
    fn filters(&self)                -> &[Box<dyn Filter<Q, C>>];
    fn scorers(&self)                -> &[Box<dyn Scorer<Q, C>>];
    fn selector(&self)               -> &dyn Selector<Q, C>;
    fn post_selection_filters(&self) -> &[Box<dyn Filter<Q, C>>];
    fn side_effects(&self)           -> Arc<Vec<Box<dyn SideEffect<Q, C>>>>;
}

Common Misconceptions#

These myths persist because the algorithm is opaque by default. The open-source code corrects them: