Learning / 06 May 2026
Cold Start, Warm Graph
A local validation benchmark, three evaluation traps, and one feature with no parameters.
The leaderboard said 0.9357. The task was temporal link prediction, and 54% of the candidate pairs involved a node with no history. A model that had learned the warm part of the graph could still look competent on average and fail exactly where the task was hard.
The team's first instinct was to tune the model. The better move was to build a validation signal that could tell us whether a change was real.
Three traps
The public split had three problems that made local scores lie.
Temporal leakage. A random split lets the model see future edges while predicting past ones. The score goes up; the task gets easier; the result stops meaning anything. We split by time, not by row.
Candidate-pool mismatch. The official evaluation samples negatives in a way that differs from the training distribution. A model tuned on the training negatives can look better locally than it will on the leaderboard. Val-Hot rebuilds the candidate pool to match the evaluation protocol.
Cold-start blindness. Aggregate MRR hides the population that matters. We reported warm and cold subsets separately, and optimized for the cold subset even when the aggregate moved slowly.
Val-Hot
Val-Hot is a small, open local benchmark. It is not a leaderboard replica; it is a stable signal. The point is not to predict the exact public score. The point is to rank two candidate changes correctly, so the team can spend its submissions on ideas that are likely to survive.
The benchmark ended up being more valuable than any single feature. It turned arguments into experiments. When someone proposed a change, we could ask: does it improve cold MRR on Val-Hot, or does it only improve the average?
The heat feature
The largest single gain came from the least sophisticated idea: a zero-parameter heat feature. Count how often a node appears in the recent graph, normalize it, and add it to the score. No training, no embedding, no learned weight.
It contributed +0.047 MRR. On a task where the cold nodes are the difficulty, recent frequency is a strong prior: a node that has been active lately is more likely to appear again, even if the model has never learned anything about it. The feature is not elegant. It is useful, which is better.
The result
The team moved from 0.9357 to 1.2443. The gain came from a combination of a better validation loop, a repaired evaluation, and a handful of simple features. The lesson is not "handcrafted features beat deep learning." The lesson is that on a noisy benchmark, the ability to measure is part of the model.
I still think about the 54%. In most machine learning stories, the cold-start nodes are an edge case. Here they were the majority. The aggregate metric was the edge case.