What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team

Takeaways

• Data market grows from $10bn toward $100bn. Sahil says the market was $1bn three years ago and is $10bn now, and in three years he does not see how it is not $100bn ¹. Models cannot be tested in the real world until launch, unlike code with its tests ³.
• Frontier data now needs expert-authored environments. Sahil says frontier models now need domain expertise in investment banking or medicine, and a two-year banker can supply it ². Experts write prompts and rubrics running to hundreds of rows ⁵. Asking a model to generate a data room produces slop ⁶.
• Templated data can look good and train badly. Nishant says misaligned or templated data can give good loss curves and rewards but a final checkpoint that makes no sense ⁷. He would spend a full day on one piece of data, judging individual datasets and their diversity ⁸.
• Life science models fail real lab tasks. On Vials, Handshake's life sciences benchmark, all models score 50% or worse and MiniMax and GLM fall below 20% ⁹⁴, while pharmaceutical experts finish the tasks in under a minute ¹⁰. Models report near 100% confidence on wrong answers ¹¹.
• One error can poison an entire finance model. Atlas has agents work as bankers across 100 finance tasks, and models score under 20%. In a 2019 sponsor LBO with a 338(h)(10) election, the agent tied its Python to Excel and shipped the deliverable without flagging an error ¹²¹³. A failed step-up gate zeroes 180 tax points ¹⁴.
• Safety must be built into training. Nishant says safety has to be built through training rather than bolted on at inference, because much harm comes from repeated interactions over a long period ¹⁵. Care Bench, a 500-prompt child safety benchmark with 12 risk categories, was annotated by clinical psychologists and child protection investigators ¹⁶.

What comes next

• Nishant predicts that within two years all, or most, of the data industry becomes the alignment industry ¹⁷.
• He says today's tasks are single points in time, and data and models must move to long-horizon, end-to-end work, such as a scientist spotting a problem across thousands of lab assays and asking the CFO for equipment ¹⁸.
• Sahil says labs are adding preference ranking inside environments because great taste in a domain is hard to capture in a rubric ¹⁹.
• Shawn Wang says modeling is only the first step of financial analysis, and the buy side also needs macro news flow and long-context industry understanding, pointing to industry models rather than single-company models ²⁰.