Latent Space: The AI Engineer Podcast
00:19:48 · Fabian Theis, RJ Hanaki
So I think in the future, the bigger part, at least of the sort of more used models will be those more flexible, very much database ones where you then a posteriori put the interpretation on top. [00:19:59] S1: This also brings up the question for me. So you actually, I think, recently wrote a or co authored a paper on the statistical power that the current generation of virtual cells have in terms of the data that the the lift that you get from additional data in those current generation of virtual cell models? And I think you had said basically that the objective function or the training methodology is potentially the problem there and not necessarily that the information is not in the data. So, like, what... Can you explain that a little bit and why you think that maybe there's plenty of untapped information in the information? And then, like, what... Where does that get us, and what what do we need to do to get further?
00:20:44 · Fabian Theis
[00:20:44] S0: Yeah. I I I think it's fair to say that, you know, early on, we had some sort of very small sort of more limited prediction models of learning, let's say, latent spaces first, and also sort of perturbations on top. And then, you know, when like this really large scale foundation model representation came out, everyone was sort of buying into the hype, and maybe potentially sort of a bit over buying, and you know, obviously after this, you know, comes, what's it called, the sobering up value or something like that. Yeah. And then there were a whole bunch of papers coming up where they were basically saying, you know, hey, maybe even more or less linear model were doing a good job. And actually in our scGen paper early on, we had already some of these benchmarks showing here. It's not so not so super easy and simple task to improve on that. I think I think there's a truth to it that particularly in the current data state, for some things, let's say you want to predict the effect of a CRISPR in a cell line, you might not need the most fancy model. Whereas if you ask a bit... Or if you want to predict cell type or something like that, you know, for that, most of your logistic regression is fine. But if you wanna predict something a bit more difficult, for example, in our niche format paper, which is a foundation model for spatial omics, we showed... We had a new task which we introduced where you say you have a... You have the gene expression, you have sort of this state, the latent space of a single cell, can you predict the gene expression state of its neighboring cells? Because, you know, they're always in communication, or maybe the other way around. So, like, they're always in communication, so because, you know, they talk to each other, you sort of see a remnant of that sort of cell state in this one. And actually, this is harder to do. And if you sort of have these harder tasks, need these stronger, more nonlinear things, and you particularly need to look for more subtle data. So your question, can you sort of find out more sort of information, which type of things, I think very much depends on what you sort of ask at the end of it. Of course, like in the self supervised learning phase, I think we're definitely not there yet with the best way how we represent, how how we learn these things.
00:22:31 · Fabian Theis
There's a lot of copying of what happens in NLP type of LLMs. Right? And like mass autoencoders, so was the typical way to go. But the type of losses, the type of representations, think people are still debating, maybe... Because, you know, this is sort of often kind of tabular data, so even just like tabular foundation models are now sort of being tried out. There's a lot of JEPA type of approaches now, because it's like we have one thing where we use contrastive learning. So it's just people playing around with the losses, but also with the representations. And I think nowadays, reviewers, and at least I've been advocating for a long time, are really much more closer looking that you really properly show scaling loss. This was what has missed from the early papers of the foundation model, and so people haven't really shown that you need this complexity to actually sort of really scale up. But if you go really... If you can really demonstrate at scale, like across examples and and samples that you have this sort of scaling behavior, then I think you have this buying into these models. And for these, of course, there's not so much at least deductive bias in these use. So I I do think we're gonna gonna hit this regime actually quite soon, we do see this also in a few papers. So for example, in our contrastive learning paper, we call it SE concept, we do show already quite quite quite strong scaling behavior. In the paper, we had something for going... Because we compared with Geneformer going to maybe 30,000,000 cells, but on on Hugging Face now, we have a 350,000,000 cell model with I think going up to a 100,000,000 parameters. So really going to the big scale for us, and and we do see nice scaling behavior already starting to emerge.
00:23:53 · RJ Hanaki
[00:23:53] S1: Just to be clear then, at a 100,000,000 parameters and three... You said, I think, 300,000,000 cells or something like that, that you you get a lift in what kinds of applications? In these these sort of the niche former kinds of questions where you're saying, what is my environment based on the... My transcriptome?
00:24:11 · Fabian Theis
[00:24:11] S0: Yeah. I I mean, you know, the the ultimate thing, of course, we wanna do is we wanna be doing this perturbation prediction better, because that's sort of what people nowadays understand by virtual cell reign. And for this, I would not claim that pure representation learning is the right way to go. Sorry for making the subtlety again, but I think typically you learn the state space with these foundation models, but then sort of the way how they behave under an action, under perturbation. This is where you need this more world model like type of approaches. World model being the very hip term, I'm sure you had this already at some point in your podcast. So it's been on all over machine learning conferences in last year. But de facto, a perturbation prediction model, like a virtual cell, is kind of a world model in terms of transcriptome or other type of omics space. Right? So it's it's not just a model that is sort of f of x, but f of x comma a. Right? So the state x plus the action. Right? And then sort of predicting the next state x prime or something like that. And I think for this, you might need other combinations. So what we typically do, we learn the representations with, for example, as a concept, but then we put a generative model for the push forward along the action. So for this, we use something like flow matching or something like that. And in our self flow paper, this is not just got accepted for Nature Methods. We actually really show that these approaches are much better generating the effect of of perturbations, but they're very much dependent on a good latent space. And you know, on this as a concept latent space, in this case, for this 350,000,000 cells, we actually went into multi species, so you have now like 16 different species where you can potentially look for similarities suddenly, you can maybe ask some evolutionary questions that you couldn't before. Not that we can do this so well, it's not our core expertise, but you know, sort of we can start doing these fancy, sort of cross species, cross tissue type of latent spaces.
00:25:47 · Brandon Anderson
[00:25:47] S2: I'm kind of curious about the history of some of these different models. I think you just gave us a sort of a rapid version of them. The one of the things that, like, I think was, like, somewhat famous in the community was this thing about linear models being kind of a a big point. Can you explain specifically what was the, like, key point here? Because I think this is, like, one of those things in classic machine learning, make sure you understand your baselines well. You know, otherwise, you you spend a lot of time spinning your wheels and fool yourself. I think there was a specifically a a famous open competition on... With the... To the ARC Institute that was a sort of a class of, like, cold water in everyone's, you know, faces. Can you maybe give a little bit of history about that and like why we thought this was exciting and then where... Why this had us pivot a bit? Yeah.