Latent Space: The AI Engineer Podcast
00:16:04 · RJ Hanaki
[00:16:04] S1: So on the one hand, you have the bitter lesson and, you know, sort of you, you remove inductive bias, and you end up being able to have a larger data set that's applicable to your problem. And then that helps you to get more accurate models because your, the your distribution is wider and whatever. So that's like the one and then but then there's this, like, countervailing tendency, which is like, maybe you could call it the anti bitter lesson, or like the other the flip side of the bitter lesson where when you're in data poor regimes, for one thing, then you have to induce introduce a lot of inductive bias to get enough statistical power to make good inferences. But there's also a second side of that, which I think John Platt in our discussion with him not too long ago, brought up was that you have these these, like, predictive and then descriptive models. And what you really want as a scientist is to get to a simplification, a descriptive model where you can say, oh, well, I've tested all the apples and the gravity works in such a way on apples. Then but I'm modeling this as a universal concept simply so that we know that it works on plants as well. Right. And and so that I want to understand from you, what are the simplifications or, you know, sort of maybe simplifications, maybe insights, maybe actual mathematical or other wise scientific models that are becoming apparent that are gonna help to reduce all the complexity so that we're able to actually make meaningful... Gain meaningful understanding from these really large datasets.
00:17:39 · Fabian Theis
[00:17:39] S0: Yeah. That's that's a... That... That's a very important... Very... I think kinda deep point, which I think sort of breaks the field into maybe sort of a bit two different groups of people with answers. Right? So how much do we need to put inductive biases maybe priors into a model to maybe make it better explainable, right? But potentially also more robust in going out of distribution? Or how much are we gonna just rely on bigger data to sort of have the model be more expressive and so on? And I think... And I need to be very clear before we may go into the philosophy. I think literature has really clearly shown over, I think now decades, at least for our very many data sets, that if we build an a priori inductive biases too strongly, that we might lose predictive power, right? So it comes at a cost. So if you're now, let's say, a physician, and you just want to... Or maybe even if you're a patient and you you maybe you might have cancer or not, maybe you want to have the best model as a predictor. Maybe she getting treated, you want to have a model that is maybe a bit less good, but you understand it a bit more. But for the prediction, at least I think you you want to have the best. And this is I think I think not entirely clear. And I think in our field with this perturbation space scale, the sales guy I think is big, but the perturbation space scale, like the stuff that we do to it not being as big, I don't think we're yet in the super data rich regime. So I think in our case, these hybrid models might be still the thing to go for. But more and more, of course, and this is I think this, I don't know if you can call it sad, but the honest learning from from big scale machine learning is that at some point data beats human made additional concepts in these models. And I have at least one simple example I often put is AlphaFold, where, you know, the first model sort of made this extraordinary effort to to make, you know, the molecules rotationally variant, they sort of build it into the learning process, and only in later versions, you know, sort of rotation symmetry was automatically found by the model just with more data, it was a bit more flexible because whatever physical constraint you put in, it's not always as rigid, and then you sort of you gain something if you let the model learn from this.
00:19:48 · Fabian Theis, RJ Hanaki
So I think in the future, the bigger part, at least of the sort of more used models will be those more flexible, very much database ones where you then a posteriori put the interpretation on top. [00:19:59] S1: This also brings up the question for me. So you actually, I think, recently wrote a or co authored a paper on the statistical power that the current generation of virtual cells have in terms of the data that the the lift that you get from additional data in those current generation of virtual cell models? And I think you had said basically that the objective function or the training methodology is potentially the problem there and not necessarily that the information is not in the data. So, like, what... Can you explain that a little bit and why you think that maybe there's plenty of untapped information in the information? And then, like, what... Where does that get us, and what what do we need to do to get further?
00:20:44 · Fabian Theis
[00:20:44] S0: Yeah. I I I think it's fair to say that, you know, early on, we had some sort of very small sort of more limited prediction models of learning, let's say, latent spaces first, and also sort of perturbations on top. And then, you know, when like this really large scale foundation model representation came out, everyone was sort of buying into the hype, and maybe potentially sort of a bit over buying, and you know, obviously after this, you know, comes, what's it called, the sobering up value or something like that. Yeah. And then there were a whole bunch of papers coming up where they were basically saying, you know, hey, maybe even more or less linear model were doing a good job. And actually in our scGen paper early on, we had already some of these benchmarks showing here. It's not so not so super easy and simple task to improve on that. I think I think there's a truth to it that particularly in the current data state, for some things, let's say you want to predict the effect of a CRISPR in a cell line, you might not need the most fancy model. Whereas if you ask a bit... Or if you want to predict cell type or something like that, you know, for that, most of your logistic regression is fine. But if you wanna predict something a bit more difficult, for example, in our niche format paper, which is a foundation model for spatial omics, we showed... We had a new task which we introduced where you say you have a... You have the gene expression, you have sort of this state, the latent space of a single cell, can you predict the gene expression state of its neighboring cells? Because, you know, they're always in communication, or maybe the other way around. So, like, they're always in communication, so because, you know, they talk to each other, you sort of see a remnant of that sort of cell state in this one. And actually, this is harder to do. And if you sort of have these harder tasks, need these stronger, more nonlinear things, and you particularly need to look for more subtle data. So your question, can you sort of find out more sort of information, which type of things, I think very much depends on what you sort of ask at the end of it. Of course, like in the self supervised learning phase, I think we're definitely not there yet with the best way how we represent, how how we learn these things.
00:22:31 · Fabian Theis
There's a lot of copying of what happens in NLP type of LLMs. Right? And like mass autoencoders, so was the typical way to go. But the type of losses, the type of representations, think people are still debating, maybe... Because, you know, this is sort of often kind of tabular data, so even just like tabular foundation models are now sort of being tried out. There's a lot of JEPA type of approaches now, because it's like we have one thing where we use contrastive learning. So it's just people playing around with the losses, but also with the representations. And I think nowadays, reviewers, and at least I've been advocating for a long time, are really much more closer looking that you really properly show scaling loss. This was what has missed from the early papers of the foundation model, and so people haven't really shown that you need this complexity to actually sort of really scale up. But if you go really... If you can really demonstrate at scale, like across examples and and samples that you have this sort of scaling behavior, then I think you have this buying into these models. And for these, of course, there's not so much at least deductive bias in these use. So I I do think we're gonna gonna hit this regime actually quite soon, we do see this also in a few papers. So for example, in our contrastive learning paper, we call it SE concept, we do show already quite quite quite strong scaling behavior. In the paper, we had something for going... Because we compared with Geneformer going to maybe 30,000,000 cells, but on on Hugging Face now, we have a 350,000,000 cell model with I think going up to a 100,000,000 parameters. So really going to the big scale for us, and and we do see nice scaling behavior already starting to emerge.
00:23:53 · RJ Hanaki
[00:23:53] S1: Just to be clear then, at a 100,000,000 parameters and three... You said, I think, 300,000,000 cells or something like that, that you you get a lift in what kinds of applications? In these these sort of the niche former kinds of questions where you're saying, what is my environment based on the... My transcriptome?