Latent Space: The AI Engineer Podcast
00:24:11 · Fabian Theis
[00:24:11] S0: Yeah. I I mean, you know, the the ultimate thing, of course, we wanna do is we wanna be doing this perturbation prediction better, because that's sort of what people nowadays understand by virtual cell reign. And for this, I would not claim that pure representation learning is the right way to go. Sorry for making the subtlety again, but I think typically you learn the state space with these foundation models, but then sort of the way how they behave under an action, under perturbation. This is where you need this more world model like type of approaches. World model being the very hip term, I'm sure you had this already at some point in your podcast. So it's been on all over machine learning conferences in last year. But de facto, a perturbation prediction model, like a virtual cell, is kind of a world model in terms of transcriptome or other type of omics space. Right? So it's it's not just a model that is sort of f of x, but f of x comma a. Right? So the state x plus the action. Right? And then sort of predicting the next state x prime or something like that. And I think for this, you might need other combinations. So what we typically do, we learn the representations with, for example, as a concept, but then we put a generative model for the push forward along the action. So for this, we use something like flow matching or something like that. And in our self flow paper, this is not just got accepted for Nature Methods. We actually really show that these approaches are much better generating the effect of of perturbations, but they're very much dependent on a good latent space. And you know, on this as a concept latent space, in this case, for this 350,000,000 cells, we actually went into multi species, so you have now like 16 different species where you can potentially look for similarities suddenly, you can maybe ask some evolutionary questions that you couldn't before. Not that we can do this so well, it's not our core expertise, but you know, sort of we can start doing these fancy, sort of cross species, cross tissue type of latent spaces.
00:25:47 · Brandon Anderson
[00:25:47] S2: I'm kind of curious about the history of some of these different models. I think you just gave us a sort of a rapid version of them. The one of the things that, like, I think was, like, somewhat famous in the community was this thing about linear models being kind of a a big point. Can you explain specifically what was the, like, key point here? Because I think this is, like, one of those things in classic machine learning, make sure you understand your baselines well. You know, otherwise, you you spend a lot of time spinning your wheels and fool yourself. I think there was a specifically a a famous open competition on... With the... To the ARC Institute that was a sort of a class of, like, cold water in everyone's, you know, faces. Can you maybe give a little bit of history about that and like why we thought this was exciting and then where... Why this had us pivot a bit? Yeah.
00:26:35 · Fabian Theis
[00:26:35] S0: It sounds such an... Like such an obvious learning, right, to your baselines properly. But in a field that is rushing towards the gold part of the virtual cell, people sometimes maybe were not as careful with the baselines, and there's a whole bunch of subtleties with respect to holdout and so on that, you know, wasn't always fully observed, and why some of these things happen. So one of the first observations there was in a thing as basic as cell type prediction. Okay. So nothing super fancy. Okay. Basically, you you give me a new transcriptome, tell me from which cell type it is, right, maybe even from which organ or something like that. And it turns out that even nowadays, I think some close to state of the art models are really just logistic regression, right. So for such a simple categorization, you know, this is actually not not so bad. There's one thing called cell typist, for example, which is doing a good job, and I think we had a paper maybe five or six years ago where we had the title very plain, Deep Learning is Not Needed for Cell Type Prediction. This is true if it's within an organ. If you sort of go cross organ, suddenly it gets a bit more subtle because you know, of cell types reoccur in different organs and immune cells, of subtle change depending which tissue they are. Suddenly, you know, there you need to do a bit more, and we, like for us, there was one paper called SETAP, like a tabular foundation model, predictive model type of thing. We actually showed that this nonlinearity was really adding beyond the baseline. So that was sort of for that class. For the perturbation models, which then you were asking Brennan similarly, like early on, many analysis, particularly perturbsiek, were done with classical linear models in SuperBulk. We had then our our scGen that was like something around 2019. We started introducing linear shifts, but then the latent space was sort of nonlinear encoded, and we always compared with subtle... Sort of simpler linear shifts. But then the field getting complex and, you know, sort of datasets popping up, people started asking more and more perturbation type of questions. There have been a whole bunch of these early on competition for, like, a regular one run, for example, by MIT.
00:28:24 · Fabian Theis
We've been running a regular NeurIPS competition on prediction problems in single cell data, sometimes for integration data, also for perturbation prediction. That was very well attended, and the cherry on cake was most definitely that ARC competition, which called the thing not just perturbation prediction challenge, but Virtual Cell Challenge. That sound a bit cooler, right? It picked up very well, And but with metrics that are robust, were maybe... But maybe not not that easy. So for... Actually actually our solution in in this this this consortium, where a lot of labs contributed called open problems, was actually that people need to submit their own metrics in addition, so in order to really quantify maybe subtle things that you might not have thought before. The ARC one sort of fixed a bunch of metrics that weren't even so so single cell specific, and they could be just just exploited. So actually, I think the winning model that was then in the end actually sort of crowned as the actual winning model, but essentially it's a flow matching based thing. So based on on this idea that, you know, people pull forward with this flow, with this self flow type of idea, I think shows you that some nonlinear things are in this case needed, so that the actual real winners, or if you sort of remove the exploitation, was actually something that is this is taking some some nonlinearities. And I think nowadays, we do see that just linear or additive type of models are not cutting the the the cake for more complex things, for I think even nowadays, if you go in a simple with few cell lines and you have sort of gene knockouts then, particularly when the effects are a bit subtle, all these sort of more complex models are sometimes a bit overfitting, and that's why linear baseline is always very good. So I always tell my students, guys, I know we we... Guys, I mean guys and girls, we always need to do... I know you wanna do the most fancy deep learning thing, but please, please, you know, have the baseline. And like say for drugs, know, baseline as simple as just predicting getting better at the mean, you know, like that's like the hardest place. Like not even linear model, like better than the mean. Sometimes not as hard if there are so much noise in the data, and this stuff is super noisy.
00:30:15 · RJ Hanaki, Fabian Theis, Brandon Anderson
[00:30:15] S1: I just wanted to add one point that Bo Wang and some of his collaborators actually just had a nature paper on on how the metric was actually the problem in some of the... Those linear baselines beating some of the models as well. So it's clearly... It's not only metric hacking, but also just that the asking the wrong question leads to the wrong answer. [00:30:35] S0: Or choice of the metric. Yeah. And I I think that they could... You can discuss it. At some point, we're gonna have agents sort of trying out all kind of combinations of whatever algorithmic thing you can come up with. In fact, we actually do have some collaborations with some of of those sort of AutoML couple labs to try this out. So maybe what is left sounds a bit too harsh. Right? But like, boy, maybe our biggest ideas go into it is actually this method formulation, RJ, right? [00:30:58] S2: So one thing I've been really interested in is the different models of funding and research academia. I think Germany has a different structure fundamentally from The US. How is your institution set up? And like, how does this give, like, distinct opportunities versus like a US based research model?
00:31:15 · Fabian Theis
[00:31:15] S0: I I very much agree that these funding models very much matter for the flexibility, but also the set of teams that you can set up and like the long term vision that you can have. So Helmholtz, this is not just Germany, but I would say maybe Europe's largest, what's called research organization in terms of funding. It has a different set of areas that it focuses on, health being one of the big ones, but then it has sort of material information sciences, physics, and so on. And essentially getting fed federal money to put into a deep research question. I've been building up what's called the computational health center at Helmholtz Munich, which I think has now something like maybe 45 group leaders and professors, and maybe 500 to 600 people now. So we are, I think, the largest academic place for health AI type of research in Europe, where we have people really working on all scales. Like, you know, I'm sort of focusing on the cellular scale, but, you know, there's people doing DNA foundation models. There's people sort of doing interesting things on protein design. There's people doing like, you know, patient cohort type of representation, genetics, and so on. And now you sort of can start putting them together, and one of our big visions is sort to to intersect and to go virtual human and so on. But having said that, the key thing is in contrast to universities, that even though we all have affiliation to universities, mostly to the Technic University or the University of Munich, the two top ranked universities interestingly in the European Union. So we have this sort of place where we have these affiliations at universities. Right? But we don't have undergraduate teaching. We essentially just have PhD students and postdocs, but we interact very much with the universities and in particular also the clinics because you wanna have sort of this translational thing in there. And increasing... That's maybe an interesting link. So this is maybe a bit similar to NIH type of models like in US, but sort of distributed all over the country, not just sort of in one place.