Latent Space: The AI Engineer Podcast
00:32:23 · Alessio Fanelli, Nick Magazine
[00:32:23] S4: And Vials? Yo. [00:32:26] S8: Yeah. So at Handshake, one of the things that we find very important are the sciences in general, in particular life sciences. Fundamentally, the promise of AI is really to better facilitate discovery in life sciences, in engineering, mathematics. The social contract is basically that we take all these risks with AI, and the reward is that we can facilitate like the hyper progression of innovation. So, life sciences in our opinion is very important there because it ties directly to human health and disease research, drug discovery, therefore it is the most valuable vertical in which that promise can exist. [00:33:11] S4: And it looks like if you scroll down a little bit, it's one of the domains where models out of the box do very poorly. Right? So your chart on the side, the minimax, the GLM, the GROQOL, sub 20%. Right? [00:33:24] S8: Yes. So, most models out of the box do fairly poorly. All of them perform at 50% or worse, even the very top of the frontier models, which if you're a scientist is highly problematic. I mean, ultimately, if you want to be doing any kind of biology or biochemistry research where human health is on the line, anything under very near 100% would not be considered acceptable or feasible for use if you're an agent.
00:33:24 · Nick Magazine, Alessio Fanelli
[00:33:24] S8: Yes. So, most models out of the box do fairly poorly. All of them perform at 50% or worse, even the very top of the frontier models, which if you're a scientist is highly problematic. I mean, ultimately, if you want to be doing any kind of biology or biochemistry research where human health is on the line, anything under very near 100% would not be considered acceptable or feasible for use if you're an agent. [00:33:51] S4: I guess I'm gonna take this time the naive, not bio person to ask my questions. Is this a combination of different tasks? Is it one task? What are the tasks? You hear a lot about wet labs. So how do you close the loop of using a model for sciences and then testing it? And if you're running in a simulation, is it accurate? Explain that to the non bio guy.
00:34:16 · Nick Magazine
[00:34:16] S8: Yeah. So wet labs are a goal, if you will, for the state of AI right now. So, a wet lab is basically where somebody is validating physical research in the real world based on a model's hypothesis and just trying to gather empirical evidence based on things that the model is developing. One of the prequisites to that is going to be just a strong physical understanding that these models develop, which most, if not all scientists have during their training, that allow them to actually process these experiments. The atomic unit of these workflows... So, if think about an experiment, it's more of a narrative than a specific point in time. No medication, any kind of clinical investigation is ever just like a singular test. It's a series of tests that gradually build towards conclusion or confirmation of your hypothesis. And each of these units in life sciences and chemistry, we call assays, an assay just being like a singular measurement or like small experiment where something is being validated or measured. What this benchmark is testing is how well a model or an agent can actually interpret the outputs of each of these individual assays, which is the fundamental prerequisite of actually working in these lab in the loop environments and processing all of these complex workflows that are required for real innovation in the field. And unfortunately, the models aren't quite up to snuff yet, but hopefully with our data and innovation in the field, we'll get there eventually.
00:35:52 · RJ Hanaki, Nick Magazine, Sahil, Nishant
[00:35:52] S9: Yeah. So, can we look at a few of the examples that Yeah, you have absolutely. Yeah. Maybe we can start with the molecular diagram there. [00:36:04] S8: This guy? Yeah. So, just kind of rewinding, all of these are different various lab assays that you would use in a life sciences setting. These would be all kind of required for any basic clinical work or drug discovery, just an understanding of the physical space that these inhabit, and the diagram I believe you're referring to is here. So, one of the things you'll notice with a lot of these is these are very visual and very physical in a way. Like these are three d structures that are existing in the space. So, on the left here, this is a molecule. And in this case, it looks like it is a ligand or some kind of protein that exists within some kind of organism that's bound or unbound to a drug. One of them is being shown in red, and one of them is being shown in green. [00:36:54] S9: And the model It's actually GLP-one, right? Peptide. [00:36:56] S0: Oh, yeah. One of the greatest technology [00:36:59] S8: companies. I [00:37:00] S7: didn't know if this This is Ozempic.
00:37:00 · Nishant, Nick Magazine, Alessio Fanelli
[00:37:00] S7: didn't know if this This is Ozempic. [00:37:02] S8: Yeah. That's right. Yeah. Correct. Yeah. So so one of the things that people in, like, biochemistry and microbiology do all day when they're trying to develop drugs or just kind of figure out the mechanism of action for these sort of things is they're taking these biomolecules or spinning them around in all these various applications, taking screenshots, pictures, passing them to colleagues, and trying to just do, like, basic determinations based on what they're seeing in the space. So here, the model has been given the very specific but also fairly straightforward task of identifying how many hydrogen bonds are being formed in one structure but not another, which is something that you would do fairly routinely whenever investigating a drug. And on the right here, we can see the answer. And in this case, of data points within this set, the models are performing under 50% of the time as an aggregate. So it's something that the models are actually fundamentally incapable of doing at the moment with any degree of consistency. [00:38:03] S4: So, for this example, we would assume if you put this prompt and image into the models, like, it's not getting it right?
00:38:03 · Alessio Fanelli, Nick Magazine, RJ Hanaki
[00:38:03] S4: So, for this example, we would assume if you put this prompt and image into the models, like, it's not getting it right? [00:38:12] S8: Yeah. At least not consistently. And, yeah, some models may get it right, some may get it wrong. It depends on, you know, how the model is weighted. Some of them have, you know, strengths in specific sub domains or on data types that the researchers at those labs have trained on more. But across the board, we would assume that this would perform less than half the time [00:38:34] S9: accurately. And I I I guess I would call, especially for the listeners, that there's these two overlapping molecules. They're very close to each other. They're somewhat... And then, like, maybe if you look in the very center, there's this, like, 3.4 angstrom and little bond there, and it binds on one and not the other, the one that goes to the left. And so being able to pick that out from all the visual noise, you can imagine how that would be very difficult for a vision language model. [00:39:05] S8: Yep. Exactly. And for anybody who's listening, angstroms are like a unit of measurement at the molecular scale. I know it's not a common term. [00:39:14] S4: I guess at a high level, what grade or so, how... What's the level of difficulty of these tasks? Is this stuff that you would learn in, like, high school sciences, college, undergrad, master's, PhD? Where where does this range?