Unsupervised Learning with Jacob Effron
00:51:15 · Jacob Effron, Diogo
[00:51:15] S0: You mean on on the compute side? [00:51:17] S1: Yes. On all sorts of sides, but especially compute. I think that financial engineering, as far as I've heard, is their greatest strength. And I have no idea how much of a rat's nest that is and how vulnerable they would be to any which direction of the market going. You know, like, a bubble can pop just because expectations are not met, not because it's not truly a transformative technology. So I'm a little... I try not to... I try to be calibrated in my predictions. I try not to be overly confident when I have no privy information. And I I would... I'm I'm a more conservative person. I I get the sense that a lot of the expectations baked in are ones that are not just product level, but based on, like, misunderstandings of the future of AI and what it can do. So my guess is that because some people might be valuing, like, building god as basically infinite, there'll be some amount of miscalibration there. And I think that, you know, like, the let them cure cancer is going to be like Marie Marie Antoinette's, like, let them eat cake. [00:52:28] S0: That is a that is a good line. Yeah. Mean, obviously [00:52:31] S1: Oh, maybe I... Maybe it didn't spicy then. [00:52:33] S0: That that that was that was a great one. I feel like the... Yeah. Mean, obviously, so many people are underwriting, you know, the move into... From coding agents to general knowledge work to bio to these other spaces and, obviously, like, not the perfect use cases for RLVR by any means.
00:52:47 · Diogo
[00:52:47] S1: I still get the sense that the, like, the the nuanced way to understand agents perhaps and maybe all of AI right now is what decisions can you trust them with in in terms of, like, stakes for the company. And, like, can you, like... Can you get, like, value without too much of the the the downside? And coding is, like, quite unique in this. Or or, like, math is actually, like like, super duper unique, where, like, there's no downside other than the compute you spend. Right? Like, math models do not cost companies anything except if they're, like, paying people to validate their proofs, which it's unclear if they even pay them for that. Yeah. To me. Maybe someone knows. But code also has, like, somewhat more capped downside because you have, like, a git. Right? And you have humans in the loop, and, like, there's ways to protect you from unwanted side effects, which is the main downside of powerful tools. And powerful tools are necessary to do knowledge work because knowledge work often involves, like, side effects in the world. I'm talking a lot like a computer scientist right now. My sense is that for a lot of other fields, they don't have anywhere close to the same levels of, like, safeguards and ability to revert and, like, tracking and all of that. And that is why, to my knowledge, no one has automated, like, a drive through yet even though people have tried really, really hard. And the stakes of a drive through are a lot lower than the stakes of many other forms of knowledge work. So I think that the the truth will be, like, there'll be, like, nuance and skill into it, and the people who will be the big winners are not the ones who just YOLO all in, but, like, understand reality and, like, deploy them, you know, in the... In in line with the realities of technology. So I am... I... And I'm not... Given my state of AI, I feel like most people are very FOMO driven, and they do not have a great understanding of the state of AI. And they will probably on the overoptimistic, which will get them burned with many side effects, or, like, the overpessimistic, which is business as usual, which is how most businesses have been for this AI boom.
00:54:53 · Jacob Effron, Diogo
[00:54:53] S0: Well, one thing you said there, like, you know, you... I'm curious. You said you have an estimate for how much these coding agents actually improve engineering productivity. What what is that estimate? [00:55:04] S1: Okay. I I will say I have, like, error bars on this because I think that there's, like, long term effects like, you know, skill degradation and tech debt and stuff like this. So my estimates are in the 20 to 40% speed up range. But, again, it really depends on the use case, it also depends on, like, the skill of the user. And if you can, like... You know, because the side effects are the bad thing, if you can isolate the side effects in a way that those are very legible. So depending on the use case, for some, it's kind of like an infinite percent speed up. You know, you would do things you would know. You can do things you can never do otherwise. For others, they are, like, just so bad that they're as bad as, like, coding the thing by hand. I would guess on average, it's, like, a decent but small boost. And I I believe that is the consensus that I've seen in the field right now. [00:55:54] S0: Yeah. Another question I'm not totally sure how you how you answer. Like, how can... Know, there's obviously so much discourse around AI safety these days. Like, how concerned are you about a a lot of these... You know, this AI safety conversation?
00:55:54 · Jacob Effron, Diogo
[00:55:54] S0: Yeah. Another question I'm not totally sure how you how you answer. Like, how can... Know, there's obviously so much discourse around AI safety these days. Like, how concerned are you about a a lot of these... You know, this AI safety conversation? [00:56:06] S1: I think AI safety is not one monolith, and there's, like, a ton of different slices of it. I think that the ASI discussion is actually extremely different from the bio risk discussion, which is very different from, like, the cyber risk discussion, which is different than, like, trust and trust and safety. [00:56:25] S0: Maybe, like, ASI versus, like, bad actors, you know, using bio and and cyber and whatnot.
00:56:30 · Diogo
[00:56:30] S1: I get the sense that most people concerned with ASI are a little bit divorced from the reality of the technology, and this has been the case since day one. You know, like, you... I I feel like you need to, like, actually build a reliable system with AI to have a voice in the room that something's gonna be, like, so reliably, maybe not evil, but misaligned that it can destroy the world. But I I think we're pretty far from stuff like that. Additionally, there's actually a bunch of nuance towards, like, you know, ASI, RSI, and, like, whether or not the models have a sense of agency on their own. Like, paper clip maximizer is actually very different than bad actor programs super employee to destroy the world type things. And, like, actually, I think that those end up becoming quite different in threat model. I think that we're breakthroughs away from true agency as much as people like to, like, read these chains of thought and try to ascribe real agency to it. I would guess that we are something like two breakthroughs of gigantic magnitude away from this. I think even one breakthrough would not be sufficient is my guess, but maybe one gigantic one could be. And I think a threat model of, like, humans with super tools who want to do damage is very different than the the ASI RSI thing. I don't... I think a lot of people who also believe in RSI both have not studied the history of AI very much and also, like, have no appreciation of Amdahl's law, where, like like, you know, you know... Like, if you assume that intelligence is the is the spectrum and it's the only one that matters and you have to map things into that spectrum, and you can also, like, map humans and chimpanzees, it's a horrible argument in my opinion, then it does look bad, but the reality is that it doesn't work that way. And, like, the nuance of, like, the many different types of intelligence and how to do them reliably really matters a lot in this whole analogy. Could the field of AI development become a little bit faster? Yes. But people have been trying this for a long time, and it usually doesn't work out. And it... I don't think that people would count, like, a 10% speed up on AI research as RSI if it doesn't actually increase over time. So I'll... I...
00:58:50 · Diogo, Jacob Effron
I'm very open to being convinced by evidence, but I I think that it's more of, a philosophical problem rather than a technological one. The kinds of people who, like, read, like, a Bostrom's superintelligence and treat it like reality over, like, a very interesting sci fi book makes makes me kind of worried. [00:59:09] S0: Yeah. Well, this has been a fascinating conversation. I thought we covered a a ton of things. I wanna make sure to leave the last word to you. Like, anything you wanna point our listeners to... I mean, I'm sure they're all familiar with, and certainly by the end of this podcast, certainly hope they are. But, like, anything we haven't discussed or... I don't know. The floor is yours.