Invest with AI
00:29:21 · Doug Clinton
[00:29:21] S0: Yeah. I agree. And I think that's actually a great point, too, Kay, because if you think about in the investment space, the ability to control your own destiny too. This is just a philosophy that we have at Intelligent Alpha. But to the extent that we can sort of build something ourselves or use open source and modify ourselves very quickly, we've tried to very much do that and not rely too much on third party vendors because of what you just described. Like, you know, the vendor could be awesome when you install them day one, and something new could come out. And and by no fault of their own, just the fact that they bet on a certain idea or a certain technology, maybe they fall behind a little bit. And six months later, maybe they're awesome again. Like, it's just really hard to not be able to sort of control your own destiny given how fast things are moving. So that's the other piece for us is just philosophically kinda how do you think about how quickly and and what kinds of investments you wanna make in building your technology infrastructure, and is it something that you wanna have a lot of control over? Or or are you okay with relying on good third party vendors for that?
00:30:23 · Brett Caughran
[00:30:23] S2: Doug, have you thought about model model routing? You know, sort sort of feels like every few weeks you get a new model and, you know, the OpenAI models were ascended. We need this sort of h one twenty six has been all about Claude. Now this sort of conversation shifting back to Kimi and DeepSeek and the open source models and toke... Token efficiency argument, etcetera. How have you thought about evaluating that systematically? And when do you make the decision to plug in a new model, sort of pivot pivot your routing structure?
00:30:52 · Doug Clinton
[00:30:52] S0: Yeah. We do a ton of benchmarking internally. We've actually started to publish some of this work as well externally literally today. So it'll be a few days, I guess, maybe prior to when this gets launched. But we launched something called the IA 500, which is a nod to the S and P 500, obviously. But it's a it's a stock picking benchmark where we have a quarterly rebalance structure just like most indexes, and we test 10 of the top models. Top meaning, you know, most influential. It would be all the names that you would expect to kinda be in that group. And we look at their ability to pick a portfolio of about 200 stocks, And then we actually ensemble all of the picks from the 10 models into a single portfolio of about 500 stocks. And it's performed quite well versus the S and P 500 over the past year since we've been kind of putting this data together. So you can check it out at ia500.com. But the the kind of answer to your question, though, Brett, is we're always doing these tests on an ongoing basis to see what models sort of stand out where. And what I would tell you is this. There, in our view, there is a separation that is notable between the closed source models and the open weight models that we see on most of our benchmarks. In general, I would say when we're talking about investment tasks, not just stock picking, the the closed source models and GPT has consistently been the best one that we test in this section. They always have a little bit of outperformance as a group relative to the open weight models. So we do see some superiority there. And I know, like you said, Claude has been, probably the hottest one this year. And, this is a contrarian take, but but I do think that GPT is probably a better model overall for the stock related work that we do. It generally is our top performing model in most of the things we look at. So that's piece number one. On the open source stuff, I actually think that, to your point, is is really interesting because Kimi's had a moment. Nematron is is actually, I think, quite good as well on the American side. GLM, I know, was was hot before Kimi came out with k three. I think we're gonna see kind of the same thing there in in this open source space, which is just like the closer...
00:33:17 · Doug Clinton, Brett Caughran
Or the the closed source models, very rapidly sort of pushing the boundary up but maybe not quite hitting the closed source. That's sort of my bet is that we'll see a persistent gap between closed and open, mainly because I think a lot of it is, you know, based on distillation, how fast some of the the the open source stuff is moving, which is totally fine. But if you really wanna be on the frontier, and I think you do want that in the investment space where 1% edge is important, I think you really do have to spend more time looking at the closed source models. [00:33:53] S2: Yeah. Okay. How have you thought about this? I mean, you've spent a lot of time with Claude specific training, Claude coworking in particular, which has been sort of the default UI for, you know, mid to small size funds. Super powerful, particularly when you sort of set it up with the right connectors and the right skills. If funds have adopted Cloud Desktop, Cloud Cowork Code, etcetera, and you see this model evolution, what's next? What's the counter punch that evolution?
00:33:53 · Brett Caughran, Khe Hy, Doug Clinton
[00:33:53] S2: Yeah. Okay. How have you thought about this? I mean, you've spent a lot of time with Claude specific training, Claude coworking in particular, which has been sort of the default UI for, you know, mid to small size funds. Super powerful, particularly when you sort of set it up with the right connectors and the right skills. If funds have adopted Cloud Desktop, Cloud Cowork Code, etcetera, and you see this model evolution, what's next? What's the counter punch that evolution? [00:34:27] S1: Folks are scratching their heads right now. I think that the smaller funds that I work with are they're not ready to go to the open source. I mean, a lot of these funds use managed, you know, managed IT services. Like they don't even have a dedicated IT person in house, let alone a software engineer in house. So I think the open source question is not for the kind of the long tail of smaller funds without the technical expertise. I think that there is a huge pull into Claude. I think some folks are still using ChatGPT web because they haven't like Codex was very scary, you know, just the name intimidated people. But Codex is having its co work moment. I mean, it had it and it's very, very good. The new Codex or ChatGPT work, whatever. I don't know. I don't even know what to call it anymore. Yeah. They need to [00:35:26] S0: fix that. Got a branding problem. You're right?
00:35:28 · Khe Hy
[00:35:28] S1: They've got big branding problem. I still call it Codex just so people know what I'm talking about. And I think some folks are starting, the folks that kept their ChatGPT subscription open are starting to dabble. I've found that in the folks I talk to, intelligence, so much of the way LMs are being used is like the end to end workflow or as much of the workflow that you can capture. So yes, obviously you want the model that is 2%, 3% better. But if the harness isn't there, then that raw intelligence doesn't really matter to my clients. And so they are kind of poking around on Codex. I think that until they fix the branding problem, though, no one's really pulled the trigger because no one understands what Codex is, but it's really co work. And I think that's the problem that people haven't had. And I think that just the issue of cost, I think one of the things that's really interesting about the chat GB, the codex versus co work conversation is that, as Doug said, the 5.6 model series is very good. It's very good at agentic work. I can't speak at the nuances of financial analysis, the edges of financial analysis, but it's also significantly cheaper. And I just as a tangent, I've got a $200 Claude Max Plan that I personally that I get close to using up regularly. And I've got a $20 GPT plan. And they're generous on their resets, but it takes a long time for me to hit the $20 limit on the GPT plan. So I think that folks are trying to figure it out. I think eventually they'll get to a dual place, but that's really like an IT and a mind share question until there's like a 20% delta in performance that's not there at the moment.