Matt Jacobson (00:00):
I'm Matt Jacobson, one of the partners here at ICONIQ, and thrilled to do this with Edwin Chen. I believe Edwin has built one of the most consequential companies in AI while working behind the scenes, which kind of speaks to our ethos here at ICONIQ. He and I first met in Singapore at an event we were hosting, and since then we've spent a lot of time with coffee walks in New York, mostly me asking him questions about this fascinating business that he's built.
(00:26):
And Surge is an outlier from many of the companies that have followed a more traditional venture playbook. Edwin has grown it well past a billion dollars in revenue. I think SemiAnalysis had it at $3 billion. He is a little more discreet in terms of sharing these details. But it's a team of less than a hundred people with no outside capital. And Surge provides human data environments, evaluations behind many of the models that are driving at the frontier, and that work gives Edwin a very unique view into the frontier. He sees what the leading labs are asking for. At the same time, he sees where they're stuck before the rest of us do. And he publishes a lot of wonderful research on it, including experiments that no customer ever asked him to run.
(01:12):
He holds views on the industry that I've found to be different than the consensus ones that kind of pushes me and pushes our team in terms of our thinking. It looks at progress in terms of where it's coming from. We've talked a lot today in terms of open models and what those mean downstream, and so excited to get into it. Please welcome Edwin Chen from Surge AI.
(01:40):
Tuhin earlier from Baseten talking about compute as an important ingredient in terms of the equation, but data has been an incredible important ingredient in terms of the creation of AI at the frontier. When you started Surge, Scale AI was kind of the leading player in the space in many ways and had defined a lot of the data labeling category in V1 of this, and you built with a different presence. It was very engineering-led, automation-driven, and kind of deeply focused on quality above anything else. Would just love to take us through those insights in terms of, how'd that shape the company that you built with Surge today.
Edwin Chen (02:18):
I think our unique insight early on was that we were really, really focused on the quality of our data and thinking about what quality even means. And let me explain what I mean by that. So when we first started, every company in our industry was basically focused on computer vision. So you show someone an image and you ask them, "Hey, is this a cat or a dog?" Or "Can you draw a bounding box around a car?" And I was never really interested in these kinds of problems because they have very simple black-and white answers. Like, "Yeah, it's either a cat or a dog."
(02:53):
We have the full power of the human mind in front of us, and why are we limiting ourselves to these kinds of questions? And so in contrast, I came from this world of language and behavior. So I used to work at Google, and the question I would ask was, "Okay, you have a user who is trying to understand the rebound hypothesis. How do you decide what is the best search result to show them at the top?" Yeah, you hear. In contrast to drawing a bounding box on a car, you actually do need a lot of specialized mathematics knowledge in order to answer this correctly.
(03:20):
But correctness and expertise actually isn't enough. When you're thinking about how to show the best research result and what good even means for Google, you're asking yourself questions like, "Okay, how well-explained is this answer? Does the language in this website, does it match the level of background that this user has? Is there a nice visualization that you can show that helps them understand the answer more intuitively?" And so here you have all of these facets around expertise, but are also around taste for what kinds of behaviors you're trying to encourage.
(03:56):
I think what's interesting is that if you think about frontier models today. Like if you're asking Claude to code, a very, very similar thing is happening. You don't just want Claude to pass a series of unit tests. You also want it to produce code that is elegant enough for 50 other people to maintain. You want it to create products that are not just functional, but also really fun, and beautiful, and intuitive to use.
(04:22):
So, yeah, I think this notion of just really thinking deeply about quality and what it means, that made us very unique in this space. And today when models are as powerful as they are, one of the things I often say is that there's this infinite ceiling of quality that you want models to reach. You don't just want them to be functional. You don't just want them to give you a black-and-white answer. There are all these questions that are a little bit more subjective, a little bit more about personal judgment, and we want to push models towards those.
Matt Jacobson (04:49):
I love that in terms of the nuance of quality and the legend of Surge is that you turn away business to just prioritize these aspects in terms of quality, but it's a consistent theme that we hear in terms of Surge in the market.
(05:02):
You sit in a very privileged seat in terms of your work with all of the leading AI labs gives you a view in terms of the research frontier, and you've been generous in the past in terms of sharing what you're learning. We have frontier model releases today. What changes are you seeing in the frontier today, and what do you expect to see over the next month or six months? I think beyond that, it's very hard for any of us.
Edwin Chen (05:27):
Yep. So I think one of the really interesting things about the frontier is actually how uneven it is. For example, earlier this year, we created a benchmark called Riemann-bench, and what it did was it was a measuring model's ability to conduct frontier research mathematics. And the way we created it was by taking Ivy League professors and asking them, "Hey, go do your own everyday research, and while you're doing your research, come up with problems that AI models just can't solve." And so at the time, models were only able to solve 5% of them.
(05:59):
But since then, like in the past few months, it's kind of crazy how much mathematical progress there's been. The latest Claude and OpenAI models, they're actually solving hundred-year-old problems that a lot of really, really good mathematicians have fought long and hard about. And it's actually a little bit scary because I think of mathematics as this very profound science that gets at trying to understand the nature of the universe. And so it's a little bit scary that models are actually able to understand the universe better than we can.
(06:26):
But at the same time that these models are so good at math, they actually fail a lot of ordinary, everyday work. For example, we have another benchmark called HANDBOOK.md, which gives agents, they're basically giving agents a hundred-page company handbook. So this might be handbook like, "Here's our HR policy. Here's our expense policy. Here's our security policy," and then they throw agents inside this company environment and they tell them, "Okay, go do someone's job."
(06:54):
And here, models are actually still failing over 70% of the time. So you simultaneously have models that are better than probably 95% of mathematicians, probably solving things that even Fields Medalists can't solve, but they fail at these things that ordinary white-collar knowledge workers aren't solving.
(07:12):
And so I think the next frontier is going to be less around solving these really, really difficult research problems. Models are already on the path towards that. Instead, it's going to be this question of, "Okay, can models get through a Tuesday at the office? Can they do work for 12 hours without failing and just do it autonomously?"
Matt Jacobson (07:30):
And that is where Surge comes in. And I think labs are definitely shifting the budgets from what started in terms of a few years ago or even static data sets to this concept of RL environments. I think there's probably like 50 YC companies that are doing RL environments. You and I have talked about this. But you've been building this concept for a long time. Can you just share, what is an RL environment, and why is the budget and tremendous amounts of money going towards data and improvement of models flowing into it?
Edwin Chen (08:04):
So an RL environment is basically a world or company that an agent can act inside. One of our RL environments is basically a startup that we are creating from scratch. And so inside a startup, we have a bunch of employees. It's software engineers and accountants and lawyers and product managers, basically anybody that you would imagine within a real company. And then they're also given a code base. They're literally iterating on this code base every single day. They have access to tools like Slack and Jira and Salesforce. And then what they're doing is they're creating tasks for the AI models to solve, like, "Here are 10 bug reports that users reported. Go fix them." Or "Hey, can you create a report of our monthly financials?" And then they're teaching models how to do a good job.
(08:49):
And the reason this is really valuable is because models are already really good at producing static answers, but what they're bad at is operating over long horizons. Okay, you want the model to create a piece of code that takes 10 hours. You want them to operate through an entire workday. And what they lack is the ability to sort of work on your own, realize when they are making mistake, backtracking, fixing that mistake, and then progressing. And so our environments are really, really useful for this because they literally give the models a playground tact inside.
(09:24):
And so we have thousands of these RL environments across every economically valuable domain, like health, finance, consulting, legal, and you can really think of it as us building a day in the life of a JP Morgan analyst and then asking AI models to do the job for them.
Matt Jacobson (09:40):
And this is going to unlock longer-form, more coordinated agent workflows that we should expect to see out of the major labs in the next coming releases. And you and I were talking a little bit in terms of the green room just in the back in terms of there's a lot of noise in terms of real life, "Hey, there's some company that sells their data to a provider in terms of doing that. There's real data or synthetic data." Which ones have you seen be the most successful? Which ones are driving more towards that kind of long-form, long-running agent future?
Edwin Chen (10:15):
So there's a lot of interest in building these RL environments on top of both real company data and synthetic data. And right now, I think the industry is still trying to figure out which one is more valuable. So the benefit of real company data is that you get the actual messiness of the real world. It's actually kind of funny, but models are actually not that good at generating synthetic data. They either generate synthetic data that is too clean, or the synthetic data often leaves hints for the models to ... It gives them a hint for how to solve the problem.
(10:46):
The con of real-world data is that oftentimes real-world data is actually too easy for the models, and so you want to hyper-target the worlds that you're building in very, very specific ways. And so we do a lot of both just because there are a little bit of pros and cons to each approach, but I think we'll start to see in the next few months which one is more valuable.
Matt Jacobson (11:06):
Got it. Just segueing a little bit to the unique way that you're building Surge. I think many other players in the marketplace have been quite loud and announcing every financing, every revenue milestone, and you've taken a very different approach and stayed nearly invincible, except exceptionally close to your customers, while becoming, in our view, one of the most consequential companies in AI. What did that approach buy you, and what have been the costs of being kind of behind the scenes and under the radar?
Edwin Chen (11:39):
Yeah. So when we first started, we made a very conscious decision that we wanted to win in the space by building a really, really great product and not by creating a lot of noise and hype. And we did that for two reasons. One was I've always sort of hated how Silicon Valley is oftentimes a big status game. You have a lot of founders who are constantly tweeting and fundraising and generating a lot of smoke and mirrors. And when you dig closer underneath, you often find that there's no substance to what they or their company is building. And I never wanted to take that path.
(12:18):
And the other reason is I've always cared a lot about making sure that we're aligned with the right incentives. When your incentive is doubling your valuation every few months, you end up taking a lot of shortcuts that make the industry and the models themselves worse in many ways. And so I always wanted to make sure that our incentives, our number one incentive was simply to make AGI happen in a meaningful way, and that sort of meant that we operated more like a research lab than a traditional Silicon Valley startup.
(12:51):
And I think that was very, very helpful because a lot of our champions would always tell us like, "Yeah, your data is actually so much better than everybody else's. You guys are able to do things that they've simply never seen before." And that was because we though about quality very, very deeply and because we weren't just constantly taking shortcuts to just make more money.
(13:10):
I do think it cost us, because it basically put us on hard mode for the past six years. So I definitely do think my view has changed a little bit in that I still don't care about fundraising and valuations and hype, but I think one of the things that's been most exciting to us this year is our research team actually has been a lot more public and a lot more visible. And so we've been publishing our own research and we've been publishing a lot of our own benchmarks that the industry labs have started to adopt.
(13:38):
For example, I think the Fable 5.1 and Mythos 5.1 dropped earlier today, and I was actually really excited that two of our benchmarks, one was called GDP.pdf and the other was called Chartography, they both appeared on the Claude model card. And so I think this visibility that helps us shape the field in a very meaningful way, I've actually really, really enjoyed that.
Matt Jacobson (13:58):
And you've done it in an authentic way to yourselves. I remember talking about the pdf one earlier this summer in terms of doing that, so it's amazing to show up on that benchmark. I think it ICONIQ, we deeply recognize that choice after spending years, I think, focused really in terms of the background. We have opportunities where we convene communities like this in terms of doing this. What should we expect out of Surge in the future as you start to embrace more of the market leadership that you're occupying today?
Edwin Chen (14:28):
Yeah, so I think I'm really excited about the post-training research that we're doing. I think there are still a lot of poorly understood questions around the value of data and how that trades off against compute, or the value of high-quality data and how maybe when you have more high-quality data and data that isn't susceptible to certain kinds of reward hacks, maybe that kind of data actually prevents a lot of the, again, these cybersecurity incidents that we've been seeing in the past few months.
(14:53):
For example, I think there's a hypothesis that's part of the reason these models have become so good at cybersecurity is because the data that is used to train them has certain loopholes or because the companies themselves that are creating these data sets, they're incentivized to almost reward-hack the data in various ways. And those principles sort of leak into the models in very, very subtle ways that have long-term consequences, and I think that's just a really, really interesting area of post-training research that we want to help the industry understand.
Matt Jacobson (15:28):
And when you look at something like security where oftentimes it has a very specialized skill set and experience, is that something where you partner with existing vendors, or the customers themselves have built out sophisticated security? How do you transition in terms of that lateral movement in terms of a different buyer, different problem, different sphere?
Edwin Chen (15:48):
Yeah, so I think one of the surprising things that people often don't know about us is that they think of the work that we do as sort of like low-level gig work. Again, people think of the industry as people drawing bounding box around cars where you really don't need that many skills.
(16:06):
In contrast, what we do is we actually try to find the smartest talent in the world for any domain. So we have all of these cybersecurity engineers and Fields Medalists and Ivy League professors on our platform, and so for something like security, we actually take all this incredible cybersecurity talent, and then what they're doing is they're constantly basically coming up with either hacks that the models need to solve or ways to use AI models to prevent hacks, and they're teaching models how to do this.
Matt Jacobson (16:36):
Your team is ... It's less than 180 people today?
Edwin Chen (16:40):
So we've actually grown a fair bit in the past year. So we're a couple hundred people now. But yeah, a year ago we were under a hundred.
Matt Jacobson (16:45):
But when you look at that relative to the scale of the overall business, it's very different than companies that were built in the past. Why do you think that is, and what do you think the shape of AI-native businesses looks like in this future?
Edwin Chen (17:00):
Yeah. So I think there are two reasons why we are still relatively small compared to old companies in our space. So one is that I think historically a lot of companies in our space, they've treated data as a pure labor problem. You're just throwing more human bodies at the problem without thinking about the technology behind it. And so we always treated this notion of quality, not as like a throwing more human bodies at it, let's ask 10 people and average the answer together somehow to crowdsource the best answer. Instead, we treat it as a data science problem where we're trying to measure whether or not people are good, we're trying to measure what skills they have, we're trying to measure whether they have taste. And so we just have a lot of measurements in place.
(17:41):
I think another part of it is that ... Again, going to this notion of being a little bit more quiet than others in our space, we ... I think a lot of startups when they're growing, what happens is that they hire a VP from Google, and then that VP from Google wants to polish up their LinkedIn resume. And so their goal is going to be, "Okay, how can I hire 500 people to polish my career trajectory? How do I make myself look impressive to the CEO," and so on. And so there are almost these negative incentives that make you want to grow a company simply for growing's sake rather than thinking about what is actually most aligned with the company's goals.
Matt Jacobson (18:22):
And when you look at things like go to market and customer intimacy in terms of doing that, again, I think you've really listened to your customers and stayed at the frontier in terms of what they're looking for, sometimes even ahead of when they recognize it themselves. Where does that come from? Where does that show up at Surge as an organization?
Edwin Chen (18:43):
Yeah, so I think it comes from a lot of my own background. I think one of the things I've realized in the past few years is that actually the company is maybe shaped by myself in ways that I wouldn't have predicted when we first started. And what I mean that is when I think about the things that I enjoy, what I enjoy is talking to our customers to understand, "What is it that they want to build? How do they want to shape their models?"
(19:07):
Every model, even if it may be trained on a lot of these same, let's say, pre-training data, every founder of these different labs or the people who compose these labs, they have different goals, different personalities, different benchmarks, different performance capabilities that they want their models to hit. And we talk to them a lot because we want to understand what unique abilities, what unique feel they want these models to have, and then we use that to shape the data that we create for them. So, yeah, I think a lot of that kind of stems from me, and then by extension, the people we look for when we hire.
Matt Jacobson (19:45):
It's been our experience that some of the best companies are shaped by the visions of the founders and you could see those traits as they show up in terms of the company, and Surge is certainly that case. Thank you, Edwin, for taking the time with us. This has been totally fantastic, and appreciate you doing this.
Disclaimer:
This presentation does not constitute investment advice or an offer to sell or a solicitation of an offer to buy any securities (including, without limitation, any investment fund or investment product referenced herein) which will only be made pursuant to definitive offering documents and subscription agreements.
Certain statements provided in the presentation are made by Edwin Chen, Founder and CEO of Surge AI.
Mr. Chen is a client of ICONIQ. ICONIQ did not compensate Mr. Chen directly or indirectly for participation in this presentation, but given his role as a client of ICONIQ, Mr. Chen may be subject to certain conflicts of interest. As a client of ICONIQ, Mr. Chen’s interests may be aligned with the performance, reputation, and fundraising prospects of such funds and the adviser. Positive statements about ICONIQ could indirectly benefit Mr. Chen in his capacity as a client.
This presentation may contain forward-looking statements based on current plans, estimates and projections. You are cautioned that a number of important factors could cause actual results and outcomes to differ materially from those expressed in, or implied by, the forward-looking statements. The numbers, figures, and case studies contained in this presentation have been included for purposes of illustration only, and no assurance can be given that the actual results of any ICONIQ portfolio company will correspond with the information contained in this presentation. No information is contained herein with respect to conflicts of interest, which may be significant.
Certain of the economic and market information contained herein may have been obtained from published sources and/or prepared by other parties. While such sources are believed to be reliable, none of ICONIQ or any of its affiliates, partners, employees, and representatives assume any responsibility for the accuracy of such information.
All of the information in this presentation is presented as of September 1, 2026 (the “Production Date”) (except as otherwise specified), is subject to change without notice, and may not be current or may have changed (possibly materially) between the Production Date and the date actually viewed by you. ICONIQ assumes no obligation to update or otherwise revise any information, projections, forecasts or estimates contained in this presentation, including any revisions to reflect changes in economic or market conditions or other circumstances arising after the date the items were made available to you or to reflect the occurrence of unanticipated events.
ICONIQ is not acting as an adviser or fiduciary in any respect in connection with providing this presentation and no relationship shall arise between you and ICONIQ as a result of this video being viewed by you.
These materials are provided for general information and discussion purposes only and may not be relied upon.
