The Grid with Carl DiClementi

Episode Description

Carl DiClementi explains The Grid, an AI inference marketplace where vetted providers compete on price, quality, latency, and capacity.

Episode Summary

Carl DiClementi, product lead at The Grid and Anthony Campolo’s former QuickNode colleague, joins Anthony and Dev Agrawal to explain an AI inference platform built around competing suppliers. Instead of selecting individual models, users choose text, coding, or agentic instruments with Standard, Prime, and Max quality tiers, while The Grid routes requests to providers meeting defined quality, latency, and throughput requirements. Carl describes benchmark-based eligibility, Lab Latest routes, API compatibility, concurrency limits, bulk purchasing, and upcoming usage dashboards. He connects variable pricing to the economics of idle GPU capacity and describes using an agent to schedule nonurgent work when prices fall. Supplier vetting, zero data retention, and checks for malicious intermediary behavior are central to making the service trustworthy. The discussion also covers OpenClaw and Hermes, automated API monitoring, computer-use tools, AI adoption, deepfakes, and the distinction between online enthusiasm and regular use. Dev asks about future task-specific instruments and custom evaluation policies, prompting a discussion of shared specifications and contract stability. Carl closes by clarifying that trading offsets inference costs rather than enabling cash withdrawals, previews internal marketing automation, and offers trial credit. The hosts finish with upcoming projects and vacation plans.

Speakers

• Anthony Campolo
• Dev Agrawal
• Carl DiClementi

Chapters

00:00:02 - Introducing Carl and The Grid

Anthony introduces Carl DiClementi, his former boss at QuickNode and now product lead at The Grid. Carl describes his experience delivering blockchain infrastructure and how its economics helped shape his interest in AI capacity. Anthony proposes energy trading as an analogy for understanding the new product.

Carl starts with the user experience: The Grid is an inference provider that accepts a request and returns model output. Behind that API, suppliers compete to serve requests at suitable prices. He compares the arrangement with advertising markets and presents it as a familiar integration backed by a different way of sourcing capacity.

00:04:03 - Models, Tasks, and Quality Tradeoffs

Anthony asks which models and modalities The Grid serves. Carl explains that its primary focus is helping users choose capabilities rather than track individual model releases and deprecations. Providers must meet quality and reliability requirements before they can serve a selected category of requests.

Users choose text generation, coding, or agentic work, then decide whether speed, cost, or stronger reasoning matters most. Anthony compares the choices with several automatic routing modes. Carl agrees and distinguishes those existing controls from a possible future orchestrator that could infer the task category from each request.

00:09:33 - Instruments, Benchmarks, and Lab Latest

Carl demonstrates why The Grid calls its selectable routes instruments. Text, code, and agentic categories each offer Standard, Prime, and Max tiers, with quality, latency, and throughput specifications. Artificial Analysis indexes help determine which models qualify for those instruments rather than leaving every selection to the user.

Lab Latest serves a different preference: following the newest release in a chosen model family without continually updating a version string. Carl describes how that abstraction can accommodate users who favor a particular family. The screen share also exposes connection problems, prompting him to stop sharing while the hosts continue discussing the design.

00:13:03 - The Economics of Idle Capacity

Dev asks whether The Grid uses blockchain infrastructure. Carl says it does not, although experience at QuickNode and a cryptocurrency exchange influenced his thinking. Infrastructure providers must provision for peak demand, leaving expensive hardware underused during quieter periods even as it continues to depreciate.

The Grid gives suppliers a way to sell that otherwise idle capacity. Carl calls the model sell as you go: providers can seek usage when they need it instead of only waiting for direct customers. The discussion connects the market’s pricing behavior with the practical need to improve utilization across a day.

00:15:39 - Supplier Requirements and Data Protection

Anthony asks what a supplier needs to participate. Carl distinguishes The Grid’s current partners from individuals serving local models on personal machines. He points to reliability, exposed data, leaked credentials, and malicious tool calls as reasons to evaluate suppliers carefully before businesses depend on them.

Current partners must operate at scale, have a verifiable reputation, and support zero data retention. Carl leaves open the possibility of broader participation later, while emphasizing the platform’s immediate enterprise requirements. Most customers purchase an abstract inference service and do not need to manage the underlying providers themselves.

00:17:30 - Variable Prices and Scheduled Work

Carl describes using a Hermes agent to watch price dips and schedule daily tasks when inference is cheaper. Those tasks need to finish each day but do not require a fixed starting time. He says some instruments move around 20 to 30 percent during the day as suppliers balance capacity and demand.

Anthony sees an opportunity to move nonurgent work away from expensive periods. Carl adds that users need not monitor every provider: The Grid routes to the least expensive qualifying supplier while enforcing the chosen quality, latency, and throughput requirements. Direct interaction with the market remains an option for users who want more control.

00:20:42 - API Integration and Onboarding

Anthony asks how people use The Grid within their existing tooling. Carl describes generating an API key and placing it in a harness, with the platform handling supplier changes behind the scenes. An upcoming dashboard will show requests, tokens, instruments, and the models actually serving them.

Anthony creates an account and completes an API call during the explanation. They review examples for curl, Python, and JavaScript. Carl describes compatibility with chat completions, Messages, and Responses APIs, noting that some support is not yet visible in the interface. The goal is a straightforward integration that requires little ongoing attention.

00:24:02 - Concurrency, Bulk Purchases, and Integrations

Carl outlines account limits, saying free accounts receive three concurrent requests while larger balances can support up to 400. He wants the interface to explain those limits more clearly. The dashboard also exposes supported modalities, context windows, and input and output limits for individual instruments.

Bulk purchasing can address more specific needs, such as European data governance or certified providers. Carl says some customers have obtained discounts exceeding 30 percent. The interface also supplies harness setup guidance, streaming instructions, token-limit recommendations, and a tool for deploying an agent with The Grid already configured.

00:27:11 - Agent Platforms and Automated Monitoring

The hosts examine The Grid’s agency product and discuss OpenClaw’s popularity. Carl describes switching to Hermes after an OpenClaw update lost his setup. Anthony is curious about higher-level agents but still mostly uses coding harnesses directly. They compare interest in OpenClaw, Hermes, Grokbot, and Meta’s Muse.

Carl’s practical example is recurring API monitoring: an agent tests requests, watches latency and pricing, and sends Telegram notifications when conditions change. Anthony describes it as a richer health check than a simple ping. The discussion also questions how closely online enthusiasm tracks regular use and considers the major labs’ responses.

00:31:52 - Computer Use and Product Boundaries

Carl describes difficulty navigating websites with agents and compares that experience with OpenAI’s computer-use tools. He found some earlier screenshot-driven workflows slow and easily blocked. Anthony mentions Tracy Lee’s enthusiasm for ChatGPT Work, prompting a discussion of how it relates to ChatGPT and Codex.

Both speakers acknowledge uncertainty about the boundaries between these applications and their features. Carl sees value in combining coding, chat, and computer interaction rather than requiring a separate tool for each task. The conversation then returns to adoption and whether mainstream users will want agent capabilities without configuring an open-source project themselves.

00:35:56 - Everyday AI, Regulation, and Risk

Carl argues that AI is more useful when it improves ordinary software without requiring users to think about its label. Photo enhancement provides an example. Anthony compares resistance to AI with attitudes he encountered in blockchain, and the discussion turns to politics, regulation, and the limits of controlling a global technology.

Carl considers the acceleration possible when AI helps improve further AI research. Anthony recalls becoming interested in coding through existential-risk debates, then explains why he now worries more about misuse. These are the speakers’ assessments of uncertain scenarios, with practical harms forming the bridge to the next discussion.

00:40:47 - Deepfakes and Model Benchmarks

Carl recalls early demonstrations of deepfakes and contrasts them with increasingly convincing real-time impersonation. He discusses family verification phrases and the difficulty of deciding whether a video or voice is authentic. Anthony says brief generated clips can be difficult to recognize without the context in which they are shared.

They consider how familiar detection clues may weaken as models improve. Examples include Will Smith eating spaghetti, SVG video experiments, and the pelican-on-a-bicycle benchmark associated with Simon Willison. The exchange connects entertaining model tests with the more serious problem of establishing trust in synthetic media.

00:43:35 - Vetting Suppliers and Earning Trust

Dev asks how The Grid expands its supplier pool. Carl says providers are already seeking to join, making verification more pressing than recruitment. Reputation checks, recurring benchmarks, and secret-shopper testing help determine whether suppliers deliver the quality and behavior promised by the platform.

The group discusses warnings about suspiciously cheap inference. Carl describes checking for prompt manipulation and malicious tool calls while withholding some procedures to avoid making them easier to bypass. Transparency about the source of lower prices is part of the product’s explanation, even for customers who never use its market interface.

00:47:32 - GPU Shortages and Market Transparency

Dev questions whether a GPU shortage should also mean a shortage of suppliers. Carl distinguishes scarce hardware from continuous utilization: a provider can need more capacity at peaks and still have idle machines at other times. The Grid lets suppliers monetize troughs and change prices when demand rises.

Variable pricing is deliberate, and Carl accepts that customers wanting a fixed rate may prefer another service. Anthony returns to energy markets and describes learning about Enron and California blackouts. The comparison raises concerns about manipulation and underscores why users need a clear account of how infrastructure prices are determined.

00:52:33 - Future Instruments and Quality Tiers

Dev asks how The Grid selected its categories and tiers. Carl describes beginning with common use cases and credible benchmarks so that the market can attract enough activity. Artificial Analysis’s intelligence, coding, and agentic indexes provided the basis for the initial categories.

The naming follows a familiar low, medium, and high quality model through Standard, Prime, and Max. Carl anticipates instruments for tasks such as OCR, PDF extraction, legal work, and accounting. He also considers an orchestrator that chooses another instrument, while clarifying that image and video output require infrastructure the team has not prioritized yet.

00:58:34 - Custom Policies and Evaluation Culture

Dev asks whether users could define their own evaluation criteria. Carl explains that The Grid currently sets specifications, but hopes buyers and suppliers will eventually help determine them. Dynamic personal policies are harder to treat as tradable contracts, so changes would require careful coordination and versioning.

They consider community experimentation and organization-specific policies before discussing how much inference consumption comes from testing new models. Dev distinguishes formal evaluation from trying a model on a familiar project. Anthony argues that seemingly unproductive experiments can still lead to useful discoveries, while Carl jokes that users can run them on The Grid.

01:05:23 - Cost Offsets, Product Plans, and Closing

Carl previews dashboard improvements and clarifies that trading cannot produce cash withdrawals. Users can offset inference spending, but their balances remain within the platform. He also describes an internal marketing tool that creates campaigns, landing pages, A/B tests, and analytics to explore which messages resonate.

Anthony reflects on his initial misunderstanding of the market and shares Carl’s contact information. Carl offers listeners extra trial credit through a direct message. The closing covers Anthony’s vacation, Dev’s experiments with a code-review bot and deterministic checks, livestream spam, and an invitation for Carl to return. The recording ends with final thanks at 01:10:59.

Resources and Links

• Carl DiClementi on X
• The Grid
• Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain
• Dax on inference-provider risks
• Dev Agrawal on Jev Review

Transcript

00:00:02 - Anthony Campolo

Hello everyone. Welcome back to AJC and the Web Devs. This will be our last episode before I go on vacation for about a month. So very excited to have my old buddy and old boss, Carl. Carl was my boss at QuickNode, which was back in my blockchain days. It was a blockchain infrastructure company. So really happy to have you here. Why don't you tell our audience who you are and what you do?

00:00:28 - Carl DiClementi

I just want to say thanks a lot, Anthony, for having me on. Good to see you again. And yeah, I'm Carl. I've been working in the dev tools space on and off for pretty much my whole career at this point. It's been an exciting journey. Most recently, before the Grid, I was at QuickNode with Anthony, kind of making sure that we were able to deliver the highest quality blockchain infrastructure in the space.

00:01:01 - Carl DiClementi

Pretty exciting, the level of scale that we were working at there. And then, you know, while we were there, we were always fighting against these smaller companies or other organizations that were trying to commoditize infrastructure. And I kind of looked at where the AI space was going and gave me some interesting ideas. Then had an opportunity to come over to The Grid and lead product over here and hopped right into it.

00:01:30 - Carl DiClementi

It's been about a year that I've been here and working with some good friends that I've worked with many companies in the past, and I always enjoy that, you know, when you can work with the good folks again. So it's been a good ride.

00:01:45 - Anthony Campolo

Awesome. Yeah, very cool. Let's just dive right into it because The Grid sounds very cool. I'm going to try and first draw like an analogy. Um, see, you can tell me whether this actually makes sense or not, because I know that there is something called like energy trading where people can kind of trade based on what they think the value of like certain types of energy will be in certain places based on what people are going to be willing to pay based on like historical usage rates and stuff.

00:02:16 - Anthony Campolo

So, When I look at what the grid is, I think it's like that for AI. Is that about right?

00:02:23 - Carl DiClementi

Absolutely. Yes, that's a great way to think about it. I'd actually start really simple. For most users of the grid, we are simply an inference provider. If you don't know what inference is, that just is a fancy word for LLM output. You send an API request to us and we send you the output from an LLM. Simple, full stop. Where it's complicated is what's behind that, and it's very similar.

00:02:53 - Carl DiClementi

I like to use the example of Google Ads. When you go to a website and there's this little square on the screen, it serves you an ad, and that's all the person running the website needs to know. Well, what's actually happening behind the scenes is it's going to an exchange and it's finding the person that's willing to bid to be able to get that spot.

00:03:12 - Carl DiClementi

And pay the most money for it. Well, that's a very similar thing for us, where when you send an API request to us, it doesn't just— we're not the ones that directly serve it. There's a number of suppliers behind the scenes that are competing to be able to serve it to you at the best possible price. So for most users, think of us as a drop-in replacement for OpenRouter, but for anyone who really wants to get into the details, The way that we achieve some industry-leading prices is by suppliers competing to be able to serve your costs at the best price.

00:03:47 - Anthony Campolo

Okay, cool. Yeah, I didn't know that you were actually like literally just doing inference as well. That's really cool, 'cause actually I think I'm about to start working at an AI inference company very soon. So I'm just about to sign the contract, so I don't wanna say the company's name yet.

00:04:01 - Carl DiClementi

Congratulations, that's really exciting.

00:04:03 - Anthony Campolo

Yeah, yeah. Also, you're still lagging just a little bit. We're still able to hear and understand you, but if you're able to, tweak your internet a little more to any extent, that would be good. So yeah, thinking of it as kind of like a drop-in replacement for OpenRouter makes a lot of sense. Just before we kind of get into the market angle of it, like what models are you serving and are you just doing like text models?

00:04:31 - Anthony Campolo

Are you doing like image stuff like that?

00:04:33 - Carl DiClementi

That is a great question. So the Um, let me just drop— let me just set my laptop as a preferred device on my network really quick. Um, we do not actually focus on, um, serving a specific model to users. That's another way that we are kind of standing out, is our focus instead is to offer users the ability to kind of just focus on their task and not say, hey, oh, this model came out, oh, that model came out, oh, this one got deprecated, oh, this provider is no longer serving this model.

00:05:15 - Carl DiClementi

And staying up to date on that can really get exhausting. We've kind of coined the term model anxiety just because it's constantly changing and it's a lot to stay up to date on. We go and we vet every single provider. We make sure that they're able to deliver the quality and reliability that's necessary. And that is part of our offering. If somebody's not able to deliver that, then they cannot serve the inference.

00:05:43 - Carl DiClementi

So the way that it works on the grid instead is that somebody would say, hey, I am doing coding and I want the highest quality, or I want good enough quality, or I want the fastest, lowest cost quality. Right? And then from there, what we're constantly doing is we're working to evaluate which models qualify for each of those. So as of right now, we would say like the— I don't know, the lowest cost one is something like GPT-OSS-120.

00:06:18 - Carl DiClementi

That's very fast, very low-cost model. Also things like DeepSeek Flash. Things along those lines. And then you'd move up to the next level where you get things like GLM full size or Kimi full size. And then when you get to the leading models, you get to the Opus, Astra, Fable, things along those lines. So we evaluate and we make sure that users continue to get whatever tier that they select.

00:06:50 - Carl DiClementi

And then we also handle that between use cases. So there might be something like, code. There might be agent harnesses, right? Let's say you're using OpenClaw or Hermes, or it might be just general text generation. Say you're creating a chatbot. So that's kind of how we walk people through it is we say, all right, what is the use case that you have?

00:07:12 - Carl DiClementi

Is it text generation? Is it code generation? Is it agentic harnesses? And then what quality trade-off do you want? Do you want super fast? Do you want super smart or do you want some balanced offering?

00:07:26 - Anthony Campolo

Okay, that's really interesting because that's kind of how I do it with AutoShow where I made it where there'd be different models you can choose, but they're kind of ranked by different things like quality or speed or cost. Because a lot of people, they don't really want to have to care about the model, but they want to be able to choose the trade-off.

00:07:45 - Anthony Campolo

And because when you first said you don't pick the model, I was imagining in my head something like Copilot Auto Mode, where you go on auto mode and then you do a prompt and then it just picks a thing for you. But really it's more like you have 3 auto modes based on whether you want high, medium, or low quality auto, and then it will select those within that category.

00:08:04 - Carl DiClementi

That's perfect framing. Yes, absolutely. We've explored the idea of having an orchestrator type model that sits in between, that's doing the auto selection of what it is for you, maybe deciding if your current request should be text generation, because I don't know, every now and then I just like talk to Cursor like it's a text thing, right? Um, as well, I ask it to like write a Slack message for me or something along those lines as well.

00:08:32 - Carl DiClementi

Um, so we've thought about that, but that currently it is exactly as you described, um, where you select the use case and it's kind of multiple auto modes within

00:08:43 - Carl DiClementi

that.

00:08:43 - Anthony Campolo

Nope. Um, do you have questions, Dev, before we start talking about maybe some of the spot pricing stuff?

00:08:50 - Dev Agrawal

Yeah, I mean, so, uh, yeah, I'd love to hear more about— so, as— so, so far the value proposition I'm hearing is that OpenRouter lets you route to like any provider for any model that exists, but you're the one who has to make all the decisions about when and where a request should go. Uh, so you're taking away a bunch of that work.

00:09:15 - Dev Agrawal

Um, by instead create like building some policies that the providers actually compete for, and then the users have like much simpler levers to control, like, uh, cheap, medium, balanced, and smart. Is that what I'm gathering?

00:09:33 - Carl DiClementi

Very much. And I think it might help to get a visual on it, so let me just go ahead and share my screen really quick. Because this makes it pretty clear. So when you come to our consume page, like most of the others, this is where you're coming to understand how many credits you have, how to make your requests and whatnot. We, instead of calling them models to not cause confusion, because it's an auto-router as you described, we call them instruments.

00:10:01 - Carl DiClementi

This is going to get to the financial aspect of it as well. Instruments is a popular term that's used in the financial world. So you would select something like text, code, or agentic tests. Lab latest, I'll get to after. And within that, you have your standard, prime, or max. And these are kind of balanced based on tiers of quality, right? So you'd say like a rating 2 out of 4, 3 out of 4, 4 out of 4.

00:10:30 - Carl DiClementi

And then latency and throughput also have their different ratings, right? The way that we do that is they're based on very, very detailed specs that we go out and we evaluate. We lean on Artificial Analysis's indexes. They have things like the Intelligence Index, the Coding Index, and the Agentic Index, and we set tiers of grades for those. So for example, for text instruments, you would have greater than 24 for Text Standard, greater than 41 for Text Prime, and greater than 56 for text max.

00:11:05 - Carl DiClementi

So over the last 30 days, we've been serving GPT-OSs, MiniMax M3, and Opus 5. We're actually about to change this to a GPT-5.6. And then we also have code agent as well. And those work on different indexes. Those use the coding index or the Here, the agentic index. So you can start to see how we do that based on kind of quality tiers, and then also latency and throughput come into it as well.

00:11:40 - Carl DiClementi

What we also found is that there are plenty of users who are kind of like diehard Opus fans or diehard Star Wars fans or whatever it might be, and we didn't want to exclude them. Those people still struggle with the fact that, oh hey, the latest 5.6, 5.7, 5.8, 5.9 are coming out. Where the older ones are deprecating and staying up to date on that, while it might be fun for people, for some folks, a lot of people don't wanna deal with it, right?

00:12:08 - Carl DiClementi

So that's where we came up with the lab latest for them. So instead you would just say Claude Opus latest and then you'd send a request to that and whatever the latest version of it is. So 5 came out some time ago where we're still out where, This will be 0 as Claude Opus has been out for over— 5 has been out for over 30 days and similar situations for each of these.

00:12:36 - Carl DiClementi

Does that make sense?

00:12:41 - Anthony Campolo

Yes. Your lagging is getting a little worse actually. I just want to let you know.

00:12:47 - Carl DiClementi

I'm going to stop sharing. Maybe that'll help.

00:12:49 - Anthony Campolo

Yeah. Yeah, we might— we should probably hold off on, on that. Um, I could pull stuff up though if you, if you need me to. Um, awesome.

00:12:58 - Dev Agrawal

Yeah, that definitely helped though.

00:13:00 - Anthony Campolo

Yeah,

00:13:00 - Anthony Campolo

cool. Did you have any follow-up on that, Dev?

00:13:03 - Dev Agrawal

I guess— I mean, uh, I, I don't know exactly how much this, uh, like this new project, uh, of, of yours is also, uh, connected to like, uh, blockchain infrastructure. So I guess I was curious about that part.

00:13:18 - Carl DiClementi

Yeah, not at all, actually. Um, I— my background is in blockchain infrastructure. I also have a background in, um, working at a cryptocurrency exchange, and those 2 things in my head, um, actually kind of made a lot of this make sense. Uh, perfect segue into the talks about how the trading aspect of this works. Um, one of the areas that I learned while we were at QuickNode was the struggles that an infrastructure company has in meeting peak demand.

00:13:53 - Carl DiClementi

So what ends up happening is you have to have enough infrastructure provisioned to be able to support your peak demand, right? Unfortunately, peak demand is at only small intervals during the day. At all other times of the day, you are way over-provisioned. So what do you do with all that excess capacity, especially in the um, AI space, you are literally lighting money on fire for every second you are not serving tokens.

00:14:24 - Carl DiClementi

That hardware is super expensive, it depreciates in about 3 to 5 years, um, and if you're not serving tokens, then you're losing money. Well, um, that makes perfect sense in the concept of, hey, what if I could go and I could just make sure I'm always operating at, I don't know, 80, 90, 100% by going and selling, as in it's more of a push model than just hoping customers come to fill your gaps.

00:14:52 - Carl DiClementi

And that's what the market is good for, is that someone can come to the market who is a supplier and say, hey, I have excess capacity. I'm going to sell that excess capacity right now to be able to earn back some of my money and not be lighting my cash on fire. So that, I think, really opened up my eyes was, hey, What do we do when we're not monetizing our capacity?

00:15:17 - Carl DiClementi

Is we gave them an opportunity now to come and like, I call it almost sell, like instead of pay as you go, it's like sell as you go, right? Hey, I have excess capacity. I need users, I need usage. Well, you can just sell as you go on the grid.

00:15:39 - Anthony Campolo

Anyone, so what are the requirements to do this? Do you need your own hardware or are you just paying? How does this work? This is totally new to me.

00:15:48 - Carl DiClementi

Absolutely. Good question. There is a whole number of companies out there that have tried to approach this for people running local models on their laptop saying, oh, hey, I have this gaming rig set up. If I just put up Llama on there and try to serve that, and earn some money on it. That is not our target suppliers. There's a lot of risk involved in that around people being able to take your data.

00:16:16 - Carl DiClementi

What if you accidentally leak a key? What if somebody inserts a malicious tool call to pull your, you know, .env file? There's all sorts of risks that we've seen again and again in this space when it comes to that. And we want to make sure that people can run a business on the grid. We want to make sure that we are a reliable, scalable solution.

00:16:42 - Carl DiClementi

Obviously, there's even the reliability concern of some person running a model locally. So we prioritize being able to serve something for largest enterprise and meeting their demands. And the way that we approach that is the only suppliers that we're welcoming onto the platform right now are those that can operate at scale, that have a clear reputation, and that support zero data retention.

00:17:12 - Carl DiClementi

So that's really what it is right now. In the future, we maybe will get into allowing anybody to be a provider, but that's not the case at this point. We really wanna make sure that we can be the trusted solution for businesses to run on.

00:17:30 - Anthony Campolo

Okay, so it really is— the usage is just kind of like, like a share. It's like a— it's something very abstract that you're just kind of purchasing.

00:17:39 - Carl DiClementi

Yes, absolutely. And I think most users will never interact with buying and selling directly on the market. They don't even need to necessarily pay attention to the fact that there's the trading capability that's there, but it is there if somebody did want to directly participate. I actually played around with having my Hermes agent like spin up a little trading bot to be able to optimize my cost.

00:18:04 - Carl DiClementi

And the way that it does that is it detects when there are dips in the market and then it runs my scheduled tasks at that time, right? Because I have daily scheduled tasks I want my agent to run. I don't really care if it runs at 9 AM or 2 AM or 7 PM. I just want it to get done every single day.

00:18:22 - Carl DiClementi

And to me, if I can get it where it's gonna, you know, like buy the dip of the cost of inference, then that's great, right? Um, and it can run at a significantly lower cost. Some of the instruments, uh, they move within like 20 to 30% throughout the day, um, and that makes a lot of sense, right? Because think about the need from suppliers of kind of like leveling off their usage, right?

00:18:47 - Carl DiClementi

Um, I don't know if you've seen, but Claude Code and Claude or Anthropic and others have played around with like peak and off-peak pricing. And exactly right. And this is just an example of that, right? This is an approach for them to be able to handle that directly on our platform.

00:19:04 - Anthony Campolo

No, this is fascinating. And I'm now really fully seeing the implications of this because we are kind of now at the whims, subject to the whims of OpenAI and Anthropic and whoever to decide when they want to throttle usage, when they want to change the price of usage, you know, all of that. And it's very obscure and they don't communicate it very well.

00:19:26 - Anthony Campolo

And a lot of people are frustrated by this. So having the ability to actually see the fluctuations and then decide with that information when to do certain work tasks or not is really powerful. That's something, you know, I would like to do because I'm burning through freaking tokens all day. And then by the end of the day, I'd be like, a bunch of stuff I should have just like ran it overnight, like with fast mode off, you know, stuff like that, you know.

00:19:51 - Anthony Campolo

So, uh, no, this is, this is really cool, man. This is really, really cool.

00:19:55 - Carl DiClementi

Yeah. And there's also the aspect of, on our platform, you don't have to be paying attention to those fluctuations as much because we will constantly, you know, think about— there's, you know, 10 different supplier providers all changing their pricing throughout the day. Hey, which one is the right one for me to be leveraging? Well, you don't even have to track that. We're just going to route it to the lowest cost one.

00:20:19 - Carl DiClementi

Of course, that is with the stipulation that they are meeting the specification of the quality and throughput, things along those lines as well. We'll ensure that you get the latency minimum that the given instrument requires, as an example.

00:20:42 - Anthony Campolo

Word. Let's see, I was going to ask kind of about, so do you like, is there like a dashboard? Do you like plug it into your harness? Like how are people kind of like using this within their tooling setup?

00:20:57 - Carl DiClementi

Yeah, great question. So as of right now, our focus is really just on providing the best possible inference experience. Um, but what we have done is we actually have a brand new, like, usage dashboard that we've worked on that I'm really excited about that'll be, um, going live probably next week. Um, honestly, it's pretty standard experience of showing you, you know, the number of tokens and requests and, um, what instruments, uh, you're making calls to, what models are getting served, things along those lines.

00:21:29 - Carl DiClementi

It's a huge improvement from where we are today. But for the most part, the way that we work is very similar to any other platform that you've used, where you just get an API key, you plug it into your harness, and then it's transparent from there. You know, where I've actually found, and it was similar at QuickNode, like the people who never log in are actually some of your best customers.

00:21:52 - Carl DiClementi

That means that everything's going right for them and they're not like trying to look into any blips or anything along those lines. It's been really nice. Some of our customers are building some really interesting products, a lot of them at scale with things like data tagging or some of the most simple stuff where you just want to have something running in the background while you focus on your core benefit of a product and not really pay attention to, oh, well, that model has been deprecated, that price changed, that provider is down, whatever it might be.

00:22:25 - Carl DiClementi

And we handle all of that for you.

00:22:29 - Anthony Campolo

Yeah, I just created an account and got my first API call working within the time it took you to say that. So your onboarding is very good.

00:22:41 - Carl DiClementi

I appreciate that. We spend a lot of time paying attention to onboarding, so that's a big plus. And then you just take that API key and drop it into whatever provider you want. We support chat completions, obviously, which is what almost any harness uses. We also support Messages API and responses as well to be compatible with things like Claude Code, Codex.

00:23:06 - Anthony Campolo

So this is what you started off with is you basically create an account and then create an API key and then it gives it to you here, which I will not show. And then you can choose this curl request right here. We see how it's hitting, um, /chat/completions. So regular completions endpoint has the model, max tokens, message. Um, we still got roles in here, old school.

00:23:32 - Anthony Campolo

Um, and then you can also do Python or JavaScript and these examples. And then only— so you have basically messages one or completions one, it looks like.

00:23:46 - Carl DiClementi

We also have responses not yet documented, not yet shown in there, but if you really want to send it to responses, you can do that too.

00:23:54 - Anthony Campolo

Yes, I was thinking that's going to be the thing you're going to really want next. So cool. Yeah. So you want to kind of just look at the dashboard a

00:24:02 - Anthony Campolo

bit?

00:24:02 - Carl DiClementi

Yeah, yeah. So what we've done is we've kind of given people who just start one level of rate limit. We're kind of overhauling how that looks right now to make it a little bit clearer of how many requests per second that you can actually get. Right now it's a little opaque there. It kind of just shows based on your balance that you can increase that multiplier.

00:24:26 - Carl DiClementi

But basically the way that it works is you get 3 concurrent requests on a free account, and then as you scale up your balance, you can get all the way up to 400 concurrent API requests. So that's kind of what we're talking about there. That'll be much clearer very soon. And then we walk through the different instruments as we discussed here. So, one thing I didn't show— oh, they talk about the lab latest, right?

00:24:50 - Carl DiClementi

We did walk through that. So you have all that information as well. You can even click the arrow on the top right of your screen.

00:25:00 - Anthony Campolo

The right thing

00:25:01 - Carl DiClementi

to the right.

00:25:04 - Anthony Campolo

Sorry, click what? You lagged for a second. I didn't hear what you said to click.

00:25:07 - Carl DiClementi

Sorry, the arrow all the way at the right of the table.

00:25:10 - Anthony Campolo

Arrow, the arrow. Yeah, I missed the word arrow.

00:25:12 - Carl DiClementi

And you can scroll over and you can see all the information about what modalities are supported and the context window, max input, max output, things along those lines. And yeah, that's kind of the— kind of like our models table. We have an ability to kind of like pre-buy and buy in bulk Some of our customers have seen like up to 30+% discount by doing that.

00:25:39 - Carl DiClementi

What we do is we go and we work with all the suppliers that are already our partners, and we can go ahead and find you the best price. It's usually for people who have more specific needs, right? Let's say they want to keep it, I don't know, data governance of it only being in the EU, right? Or they need an ISO-certified provider, whatever that might be, those users are able to buy at scale through there.

00:26:07 - Carl DiClementi

Then we have standard getting started, very similar to the component you were just showing. But let's say you want to integrate with any existing harnesses that you're used to using. We have kind of like one-click or some documentation there about how to—

00:26:20 - Anthony Campolo

I've never heard of some of these. I've never heard of KiloCode.

00:26:23 - Carl DiClementi

It just got acquired. Yeah, I think all of these platforms, all these coding harnesses, little by little are really getting acquired pretty quickly, which is interesting. I think they're all going to get snatched up. And then kind of just walk you through some of our best practices, make sure that you set max tokens, things like that. That makes a huge difference that most people don't really think about.

00:26:50 - Carl DiClementi

And what else? Just some instructions on how to do streaming. And we also have a tool for launching an agent with kind of deploying a fresh agent. So let's say you want to deploy OpenClaw, we kind of give you a CLI to be able to do so really quickly and already has the grid configured in there for you.

00:27:11 - Anthony Campolo

Interesting. So agency is saying the grid is building?

00:27:14 - Carl DiClementi

Yeah, yep, yep, exactly. So if you go ahead and click at the top right of that agency.thegrid.ai. Yep. So we just launched this recently and it allows you to be able to click through through and select the, um, the agent. And then where you are—

00:27:35 - Anthony Campolo

that's it. That OpenClaw has 340,000 GitHub stars. Yeah,

00:27:39 - Carl DiClementi

probably even more now. Um, I think we probably need to update those numbers. It's insane how quickly that grew. Yeah, I really love to see some data on, um, what it is, how, how their usage has changed today.

00:27:51 - Anthony Campolo

It sounds like you're, you're big on Hermes. Which you've been calling her Mez. I don't know which one it actually is.

00:27:56 - Carl DiClementi

Probably not. I just like to think of it because it makes it sound like it's luxury.

00:28:03 - Anthony Campolo

That's

00:28:05 - Anthony Campolo

funny.

00:28:06 - Carl DiClementi

Yeah, to be honest, I kind of was using OpenClaw for a while and then I went and did an update and it lost everything and started from scratch.

00:28:21 - Anthony Campolo

As a memory system, that seems kind of like a big deal.

00:28:25 - Carl DiClementi

Yeah. So essentially I said, well, this is my perfect time to go and try out the next thing because really, you know, they're all very comparable in what they can accomplish. So what I switched to is Hermes or Hermes, however you want to pronounce it. And as of today though, I feel like I should be trying out Grokbot or The Muse from Meta, um, all are getting a lot of attention right now on socials.

00:28:53 - Carl DiClementi

Uh, I personally am just a little bit, let's say, hesitant to give Meta access to anything.

00:29:01 - Anthony Campolo

No kidding. Yeah, yeah, I haven't really dove into these, the higher-level agents, too much. I wrote a blog post comparing OpenClaw to Hermes, so I kind of got like the, the 101, but I'm not really using any of them in my day-to-day. I'm still just using, you know, just straight up Codex or Claude Code or something like that. But I can see how they would probably be useful if I put the time to actually configure them correctly and get them built into my workflows.

00:29:28 - Anthony Campolo

Grokbot, I also am very curious to check out. I've been hearing a lot of good things about that. I downloaded the desktop app, got it set up, but I haven't really done anything with

00:29:37 - Anthony Campolo

it yet.

00:29:38 - Carl DiClementi

You know, the most interesting thing that I have going on is I use my Hermes to just do some consistent probing and testing of our APIs is just like, oh hey, set up a request that runs every X time and let me know if there's ever increased latency or anything along those lines. Let me know if price goes out of extreme this way.

00:30:02 - Carl DiClementi

Just writing those scripts and making sure that they're alive and notifying me via Telegram if it ever happens. That's something that in the past would take a lot of work to put together and modify and maintain. And it just does all of that for me, which is pretty convenient.

00:30:17 - Anthony Campolo

Yeah, it's like a much more in-depth health check. Like, you're not just pinging the API like, are you on? You can like kind of do a whole set of things and then get consistent like reports and notifications and stuff like that.

00:30:28 - Carl DiClementi

Yeah, yeah. Which, to be honest, I think that, you know, the big labs, they saw where this was going. They saw The, they saw how popular the OpenClaw was, right? And I think they decided that they needed to play defense a bit there. And then all of a sudden they all started launching these, you know, like a side to it or scheduled tests or whatever it is, right?

00:30:59 - Carl DiClementi

I think that's a huge reaction to, you know, OpenClaw's dominance or rise.

00:31:06 - Anthony Campolo

Yeah, definitely. I do wonder though, like, to what extent OpenClaw is being heavily used versus heavily talked about online. There's, there's always that kind of question in my mind of like, every— like, I was really interested in this and I was kind of like kicking the tires on it, but like, how much is the actual pickup on that? And that's where something like Grokbot can come in, because with Grokbot Even if you are just like a regular kind of AI user and you don't wanna have to go through the whole challenge of like, you know, cloning a Git repo and stuff like that, you can kind of start to get that functionality.

00:31:41 - Anthony Campolo

So I can see how they could kind of swoop in and pick up a lot of people who wouldn't have wanted to go through the trouble of like going high, like really deep into something like OpenClaw.

00:31:52 - Carl DiClementi

I completely agree with you. One of the things that was a huge challenge for me in using OpenClaw or any of the alternatives was it trying to go to any websites, it would just always get blocked or run into issues. I'm a huge fan of OpenAI's ChatGPT app computer use. It is incredible. It is. I mean, I haven't used Cowork in a while.

00:32:17 - Carl DiClementi

I tried it a while back, but it was super slow to do anything in the browser. It would go to a website, wait 3 seconds, take a screenshot. Oh, it loaded, or it needs to scroll down, take another screenshot. It was really slow and I've been using, I don't even know, do you call it Codex or ChatGPT computers, whatever it is, right?

00:32:41 - Carl DiClementi

I always get confused. Like, is the app—

00:32:44 - Anthony Campolo

now there's GPT Work also, or ChatGPT Work. Yeah.

00:32:49 - Carl DiClementi

Yes, that. That's great

00:32:51 - Anthony Campolo

because we actually had a previous guest, Tracy Lee. She talks a lot about, she's like, super bullish on ChatGPT work. It's like her go-to for everything. That's her main driver.

00:33:03 - Carl DiClementi

Really? How is it different?

00:33:06 - Anthony Campolo

Um, you know, I don't really know. I haven't, I think it, well, I think it's because it has the computer use part. Like, I think that's pretty much it.

00:33:12 - Carl DiClementi

So you can use computer use in a regular ChatGPT chat. You can use it in a Codex chat. So it's, to me, I think the lines are blurring and I actually just saw today. That I haven't played with it, but I think Anthropic just merged Cowork and Claude, which I think— and Claude Code. I think I just saw that they're merging it all, which makes sense to me is like at any time I could be doing any of the tests.

00:33:39 - Carl DiClementi

I don't know why I need to like switch to a different app for it. I understand maybe if you want the CLI version, then obviously it's different, but yeah.

00:33:48 - Anthony Campolo

Yes, I'm looking at ChatGPT Work. Right now. And it looks like this is where you have like projects. So that's not really a thing in regular chat. And then all the plugins is through ChatGPT Work. So if you want to connect to like your Google Drive, your Google Calendar, your GitHub, stuff like that.

00:34:07 - Carl DiClementi

I have projects in ChatGPT Chat.

00:34:10 - Anthony Campolo

Really? Interesting. Well, they, they may, they service them more in the UI.

00:34:14 - Carl DiClementi

Okay.

00:34:15 - Anthony Campolo

Yeah, no, honestly, I, yeah. I really do not know what actually makes ChatGPT work different. It's a total mystery to me.

00:34:22 - Carl DiClementi

Yeah. So, I mean, it is, you know, I think it's all again, a reaction to the whole OpenClaw thing. And I agree with you that it felt very buzzy at the time. And there is a lot of people who, I don't even, I went to a ClawCon event. And there was a lot of people there that hadn't even used it yet.

00:34:47 - Carl DiClementi

But still, I think that's a good thing that you bring people in and people are interested, right? And you get to show the art of the possible. But there's obviously the whole question of, like, hey, how is the general public going to adopt this? I saw a stat that only 1% of humanity has ever paid for AI. And like, I think it's only like, what was it, like less than 20% have ever even used it, which I don't think that's real, right?

00:35:19 - Carl DiClementi

I just don't think people kind of like understand what using AI is, right?

00:35:25 - Anthony Campolo

Yeah, if you ask how many people like have used ChatGPT, I feel like, because at this point, isn't ChatGPT like one of the top 10 most visited websites in the world? It's like, how can that possibly be the case?

00:35:37 - Carl DiClementi

So they haven't done a Google search?

00:35:39 - Anthony Campolo

Yeah, exactly. Well, yeah, I would say I wouldn't consider doing a Google search, even though you do get the AI, you know, overview. You can't then, like, you can go then and like respond to it, but most people aren't going to. So I think that's still, that's still a pretty distinct experience, I think.

00:35:56 - Carl DiClementi

Yeah, I just think the idea of using AI sounds more complicated than it is. Right. I think the more we get away from somebody having to think about the fact that they're using AI and it's just an annoying marketing thing right now where everyone's like, oh, we have AI in our app or app power. But it's just, you just do that because that makes it a better experience and you don't have to brand it for that.

00:36:21 - Carl DiClementi

Right. It's like on Google Photos, you hit like enhanced photo and it makes it look better and you hit save. Right. You don't need to know that that's AI or some other algorithm. It's just, I think it was like The Verge. They were like, just replace the word AI with software anywhere and it makes sense.

00:36:40 - Anthony Campolo

Yeah. And it'll also get, it'll also make you less offensive to people who are like bringing, taking on anti-AI as like an identity, which from, as someone from the blockchain world, I definitely understand that. I know many people who took on anti-blockchain as a personality as well.

00:36:58 - Carl DiClementi

Yeah, and unfortunately, like blockchain and crypto, AI is now becoming political for unnecessary reasons. And it's something that I really hope that we could move past. Maybe it's just something that is going to be stirred up up until midterms and then we can move past it.

00:37:18 - Anthony Campolo

Yeah, I think it's going to continue to ramp up up until something leads to. There'll be some sort of catalyzing event, I think, that will push people towards like, oh, we gotta do something. And then some legislation will be passed. And yeah, I don't know how that's gonna play out, but I'm not super optimistic about it. So we'll see. I also think that there's always going to be the open source options and there's always going to be, you know, other providers in different countries.

00:37:50 - Anthony Campolo

Offering different things. So it's really not going to be possible to like put AI in a box. Like there'll be ways to constrain certain companies in certain jurisdictions, but it's not ever going to be like we write a law that's going to stop, you know, the growth of intelligence. It's just not really possible.

00:38:12 - Carl DiClementi

I don't think— I, I think the reason AI is different is the fact that whoever moves the fastest then has the model to move even faster, right? It's, it's like an exponential growth that most other technologies weren't, right? Where you're just— it's a multiplier effect, right, into yourself, the AI researcher. And then once you have the AI researcher, right, then it can just go like this.

00:38:46 - Carl DiClementi

So it's interesting. It's also, I'd say, you know, it's easy to think of scary scenarios, and I don't know if that's just because of movies and we've all put ourselves there, but I also think that, you know, you could look at it and say, like, we've all seen this movie before and it didn't go well. So, you know, it's where everybody ignored that scientist that was like raising their hand.

00:39:15 - Carl DiClementi

But there's also always been that scientist raising their hand saying things are going to go horrible and things worked out well. So it's a really careful balance, I think.

00:39:24 - Anthony Campolo

Yeah, you probably don't know this, Carl, but I actually first got interested in tech and coding in around 2016, specifically because of AI risk. I got really fascinated by this whole question of AI existential risk. And I lived in the Bay Area and I knew a lot of people who were into LessWrong and Um, that whole world. So I got like very into this whole like Eliezer Yudkowsky kind of AI is going to kill us all type thing.

00:39:50 - Anthony Campolo

Or it was either AI is going to kill us all or AI is going to create superintelligence, which will then lead to us being able to like upload our brains into the Matrix and then we'll live forever, you know. So it's a cult. I'm just going to say that straight up right now. Um, these people do not have touch with reality. And they should not be allowed to make any decisions.

00:40:11 - Anthony Campolo

I know this because I used to be one of them. And the more we can figure out ways to persuade people that this is not actually going to happen, the better, because it's— there's so many things that AI can be used for, for, like, you know, negative, bad ways that are not that, that we should be worrying about instead. Like, the, the, problems that could arise from people misusing AI is so much more likely to actually play out than this, like, kind of superintelligence takes over the world type thing.

00:40:47 - Carl DiClementi

Absolutely. I mean, though, I remember back in, like, 2017, I was working— it was kind of my first time getting involved in the blockchain space. I wasn't all that interested in crypto. It was just about using the blockchain to ensure data provenance and that data is valid and hasn't changed, which is actually like the core thing that blockchain is good at. Um, and at the time we were talking, we were showing these like early clips of, um, deepfakes and how like, oh, one day deepfakes are going to be such a problem.

00:41:23 - Carl DiClementi

And it seems so abstract at the time, but now you really have like real-time convincing deepfake video, audio. Like, you can, you can get a video call from somebody and be convinced that it's some— that it's a person that it's not. And that's scary, where we all actually have to have like a safe word set up with our families, because that's the thing, right?

00:41:49 - Carl DiClementi

If I'm convinced sometimes, like, oh, is that video real? There's no hope for, you know, other generations.

00:41:57 - Anthony Campolo

Yeah, I think it was Minimax H3 was one that came out a couple months ago. And I think even I think I can't really tell anymore now if it's like a very short clip and it's like, you know, nothing too complicated about it. You're like, I'll see a clip like that and I'll be like, I know this is AI just because I know because the context it's like being shared in.

00:42:18 - Anthony Campolo

But just if I saw this clip out of context, there's really no way to tell at this point for for some of it. It's crazy.

00:42:25 - Carl DiClementi

And that's— and this is— I think I've seen this, people like to use the term like, this is the worst it's ever going to be, right? It's only going to get significantly better. The, um, how some of the best ways to tell is if there's some like unrealistic physics or things like that, like they'll get that figured out, right? They'll solve that.

00:42:45 - Carl DiClementi

Um, have you even— I love the SVG ones. Have you seen the new— there's a new benchmark is Will Smith eating spaghetti, an SVG rendering of it.

00:42:56 - Anthony Campolo

That's awesome. I did see Karpathy had shared how he was using SVG videos as a benchmark, and I think he created like the— he did like the first scene of Lord of the Rings, I think, as an SVG video. Will Smith eating spaghetti is obviously the classical video benchmark. And I've been tweeting about that for years. So yeah, no, that's really cool.

00:43:22 - Carl DiClementi

The pelican on a bicycle is always classic. I forget his name, but

00:43:27 - Dev Agrawal

Simon

00:43:27 - Dev Agrawal

Willison.

00:43:29 - Carl DiClementi

Yes. Yeah.

00:43:31 - Anthony Campolo

Did you have them? Do you want to jump in here, Dev? Do you have any questions?

00:43:35 - Dev Agrawal

Yeah, I mean, I guess I'd like to hear a bit about the kind of suppliers you have. How do you find more suppliers? How do you kind of grow the pool of compute that's available to people using, like, besides just incentives?

00:43:55 - Carl DiClementi

We actually have suppliers lined up knocking on our door to participate. I'm regularly fielding emails, and right now the focus is really around us making sure that we're able to appropriately vet them. There is a really good article, the, um, or study recently, I'll see if I can grab the link, of kind of the risks of using an untrusted provider. And we're really serious about that.

00:44:27 - Carl DiClementi

So that's kind of been where we're at is we focus on people who have a reputation, people who, that we can vet regularly. We have all sorts of let's say, secret shopper stuff happening where we're testing the different suppliers, we're validating that there's nothing happening, like they're not manipulating the prompts or anything along those lines. So really it's about vetted, trusted suppliers that can serve at scale.

00:45:01 - Dev Agrawal

Got it. Yeah, that would make sense because I have seen some recent tweets that, uh, about like how like find cheap inference is like kind of a gateway to scams these days. Um, there's a lot of shady stuff that people have been doing. I think, uh, yeah, Dax has been talking about it a bunch.

00:45:21 - Anthony Campolo

Yeah, exactly. I'll grab those tweets while you're going.

00:45:25 - Dev Agrawal

Yeah, so, so yeah, it makes sense that the problem isn't that how do you find suppliers, but it's more like how do you actually make sure that they're going to follow the policies that they're setting?

00:45:40 - Carl DiClementi

Exactly. So we've already set up vetting where we run, we do Secret Shopper and we run benchmarks validating that they're serving the quality that they're supposed to be running. And then we also have, since we've seen some of these studies, begun vetting to make sure that there's no, let's say, malicious tool calls or anything along those lines happening. Other processes as well, I don't necessarily want to say exactly what they are because that would maybe make it easy to circumvent, but we're just making sure that it's constantly possible for, you know, constantly safe for our users.

00:46:22 - Dev Agrawal

Yeah, of course, those policies like have to be like constantly evolving. It's like, it's like an area of cybersecurity essentially. Yeah, this is the tweet I was talking about.

00:46:32 - Carl DiClementi

Yes. Yep. And, you know, to be— this is why we talk about the market part and we make it very clear in the product and things like that. Um, it's a trade-off because it does sometimes feel a little overwhelming and confusing to tell people about this whole market. It's like, oh, there's like an exchange, what's going on here? Um, Most users, they never have to worry about it or know that it exists, but we do surface it to make it clear this is why we can give you beneficial pricing.

00:47:02 - Carl DiClementi

So that was a big part of it.

00:47:05 - Dev Agrawal

Yeah, you're muted, Anthony. Anthony, you're muted, or you have been muted for that.

00:47:16 - Anthony Campolo

Sorry, I was just saying, yeah, no, it's So you're addressing this part here. If you can't explain how someone got for cheap tokens, you should avoid it. And so you have the ability to tell people, hey, this is how the pricing is, where it's coming from, you know.

00:47:30 - Carl DiClementi

Exactly

00:47:32 - Carl DiClementi

right.

00:47:32 - Dev Agrawal

But at the same time, isn't like there, there like a GPU shortage going on? So it feels like there should also be a shortage of suppliers.

00:47:41 - Carl DiClementi

So there is, there is a shortage of GPUs. Well, so Yeah, let's actually start with the specific answer to your question. Yes, there's a shortage of GPUs, but people buying— people wanting to buy GPUs and all of the GPUs that have been sold serving at their max capacity are different things. So let's start there, right? But then also all of those people that do have the GPUs that say they are serving at their max capacity at every— any given part of the day and they want to buy more, That doesn't mean at all points throughout the day they're operating at their max capacity, right?

00:48:22 - Carl DiClementi

And that's what we allow, is that during those troughs, right, in usage, that they're able to monetize that. And then it's not only on the low side, it's also when there is peak usage, they can increase the price, right? And they can go above and beyond what they may be charging direct customers, or let's say on open routers. The reason why they want to do that through us is because it's actually, in some worlds, people would see it as a negative thing that the price moves throughout the day, right?

00:48:55 - Carl DiClementi

Some customers, they want, in some cases, some customers, they say, hey, I want a fixed price. I want to know what it's always going to cost. And that's fine, right? And that, like, we're just not the right platform for you. And I think that's an important framing. Right, where we say, hey, we are the platform where price moves. People know what they're going to get when they come to us.

00:49:16 - Carl DiClementi

If someone's going direct and they're expecting a flat price and it's moving throughout the day, that could really anger them, right? So it's just about like, hey, it's our brand positioning. This is what you get when you come to The Grid.

00:49:30 - Dev Agrawal

Got it. Yeah, that makes sense. So my next question is about like the other side of the products. But so before that, I'll let Anthony—

00:49:39 - Anthony Campolo

Yeah, I mean, I just want to go back to like that, that like comparisons like energy trading, like that just really seems like a good comparison here because, you know, people will need to use electricity at different amounts during different points of the day. And so that can lead to companies because they just have the fixed infrastructure, you know, they have however many power plants they have and They need to figure out what to do with the energy.

00:50:04 - Anthony Campolo

And, you know, for me, I didn't know this at the time, but I learned this later. Like, as a kid, I don't know if you guys know about the Enron scandal, but like Enron was a company that basically did like energy trading along with other stuff. And they realized that they could basically jack up the price and create kind of like artificial scarcity.

00:50:26 - Anthony Campolo

And this ends up leading to like rolling blackouts happening throughout like California because of all these like weird issues that were caused by how they were trying to manipulate the market. I remember as a kid, like, you know, just the power would just randomly go out and then like for a couple hours, you know, no one on the block would have power.

00:50:40 - Anthony Campolo

They eventually will like come back and this would happen like every couple months or something. And it wasn't until like later in life that like I learned that's not normal. Like when I grew up, I thought like everyone just has power outages all the time. It's just like a thing. Like, no, actually that's because Enron was manipulating the energy markets and then they ran out of energy for certain people and then had to like, you know, do this rolling blackout strategy.

00:51:01 - Anthony Campolo

I was like, man, that's so messed up. So like, I worry about stuff like this happening with like AI. And so like, it's good to actually have companies that are like, kind of transparent about like this process and how it's going. So stuff's going to happen on the backend anyway, like people are going to, you know, just trade because they have, you know, the knowledge to make deals and figure out how to move around the capacity and stuff.

00:51:27 - Anthony Campolo

So this kind of makes it you know, more transparent and more open to the average person. Yes, yes, you can ask as many questions as you want. Type them out, buddy.

00:51:37 - Carl DiClementi

Yes, for sure. And also that I watched a— I didn't know about Enron. I just like remembered hearing— I thought it was just something like accounting scandal, right? And then somebody told me to check out a documentary. I think it's on Hulu about it. And I— exactly what you're describing. Where they were like calling up these different, like, power, you know, power companies and being like, hey, like, you should have an outage today.

00:52:06 - Anthony Campolo

Yeah, exactly.

00:52:10 - Carl DiClementi

Crazy.

00:52:13 - Anthony Campolo

I think Fuzzy might ask a few questions, but while he's doing that, Dev, you should transition us to the thing you wanted to ask next.

00:52:21 - Dev Agrawal

I think we have Fuzzy's first question.

00:52:25 - Anthony Campolo

I mean, I think this is a joke. Not a joke, but it's a satirical comment. I should say not a question.

00:52:33 - Dev Agrawal

Got it. My other question was about— you mentioned that the configuration that the user has is— or actually, I was also reading it from the doc. There's text or there's chat agent and code, and then there's like a regular, uh, premium, and max. I might have the actual words wrong, but that's kind of how you have— how you have broken it down.

00:52:59 - Dev Agrawal

Um, how did you arrive at these configurations, and what are some other options that you considered?

00:53:06 - Carl DiClementi

Yeah, great question. So we actually see the future of this going way beyond what we offer today, um, and very detailed, all the way down to

00:53:22 - Anthony Campolo

Oh no, we lost Carl entirely.

00:53:25 - Carl DiClementi

Oh, can you hear me?

00:53:27 - Dev Agrawal

You're back.

00:53:27 - Anthony Campolo

Yeah, you dropped out for about 10 seconds there.

00:53:30 - Carl DiClementi

Um, okay, so, uh, the— let me shut my door, hold on one second.

00:53:41 - Anthony Campolo

Also, I saw he shared a Thing in the— I should share here.

00:53:45 - Carl DiClementi

Yes, sorry about that. So the— we, we can— we really foresee this expanding significantly to, uh, we call them task-based. Oh boy. Uh, things all the way from, um, PDF, um, data extraction like OCR to legal to, um, accounting to all the breakouts that you can imagine. What we started with was the most popular use cases for AI, and that's just because with a market you want the most liquid markets possible.

00:54:22 - Carl DiClementi

And the way that we like measure which ones that we want to launch are really around customer demand as well as where we can have a reliable benchmark for quality. Right now, artificial analysis, they go way out of the way to set their intelligence index, which is across all different benchmarks, right? And which is something that I really like. If anyone gets saturated, they replace it out.

00:54:49 - Carl DiClementi

And obviously there's a— I don't know if you guys saw, there was a whole like thing around their intelligence index when Astra came out and like they quickly made some updates to it. But it's good because they continue to stay on top of that. They also have coding index and an agentic index. So those were the natural 3 for us to start with, and we continue to work on expanding them out from there.

00:55:14 - Carl DiClementi

But those are kind of, I think, 3 use cases that cover most users right now. When it comes to the— then, sorry, go ahead, Dev.

00:55:23 - Dev Agrawal

No, no, it's fine. Go ahead.

00:55:25 - Carl DiClementi

Then when it comes to the 3 different quality tiers, the way in which we went about that was actually taking a look at what most of the AI providers do today. A lot of them have like a low, medium, high type thing. So it's a natural kind of mental model that most users already have, right? For me as a product guy, I'm always thinking about like, so it's very complicated what we do and how do we make it the most like simple mental model possible while still being differentiated.

00:55:57 - Carl DiClementi

So that's kind of how we went about that was trying to ensure that we had a very clear mental model of how this would work. Do you want fast? Do you want super smart? Or do you want balanced? Right. And that's how we came up with the 3. The naming we played around with. We did Max, kind of like your iPhone, right?

00:56:18 - Carl DiClementi

The iPhone Max. And then instead of Pro, we went Prime and the standard version.

00:56:26 - Dev Agrawal

Nice. Okay. So you have the one dimension, the chat agent and code, but you're saying that that's kind of like the 3 starting points. You can eventually have, uh, these kind of like presets for really any area that you can reliably verify using some sort of benchmarks. So like, and then maybe you can have like fine-tuned models being served there as well, not just like general LLMs.

00:56:53 - Dev Agrawal

It's actually maybe a great way to do that because you can have a bunch of fine-tuned models and someone who's using them doesn't have to think about which models they should be picking, how people are actually fine-tuning these. They just say that, oh, this is the model that I want, and the platform can automatically find the best one regardless of who, where, and how.

00:57:19 - Dev Agrawal

Is serving it to you. So it's not just compute, it's also what model is being served can be routed to.

00:57:28 - Carl DiClementi

Exactly, 100%. And I think once when we do start to really scale out the number of different instruments, as we call them, then it makes sense to also launch maybe an instrument that's like an orchestrator instrument, a model that chooses which model you might want to be routed to. That way somebody can just make that one call to there and it'll route to another.

00:57:56 - Carl DiClementi

That way you don't even have to think about it. Is this question most appropriate for an agentic task? Is it just a chat or are you doing image generation, for example? Which is a question that I think, Anthony, you asked earlier was around modalities. Our MAX instruments accept image input, None of them are image out yet, but we've also looked at image and video out.

00:58:21 - Carl DiClementi

It's just that's a whole new set of APIs and infrastructure to build out. It's coming. It just hasn't been our priority at this point.

00:58:34 - Dev Agrawal

Cool. Is there any plans of letting people define their own criteria? Something like instead of a— like however you, however you currently decide what, what model should be served for chat or for code, like, uh, can I eventually write some sort of policy of my own of like this, these are the verifiers, or these are, this is, these are the evaluators or the criteria that I want to use for this setting and now use that?

00:59:05 - Carl DiClementi

It's a great question. So the way that commodities markets often work is there is a committee that actually ends up deciding on the specifications. And it's not just any one party that's involved. There's kind of 3 different parties, right? Buyers, sellers, and the exchange itself. Right. So right now we would be considered the exchange. We're a market, not an exchange, because there's a legal association with being an exchange.

00:59:43 - Carl DiClementi

The— right now it's just us defining it, but as we scale into the future, we actually want to step back further and further and allow these suppliers, the sellers, and the buyers to be the ones weighing in on what they're looking for and being the ones to decide what the contract specifications look like. Like, right? The specification of, hey, here's what the benchmarks are that would allow something to— a model to be served.

01:00:15 - Carl DiClementi

That is something that we would love to get to as soon as possible. But it is just so much work to define and stay up to date on what these benchmarks are and how quickly the models are changing. And all of that, that right now, even if I did want to welcome in users to doing it, I don't think anyone would want to spend all that time on it.

01:00:37 - Carl DiClementi

It's like half of my day. It's pretty incredible. So, yeah, I would say that we want to welcome in others, and anyone who is interested, I welcome them to come participate. But in the meantime, I will continue to fill that gap, or we will continue to fill that gap.

01:00:55 - Dev Agrawal

That's completely fair. Um, at the same time, I think when people— when the folks were building Vim or Neovim, they probably did not imagine how many hours people will sink into tweaking their Neovim configs. So you might be at— I, I'm just saying that you never know. There's going to be people who are gonna try to optimize the heck out of that if you let them.

01:01:19 - Carl DiClementi

Yes, I, I believe that. Um, so the, the core point, I think, will be that, you know, you're kind of giving me an idea. Maybe we have some, like, Grid Labs or something like that, where people can play around with creating one, and people can, like, vote on that. The challenge becomes that having it dynamic makes it where it is no longer a, like, fixed contract specification.

01:01:48 - Carl DiClementi

That can be traded on a market. It's too dynamic. Um, it would have to be more like, hey, okay, this is what it is now, and then, hey, a change is coming. The sellers have the ability to adapt, things along those lines. Um, thanks a lot, Fuzzy.

01:02:04 - Dev Agrawal

Got it. Okay. And so for like small piece of context, I was mostly thinking about this in terms of like personal use. Like, I don't want to build a policy that then like other people use, ideally. I mean, at some point maybe, but more often, like, I want it either for personal use or maybe for like organizational use. Like, an organization should probably be able to customize it a little bit, and they probably have more resources to do it rather than like an individual person.

01:02:33 - Carl DiClementi

Yes, I think that's totally fair. Um, and we hope that the extraordinary amount of work that our data science team puts into this gives you something that would allow you to not even have to be thinking about figuring out how to set that. One interesting thing is that there are— I wonder about this, I'd love to ask this question to you guys and to the audience.

01:03:00 - Carl DiClementi

What percentage of tokens that are consumed today do you think are just people running evaluations and benchmarks?

01:03:11 - Anthony Campolo

I

01:03:11 - Anthony Campolo

mean, I would say from the labs, a lot. From the average user, not a lot. That would be my guess.

01:03:22 - Dev Agrawal

Yeah, I think if, if I count, like, if I think of like every bit of GPU capacity that the labs have and what's going on, I would think that maybe 90-95% is like evaluation and training, or maybe— and maybe like 5 to 10% is inference for like commercial public. Uh, outside of labs, it's probably like the other way around.

01:03:50 - Carl DiClementi

I just feel like something like OpenRouter, I, I just feel like everyone, when a new model comes out, just like jumps to it on OpenRouter to use it, which is great, and it's an incredible tool for that, right? It's like, hey, I don't have to change anything other than the model value in my API, right? It's like, oh, and then you're all of a sudden trying it out, right?

01:04:11 - Carl DiClementi

Which is great. I just wonder if that's propping up our numbers of the number of tokens everyone's consuming, is that every time a new model comes out, everyone rushes to go try it. They're not even actually doing anything productive. They're just creating a new benchmark and trying something interesting.

01:04:30 - Dev Agrawal

Yeah, I can also see this.

01:04:31 - Dev Agrawal

Evaluation in that sense, trying out something new, not running benchmarks or—

01:04:37 - Anthony Campolo

I

01:04:39 - Anthony Campolo

can imagine when a new model comes out, someone vibe codes their latest app that they had just built before. Now let's try it again with the new model. That's not really productive at the end of the day. And this is probably, you know, a perennial problem in tech is developers going off and doing something that they find interesting, not necessarily because it's actually useful or provides any value.

01:05:06 - Anthony Campolo

But on the whole, all that experimentation, I think, eventually leads to the breakthrough. So, you know, people want to burn tokens, you know, that's their business.

01:05:17 - Carl DiClementi

I'm doing

01:05:17 - Carl DiClementi

it on the

01:05:18 - Carl DiClementi

grid.

01:05:23 - Anthony Campolo

Awesome, man. Um, uh, anything else? Any other big topics we want to hit before we start wrapping it up? Or like, you know, things coming up, you know, what's next for the Grid, stuff like that?

01:05:33 - Carl DiClementi

Yeah, I'd say that. So a couple things that we're about to launch is a new usage dashboard to make things a lot more easier. How can we farm tokens? Um, there is no farming of tokens, unfortunately. The only thing that you're able to do on the grid that is along those lines is track the market and buy when the price dips, and then you'd be able to resell it.

01:06:00 - Carl DiClementi

But since we are not an exchange, you can never withdraw any cash. The only thing that you can do is offset your costs. So let's say you had spent, you know, $1,000 on inference and then you sold $100, then your bill is only $900. Right? So the only thing that you can do is offset your costs. You cannot, you know, spend $1,000 and then take $100 in cash off the platform.

01:06:24 - Carl DiClementi

So that's a really important distinction. You can set limit orders, things like that as well, to be able to catch price movements as well. So yeah, and then the biggest thing actually is around that topic that we're about to launch is a really nice insight into any activity that you have on the platform, how much your net spend was, things along those lines.

01:06:51 - Carl DiClementi

And we are working on some really fun marketing stuff internally as well that probably most users won't see. But this is maybe a fun topic that you can hit on with some of your guests is what interesting things are you doing from a marketing GTM perspective? We've— one of our engineers has built out a pretty incredible tool for us to just run any type of GTM campaign that we want to test out, where it spins up a Google Ads campaign, creates the associated landing page, has A/B testing all through it and tracking and analytics to be able to see what works.

01:07:29 - Carl DiClementi

It's a big thing for us as a new company. I think a lot of people, when you're starting new, you're trying to figure out what messaging works, What audiences you want to, you know, you resonate with most, things along those lines. And this is kind of just a way for us to do that at scale.

01:07:45 - Anthony Campolo

Awesome. Yeah, it's funny when I first booked you for this and I was kind of looking at the company you're working at, like, I thought this was going to be basically just like a cryptocurrency exchange, but for like trading, like AI usage. And so when you said you can't actually withdraw money, I was like, okay, I had like the very wrong mental model coming into this conversation.

01:08:05 - Anthony Campolo

So, um, this is really cool, man. Like, um, I'm glad I created an account because I'm gonna definitely, uh, check this out and use it. So, um, here's your, uh, X link as well for people who want to check you out on X. Carl D, C-A-H-L-D-E. Some fun stuff there. Carl D.

01:08:26 - Carl DiClementi

Exactly. And I can do it. Who's listening, um, shoots me a DM on there. I will load you up with an additional $20 onto your account to try out the grid. So feel free to hit me up on X.

01:08:38 - Anthony Campolo

Awesome, man. Cool. Well, thanks so much, Carl, for being here. Thanks, Fuzzy and Nicola, for asking questions. As I mentioned at the top, I'm going on vacation for the next month. I'm going to be doing a cruise in Europe, which will be very fun. So you can hear all about that when we get back. Dev, what are you doing for the next month?

01:08:58 - Anthony Campolo

Anything fun?

01:09:01 - Dev Agrawal

I, I think we were just talking before we hit live that, uh, um, I kind of went into like a rabbit hole since I saw the TypeSafe AI announcement yesterday, and I actually got early access today, so I'm just playing around with it. Um, I actually, I built and just published to GitHub a little code review bot that uses Jev to like, like review, review code for against a bunch of different metrics.

01:09:33 - Dev Agrawal

So it's like kind of like my stage 1 of playing around with it. I was literally talking about it, uh, before this. So yeah, I'm gonna be playing around with that a lot more and, uh, doing more interesting things, combining it with LLMs, with deterministic checkers. And yeah, it's really exciting to be experimenting with all of this these days.

01:10:01 - Anthony Campolo

Awesome. We'll definitely have to discuss that when we get back. Also, Dev, I turned on follower-only chats in my Twitch. You should probably do the same. I think that will help cut down on a lot of this spam. Actually, some of this is X spam, I'm realizing.

01:10:16 - Dev Agrawal

Yeah, some of it is X.

01:10:18 - Anthony Campolo

Okay, I got to figure out how to stop these X spam messages. Anyway,

01:10:23 - Dev Agrawal

this is the first time that we are seeing the Solana bot on X. Did it somehow track already? Yeah,

01:10:32 - Anthony Campolo

that is really funny that a crypto bot happened to show up when you came on. That can't be a coincidence.

01:10:38 - Carl DiClementi

I'm sorry.

01:10:43 - Anthony Campolo

That's really funny. Awesome. So yeah, we'll wrap it up here. And yeah, thanks again, Carl. And hopefully we can get you on again at some point or anyone else from your company. Standing invitation.

01:10:55 - Carl DiClementi

Thanks a lot, guys. Thanks for having me on.

01:10:59 - Anthony Campolo

Of course. Catch you next time.

Discover Related Content

• LLM Client with Spacy (Video)
• Using Pomerium to Secure LLMs with Nick Taylor (Video)
• Interpretability and AI Scaling with Eric Michaud (Video)
• AI Workflows and Tools with Nick Taylor (Video)