Quirrel with Simon Knott

Episode Description

Simon Knott discusses his open source projects SuperJSON and Quirrel, explaining JSON serialization challenges and serverless job queuing for modern web apps.

Episode Summary

In this episode of the Full Stack Jamstack podcast, Simon Knott joins hosts Anthony Campolo and Christopher Burns to discuss his journey from learning Scratch in elementary school to building impactful open source tools. The conversation centers on two key projects: SuperJSON, a JSON serialization library born from the needs of Blitz.js that preserves JavaScript types like dates, sets, maps, and undefined across the wire, and Quirrel, a job queuing service designed for serverless environments. Simon explains how traditional background job libraries like Sidekiq and Celery depend on always-running servers, making them incompatible with serverless architectures, and how Quirrel solves this by acting as a persistent intermediary that calls back into serverless functions when jobs are ready. The hosts and Simon walk through practical use cases comparing cron jobs and background jobs, covering scenarios like log cleanup, user onboarding emails, and fan-out tasks that split long-running work across multiple lambda executions. Throughout the discussion, Simon's approach to developer experience shines through — both SuperJSON and Quirrel prioritize hiding complexity behind clean abstractions while remaining open source and framework-agnostic.

Chapters

00:00:00 - Introduction and Programming Origins

The episode opens with introductions as Anthony welcomes Simon Knott, a developer and CS student known for his open source work across the Blitz.js ecosystem and beyond. The hosts set the stage by noting that Simon's projects, particularly Quirrel, have generated interest across multiple framework communities including Redwood.

Simon shares how he began programming in elementary school when his father introduced him to Scratch, the visual programming language. He recounts a fun project where he and his sister built a WiFi-controlled Billy Bass fish using a Raspberry Pi, complete with a built-in Scratch programming environment that his old school now uses to teach fifth graders. The conversation touches on Scratch's utility as a learning tool and its surprising depth, including a friend's discovery of its recursion limits.

00:04:55 - SuperJSON and JavaScript Serialization

The discussion shifts to SuperJSON, Simon's library that serves as a high-fidelity replacement for JSON.stringify. Simon explains that SuperJSON was created out of necessity for Blitz.js, which abstracts API logic away from developers. Since developers don't see data crossing the wire, the serialization layer needs to be seamless and preserve types that standard JSON drops, such as undefined, maps, sets, regular expressions, dates, and special numeric values.

Simon walks through the design constraints that shaped SuperJSON: it needed to be JSON-compatible for tooling support, easy for developers to read unlike magic-string-based alternatives, and safe for bidirectional transfer including untrusted client-to-server communication. He contrasts SuperJSON with Rich Harris's devalue library, which generates executable JavaScript to recreate values — an approach that produces smaller payloads but only works server-to-client. Christopher tests his understanding with practical examples like date objects losing their type through standard JSON serialization, which SuperJSON preserves.

00:12:29 - SuperJSON Deep Dive and JavaScript Maps

Christopher and Simon continue exploring SuperJSON's capabilities, discussing how it handles JavaScript's ES6 map and set types. Simon explains that maps differ from plain objects because they allow any value type as keys with triple-equality comparison, enabling use cases where the number zero and string zero are distinct keys. He clarifies that SuperJSON is strictly a serialization tool and does not perform calculations or opinionated type coercion.

The conversation covers how undefined behaves in JSON — values simply disappear rather than becoming strings, and array positions with undefined get replaced by null. Anthony highlights that Rich Harris created a comparison repo between devalue and SuperJSON, noting that having such a prominent developer benchmark against Simon's work speaks to its quality. The discussion then transitions toward Simon's involvement with Blitz.js and his motivations for contributing to the project.

00:18:38 - Blitz.js, Redwood, and Framework Comparisons

Simon recounts discovering Blitz.js in early 2020 when Brandon Bayer posted his vision for the framework and invited contributors. Having worked with Next.js and found API management frustrating, Simon was drawn to Blitz's zero-API approach and reached out to help, eventually becoming a level-two maintainer responsible for SuperJSON. He describes Rails as a legendary productivity benchmark that modern frameworks like Blitz and Redwood draw inspiration from.

The hosts and Simon compare framework approaches, with Simon appreciating Blitz's flexibility as an opt-in layer over Next.js and Anthony discussing how Redwood served as his entry point to GraphQL. Christopher brings up Bison by Echo Bind as a middle ground using Next with a GraphQL client, and the group discusses how these full-stack frameworks all combine front-end tooling, back-end ORMs, and monorepo structures but with different technology choices. Anthony notes the duplicated effort across frameworks and expresses hope that tools like SuperJSON and Quirrel can cross-pollinate between ecosystems.

00:24:37 - Introducing Quirrel for Serverless Job Queuing

Simon introduces Quirrel, a job queuing service built for serverless environments. He explains the core problem: traditional job queuing tools like Sidekiq, Celery, and Delayed Job require always-running servers to poll a database for pending work, which conflicts with the ephemeral nature of serverless functions. Quirrel solves this by running as a persistent service that accepts job scheduling requests and calls back into serverless endpoints when jobs are ready.

Christopher immediately connects with the use case, sharing how his own project reached a stage requiring scheduled tasks for analytics, reports, and log cleanup. Simon explains that Quirrel abstracts webhook complexity, handling signature-based authentication and communication logic behind framework-specific integrations for Next.js, Blitz, Redwood, Nuxt, Express, and Connect. Christopher enthusiastically praises the webhook abstraction, wishing similar patterns existed for services like Stripe and Postmark, while the group commiserates over webhook verification bugs across hosting platforms.

00:31:18 - Pricing, Cron Jobs, and Background Job Use Cases

Simon outlines Quirrel's pricing model — fully open source and self-hostable under MIT license, with a hosted version offering a free tier of 100 API calls per month and a paid tier at twenty dollars for 10,000 calls. The conversation then turns to clarifying the difference between cron jobs and background jobs: cron jobs are recurring tasks defined by a schedule, while background jobs are imperatively created in response to system events.

The hosts and Simon work through several practical scenarios to solidify these concepts. They discuss using cron jobs for batch log deletion versus individual background jobs per log entry, canceling background jobs when conditions change like a user accepting an invitation early, and choosing between cron and background approaches for batched notification sends. Simon explains that background jobs support full CRUD operations with attachable IDs, and that Quirrel itself uses Quirrel — its dashboard runs an hourly cron job to fetch usage statistics from the API.

00:42:07 - Fan-Out Jobs, Future Projects, and Closing

Simon explains the fan-out job pattern, which addresses serverless execution time limits by splitting large tasks across many parallel function invocations. He describes a client who needs to synchronize thousands of items from a third-party API every ten minutes — work that exceeds single-function time limits but can be distributed across many lambdas through Quirrel's queuing system.

As the episode wraps up, Simon teases a new project in development with a friend, focused on improving the developer experience of event-based and real-time systems. He describes the concept as bringing React's declarative state management philosophy to back-end event-driven architectures, potentially using a custom React renderer. Christopher encourages listeners to support Quirrel's growth through feedback or paid usage, and Simon shares where to find him online.

Transcript

00:00:00 - Simon Knott

Well, let's change the quality to maximum. That sounds right. Yeah. Who needs high quality when I have maximum quality?

00:00:17 - Anthony Campolo

Welcome to the Full Stack Jamstack podcast. Is it not like tying a knot, Simon?

00:00:23 - Simon Knott

I don't know. In German, it's not. It's very hard.

00:00:30 - Anthony Campolo

Okay. That's easy. Yeah.

00:00:32 - Simon Knott

I haven't decided yet whether I like the knot one more.

00:00:36 - Anthony Campolo

I like, as much as possible, trying to say people's real names in their real languages. I think it's just like a respect kind of thing. You are a developer who has worked on a lot of open source projects. You've contributed a lot to the Blitz.js world. You've done libraries and meetup talks, and you're building this really cool company around a project called Quirrel. So we've got a lot to get into. Thanks for joining us.

00:00:58 - Simon Knott

Yeah, thanks for inviting me. Awesome to be here.

00:01:01 - Anthony Campolo

We talk a lot about the different frameworks and how they all interact with each other, so we always really enjoy getting people who are kind of working on something that's not necessarily in the Redwood-verse. A lot of people in the Redwood-verse have been talking about Quirrel, and there's a lot of interest in it. So we're happy to have you.

00:01:17 - Christopher Burns

Should we start on your open source contributions or what you're doing to survive, make some money in this world?

00:01:25 - Anthony Campolo

Actually, before we get into that, I always like to know kind of how people first got into programming. What was your first programming language? Were you a CS student, a boot camp grad, a total self-starter? How did you get that going?

00:01:38 - Simon Knott

I started programming in elementary school when my father introduced me to Scratch, the children's programming language. It's awesome, and I started coding little games with it. That was formally my introduction to programming. When I was around 13, 14, 15, I actually learned programming in school. It's interesting that you ask me whether I was a CS student because I actually am currently a CS student.

00:02:03 - Anthony Campolo

Oh, okay. I think I knew that, actually. That's awesome.

00:02:05 - Simon Knott

I am going to university at the moment. I'm in my first bachelor's semester. I'm doing some computer science stuff and then a lot of open source work, basically fifty-fifty at the moment. That's how I got into programming, with my father introducing me to it.

00:02:22 - Christopher Burns

For my third-year dissertation, my theory was, can you control robots using Scratch? And I was like, I just want to write JavaScript. And he was like, the frameworks we use to control the robots do have JavaScript endpoints. And I was like, say no more. Then they were like, but you have to also put it to something. We'll just put the kid thing on it, like could a kid control a robot? And they were like, yep, that's good enough.

00:02:51 - Simon Knott

I have built a very similar thing, not at dissertation level. But do you know Billy Bass, that speaking fish?

00:03:00 - Anthony Campolo

Yeah, it like sings songs and stuff.

00:03:02 - Simon Knott

Yeah, correct. My sister found a video online where somebody connected that to Alexa. You say, "Billy Bass, what's the time?" Then it answers and it moves its mouth, and it's hella funny. My sister was like, "Simon, we can build that as well." I didn't want to build an Alexa integration because somebody already did that, but I also didn't want to mess with Alexa. I was like, let's build a remote-controlled Billy Bass. We got some motor controllers and a Raspberry Pi. Whenever you connect it to the power, it would boot up a WiFi hotspot and you could log in there. Then there was a remote control and another subpage where there was a full-fledged Scratch programming environment where people could program that fish using Scratch. We built that and it worked, and it was a lot of fun. You could write a program that told the fish to bang its head against the wall a thousand times, and then it rumbled off the floor.

[00:03:56] My old school actually uses that to teach programming to fifth-grade students.

00:04:02 - Anthony Campolo

That's really cool. Just to set the context for any listeners who've never heard of Scratch, it's a really high-level, high-abstraction programming language. The idea is that you're taking entire functions, or if-then-else kind of statements, and boiling a lot of them down to just single lines that you can then use in a GUI to smash everything together to write super-rudimentary programs. It's really nice for people who, like you say, have never programmed before, who want to start at total zero. I know the Harvard CS50 course always starts with Scratch on day one. It's a really fantastic learning resource, and it sounds like you can even do some fun stuff with it as an actual language too.

00:04:43 - Simon Knott

Actually, a friend of mine once was like, "Hey Simon, I found out the recursion depth of Scratch." He was like, "The recursion depth is like 63 or something."

00:04:53 - Anthony Campolo

I love that.

00:04:54 - Simon Knott

That's too good.

00:04:55 - Anthony Campolo

Cool. Let's start getting into some of your open source work. You've created something called SuperJSON, which has a tagline I really love, "JSON on steroids." It says that SuperJSON is a high-fidelity replacement to JSON.stringify. So first off, what's wrong with JSON.stringify?

00:05:13 - Simon Knott

SuperJSON was born out of necessity in Blitz.js. The thing with Blitz.js is it abstracts all the API and endpoint logic away from you. From a developer standpoint, you can call backend functions from the front end. It just looks like calling any other function. That's the amazing thing about Blitz.js, at least what I see as one of the main selling points of Blitz. Now that developers don't see that something goes over the wire, they wouldn't expect anything to happen to the data because it goes over the wire. When developers know that there is some transfer happening, they expect that they need to do some JSON transformations. With Blitz, you don't see it, so it would surprise you if data changed.

00:06:04 - Anthony Campolo

What you're talking about is called serialization, right?

00:06:07 - Simon Knott

Correct. And because serialization is hidden away from you, you want it to be as seamless as possible. The thing is, with JSON, for its use it's fine. It supports a very limited but powerful set of features: objects, arrays, numbers, booleans, null, and strings. I think that's all. But there are a lot of other JavaScript types that it doesn't support. The important ones are things like undefined, maps, sets, regular expressions, and then a lot of special numbers. NaN is not supported, and Infinity and minus Infinity. And actually there is a -0, which I found out a couple of weeks ago. -0 is also not supported.

00:06:52 - Anthony Campolo

JavaScript.

00:06:52 - Simon Knott

Because JavaScript, right? -0 is what you get when you divide zero by negative Infinity. There are these special cases. You could argue that -0 is not something that you need to care about, but especially the other ones — map, set, regex, undefined, NaN — that stuff you want to find a way to serialize. Historically, there are a couple of ways that you can do that.

[00:07:25] The first one is just write your own serialization format, toss JSON away, use something like protobuf or Avro. There's a lot of JSON alternatives, but they are not as widespread as JSON in any way, so they are mostly used in microservice environments and not at all in the front end ecosystem. And browser development tools don't have support, so you want to stay with something that's JSON compatible. That was the first big design constraint for SuperJSON: JSON compatibility, just to have the whole ecosystem compatibility, tooling compatibility, and have something that's easy to work with for developers.

[00:08:05] Then the other thing is there are JSON derivatives that achieve embedding those JavaScript specific types into JSON. They use some magic strings to achieve that. But the problem with these is they are not easy for developers to read. When you read them, there's a lot of magic characters in there and it's just not easy to inspect. Another SuperJSON design constraint came from that, which is good developer experience that's easy to read.

[00:08:40] And then the last one is it needs to work both for transfers from the server to the client, which is a direction where you have trust in the server so you can go that way, but it also needs to work for transfers from the client to the server. And that's a problematic one because you can't trust the client in those web scenarios. It could be any hacker, any mischief-maker. So it also needs to work in the other way. And that's what is the fundamental thing that rules out devalue.

[00:09:10] Devalue is another attempt at the same problem that SuperJSON solves. It's written by Rich Harris and it's really awesome, to be honest. It basically works like this: give it some JavaScript value and then it will create JavaScript code that, when run, recreates that value. That is super awesome because it creates very small payloads, but it's not very developer-friendly and it only works from the server to the client because you wouldn't want the server to execute any JavaScript code that gets sent over.

[00:10:12] Those were the main design constraints. What we did with SuperJSON is basically take any JavaScript value and create a JSON-compatible version of it. When you have a set, it is replaced by an array, and when you have a map, it is replaced by a list of key-value pairs. When you have a regular expression, it's represented by a string that contains that expression, and so on. For every non-JSON-compatible value, we have a transformation that makes it JSON compatible. And then whenever we transform these things, we take note of it.

00:10:43 - Christopher Burns

I've got niggles that I've dealt with in JSON passes before, and let's see if SuperJSON fixes them. This is my favorite: what is JavaScript doing? A JSON string that is a boolean value. The string says true or false inside of it. Does SuperJSON help you force booleans?

00:10:55 - Simon Knott

We say that value at a.b.c right now, that's an array, but it used to be a set. When the SuperJSON value is transmitted over the wire, you can take those annotations and reapply them to the JSON-compatible form to get the old version. That's basically what it does. Everything in SuperJSON is JSON-compatible, but half of it is very easy to read and just like you'd expect it to work.

00:11:12 - Simon Knott

I don't really know if I understood the issue correctly. If you have a string that contains basically anything, whether it's true or false or foo or bar, you'd want it to come over as a string, right?

00:11:24 - Christopher Burns

And this is a thing. So I've worked with some APIs where I'm receiving JSON that I need to parse, but they've made mistakes where they've sent their booleans through as strings. So it's a string saying true or a string saying false. And I had to go on this long soul-search through Stack Overflow to find this force-boolean script to make sure when JSON is passed, it passes it as a proper boolean instead of it just being a string. I was wondering if that's something SuperJSON helps with.

00:12:05 - Simon Knott

No, that is not something that SuperJSON would deal with. SuperJSON tries to not do these opinionated things and it doesn't know anything about your schema. Imagine a JSON record that represents some user, and as a username it has a username in it. Then the user decided to name themselves false or true. You don't want that to become a boolean. So we don't do anything about that.

00:12:29 - Anthony Campolo

What you're looking for is you're just getting it over the wire in the smallest possible way. That is your entire deal, right?

00:12:37 - Christopher Burns

With all the types and the expanded types.

00:12:40 - Simon Knott

With all the types. Correct. That's the big thing with all the types. And then the smallest way is something that we will be looking at in the future, but at the moment it's mostly built to work. There are a lot of very intricate ways that we can have it become smaller, but that would also increase the complexity of SuperJSON itself. So there's some very computer science specific things that you can do using hashing and directed acyclic graphs and stuff like that, but we don't do that at the moment.

00:13:12 - Christopher Burns

Also, high level, I think I've worked out one of the uses on the server. You have an updated-at date, right? So it's just a standard new Date, correct? With standard JSON, when that gets sent through, your client will then see it as an ISO string, right?

00:13:31 - Simon Knott

Correct. Because JSON.stringify turns it into a string. And then, yeah.

00:13:37 - Christopher Burns

But say if you passed it through SuperJSON, that object, so that date in the object would now be a date instead of a string, so saving you one, then compiling it to a date again on the client.

00:13:53 - Simon Knott

Correct. That is a prime use example for SuperJSON.

00:13:57 - Christopher Burns

Glad I found one because I was like, does it do this? Does it do this? I need to check. Another one is regexes, dates. I don't think I've ever sent a set through JSON.

00:14:11 - Simon Knott

Because it wasn't possible before. Easy.

00:14:14 - Christopher Burns

There you go. We're all learning something. And a map. What do they even mean in SuperJSON, a map? Is that a JavaScript map or...

00:14:23 - Simon Knott

Yeah, it's a JavaScript map. So the ES6 map and set.

00:14:28 - Christopher Burns

Could you give me a use case for that?

00:14:29 - Simon Knott

Yeah, sure.

00:14:30 - Christopher Burns

Just so I truly know if my mind is blown.

00:14:33 - Simon Knott

Since ES6, which came out in 2015, there are these two new types called set and map. The thing with set is it's basically like an array, but values can only appear once in them. They're kind of easy to make an array out of or make JSON compatible.

00:14:51 - Anthony Campolo

Yeah. Python also has sets.

00:14:53 - Simon Knott

Yeah. They're like a very fundamental thing, especially in mathematics, and they're very helpful for modeling a lot of stuff.

[00:15:03] And then there are maps. The thing with maps is they are basically like objects in the way that you put in a key and a value and then access those values by key. But with regular JavaScript objects, keys can only be strings. When you use a number as a key, it will be transformed to a string before using it as a key. There is no difference between using the number zero and the string that contains the number zero with JavaScript objects. But with JavaScript maps, it does make a difference. The key can be any value and they are compared by triple equality, the same as when you write three equal signs, also referred to as identity. That's a very constructed way of using it, but it would allow you to have two different values, one at the key of number zero and the other at the key of string with the character zero in it.

[00:15:56] That's how you could use maps. And then there's a lot of other ways that you could use it. Maybe you want to map from some dates to other dates or something. And map would allow you to use dates as keys or use regexes or anything, basically.

00:16:11 - Christopher Burns

Say, for example, I had a list of ten dates, but then I wanted to calculate the date from ten days from that date. You could add the logic into SuperJSON to then return a map with ten days added as well. No, because that would add calculations to the map.

00:16:35 - Simon Knott

Yeah. So SuperJSON does not do any calculations itself. It's really just about serializing and deserializing values and not doing any changes, any modifications. It's designed to really not do anything that's not serializing or deserializing.

00:16:54 - Christopher Burns

The reason that I'm asking these things is because I like to find the use cases, like what's the highest level that I can understand, to go from the easy one, undefined. What would undefined be in JSON? I guess it would be a string saying undefined.

00:17:10 - Simon Knott

No, actually undefined in JSON would just not appear in the JSON because it's undefined. It seems weird, but it makes sense in the end, because when you just don't include a value in JSON and then you parse it, when you access it in JavaScript it will just appear as undefined again. That makes sense in that regard. But there are some ways in which undefined is not really supported in JSON as well. Imagine an array having undefined values in it. With JSON, they would just be replaced by nulls.

00:17:46 - Anthony Campolo

This type of stuff comes up with SQL because they talk about how do we want to have nothing there, or do you want to have a value of nothing there? I think it's really cool. You mentioned how Rich Harris has his own version of this called devalue. I actually found Rich Harris has a repo where he compares SuperJSON to devalue. The fact that you created something that Rich Harris even felt the need to compare to something he made is pretty cool. I'm a big fan of Rich Harris. I really like the work he does. He's done a lot of great open source, and the fact that you're a student just making these open source projects and he's like, I need to show people why my version is better, says a lot about the quality of the stuff you're putting out. I want to get on to some other stuff so we can talk about these other things you're working on here real quick, though.

[00:18:38] I'd love to hear how you first heard about Blitz and what motivated you to get involved.

00:18:43 - Simon Knott

In the beginning of 2020, Brandon posted a big tweet where he published his thoughts about Blitz and what he wanted to create. I think it was in January or February. He asked for people to come and help him. I really liked the idea of Blitz. I worked with Next.js before, and working with APIs was a struggle. There's one talk that I gave where basically the first half is just about how bad we are at that whole API management thing, because there's a whole lot of complexity that people should not have to deal with because it's not central to their applications. Then when Blitz came around and was like, we solved that for you, you don't need to worry about APIs or anything, I was sold.

[00:19:38] I knew that I was interested in doing infrastructure and framework work and the whole developer tooling ecosystem and just wrote to Brandon. I was like, I want to help. And he was like, sure, there are these issues, go work on them. Then I worked on them and it was fun. Now I'm a level-two maintainer. I am responsible for SuperJSON.

00:19:56 - Anthony Campolo

You said in one of the talks that I've watched, you called Ruby on Rails the mother of all productivity. So it sounds like you're also someone who is a Rails fan as well, which seems to be a common theme here.

00:20:08 - Simon Knott

I have never used Ruby on Rails actually, but it has quite a legendary status. I really admire the productivity that it used to inspire. I think nowadays there are better alternatives — Blitz, Redwood, other stuff. But Ruby on Rails is something that you should take or draw inspiration from.

00:20:35 - Anthony Campolo

It's a north star we can all point to, is what I've really found, especially because Redwood and Blitz come at this from totally different ways. But like you say, they both are aiming at that same north star, which I find very interesting.

00:20:49 - Simon Knott

What I really like about Blitz is that it gets out of your way. If you don't want to use the whole Blitz feature, it can just be Next.js plus that zero-API layer. Then when you want to have other things like database integration or the whole routing or the whole generator thing, installers and stuff, you can have that. I really liked the flexibility. I tried out Redwood. The thing is, I'm not that used to GraphQL. I've actually never worked on a GraphQL project, so it was all a bit weird for me, but I think I just need to get used to it.

00:21:23 - Anthony Campolo

GraphQL definitely has a learning curve. I found Redwood to be a pretty good way to learn GraphQL because I was coming at it already knowing some React. I was able to just focus on the GraphQL layer. I would be curious once you get a little deeper into it, how you feel about it. I understand that it has a lot of startup cost. Getting that GraphQL API is really a ton of work. It's a huge struggle, but once you have it, I find that it's just such a natural way and it fits my mental model so well. I really love it. That's the thing that's kept me in Redwood.

00:21:56 - Christopher Burns

I personally think the hardest part of GraphQL is schema definition. There are multiple ways to do it. Using Prisma Nexus is one with object-oriented definitions, and then SDL is the other one. I've used both now, and I prefer SDL. But Bison, done by the guys over at Echo Bind, uses Nexus with a GraphQL client. They use Next as the base. Blitz.js is kind of the middle ground between you want Blitz or Next with no API, or you want Next with a GraphQL API, or you want not-Next and a GraphQL API.

00:22:46 - Simon Knott

Yeah.

00:22:46 - Anthony Campolo

Can you say that one more time? Say Bison was the middle one. You said Bliss by accident.

00:22:50 - Christopher Burns

Yeah. Did I? Bison's the middle ground that has a GraphQL API?

00:22:54 - Simon Knott

And Next. I didn't really catch that you made a mistake because in my head it was like, Bison is an interesting middle ground because it has a different approach to the framework thing. I think Bison is more like a template, and then some logic about updating the template.

00:23:13 - Anthony Campolo

Yeah, it uses EJS. We talk about this in our FS roundtable episode, how Blitz.js, you guys created your own templates. I think they all have templating built into them, and they just have different ways of going about them. But really all three of them are just full-stack frameworks. It's about how do we have a nice front-end framework integrated with some sort of back-end database ORM-like tool that's contained in a monorepo. That's really what they all have in common. And then they each combine a different set of technologies to achieve that goal.

00:24:09 - Anthony Campolo

And now they all have a different mix of technologies. They're each entirely their own separate thing in terms of the way they combine, because all these technologies end up so tightly coupled to each other. Whatever set you pick and put together is going to just be its own thing. This is something that I think about because it ends up with — and we've talked about this on the show — a lot of duplicated work and everyone figuring out how to solve the same problem. I'd be very interested to see stuff like SuperJSON and Quirrel and things you're working on, how these start to cross-pollinate.

00:24:37 - Anthony Campolo

Let's get into Quirrel, because this is a really interesting project. I don't know a whole lot about the area it is in, so why don't you just give us a description of what Quirrel is and what problem you're aiming to solve with it?

00:24:37 - Simon Knott

Sure. When you build a web application, at some point you will have some need for delayed jobs or some job-queuing or background-queuing kind of thing. Imagine you build some application where after users sign up, you want to wait for seven days and then send them an email, like, "Hey, how was your first seven days?" Maybe you found out that that's a nice way to increase user retention or something. You want to have some job be delayed by seven days.

[00:25:20] In the past, we used job-queuing frameworks like Sidekiq or Delayed Job or Resque or Celery. There's a ton of them, but what they all have in common is that they don't work with serverless. You need some kind of database for them to connect to, maybe Redis or Postgres, and then they will periodically take a look at a database and figure out which jobs need to be run. After seven days, they will see that job is ready and then they can execute it. But it only works if your application runs all the time, 24/7, because otherwise it couldn't poll that database.

[00:25:45] But now with serverless, we don't have servers running 24/7, and that's good. But it takes all those background queue libraries from us. The basic idea behind Quirrel is to create a job-queuing service that allows you to do all of the same things. From a technical standpoint, it works similarly. You have your serverless function, and now that calls Quirrel and says, I have that job, call me back in seven days. Then Quirrel runs 24/7, can do the same polling stuff, and works in the same way as those queuing libraries that you used to have. When the job is done, it will just call back your serverless function and it will start and execute the job.

[00:26:26] That's how it technically works. But from a developer's standpoint, nothing really changes. All that complexity is hidden behind the Quirrel client. You don't need to worry about all of that. The idea was to create the most developer-friendly queuing library possible.

00:26:49 - Christopher Burns

You need a task manager. When you get to a certain stage of a project, it's never straight away. This subject is hot on my mind because now we're at a certain stage of a project. We're thinking about things like analytics, reports, deleting logs if they don't need to survive for more than three days, for example. All these things, you start thinking, how am I going to do that easily? Even if you use this in a serverful environment, you still have to learn the cognitive load of knowing how a task manager runs. That's when we start looking at products like Quirrel and Repeater and think, can we just offload this cognitive load? This service just pings us back in 24 hours and says, do this. How do you ping back? Is that through a webhook structure?

00:27:47 - Simon Knott

It's basically just webhooks. But Quirrel provides that webhook logic for you. For Next.js, it uses API functions. For Redwood, it would use the functions, I think it's in that api/functions folder. The whole webhooks thing is hidden away from you. But it does all of the things that people would want to do with webhooks. It does signature-based authentication. Only Quirrel can call these APIs and not anyone that knows those endpoints. And it also hides all of the complexity of communicating with Quirrel from you.

00:28:24 - Anthony Campolo

You mentioned that it works with both Next and Redwood. Are there other frameworks it works with as well?

00:28:29 - Simon Knott

At the moment it works with Blitz.js, Next, Redwood, Nuxt.js (which is the Vue one), Express, and Connect. Connect is that Node.js middleware thing, basically. I am also looking to create an integration with SvelteKit. Once that comes out, I want to have that for SvelteKit.

00:28:51 - Anthony Campolo

Actually, I wrote a blog post about SvelteKit and it's by far the most read thing I've ever written. For some reason, there seems to be a decent amount of interest in SvelteKit.

00:29:00 - Christopher Burns

Sorry, we've skipped over that. I think it's so massive, your abstraction of webhooks. I've dealt with a lot of webhooks now and I hate them, especially Stripe's webhooks. But looking at what you've done at Quirrel, how it's this abstracted library where you just say, point Quirrel at this API endpoint, and we'll take the webhooks away and give you a nice little section for you to write your async function, and we'll do the rest. I want that for every single client that has webhooks. If you could just do it for Stripe, do it for Postmark, I would pay all the money in the world for that.

00:29:45 - Anthony Campolo

The common theme for Chris, he's like, ah, I just don't want to do anything with Stripe. Why can't all my frameworks solve all my Stripe problems for me?

00:29:53 - Simon Knott

One nice thing about Stripe is it's not as bad as Paddle. I use Paddle with Quirrel, and Paddle is fine, but their webhooks are crap. There's some signature-based authentication and you need to sort your request body in the same way that the standard sorting algorithm sorts it. You need to implement a special sorting algorithm for it, and it's just bonkers.

00:30:22 - Christopher Burns

Okay, that's worse, but I can maybe one up you. I hosted my Redwood project on Vercel and I came across a breaking bug where Stripe would not verify any webhooks that were sent through Vercel. After speaking to Stripe, Redwood, and Vercel, they all said it was each other's fault. That bug is still there. No clue how it was fixable.

00:30:50 - Simon Knott

Oh my God.

00:30:51 - Anthony Campolo

So that's the two Spider-Mans pointing at each other.

00:30:56 - Christopher Burns

So Vercel plus Redwood plus Stripe is kind of broken, but nobody's fixing it right now.

00:31:03 - Simon Knott

Maybe it's Amazon. That's the only missing party.

00:31:05 - Christopher Burns

Maybe it's Amazon. Exactly. Because they're all doing abstractions on top of Amazon. But the abstraction of the webhooks, say no more. I want this everywhere.

00:31:18 - Anthony Campolo

You have a paid version of this as well. Sounds like you've got a hobby-dev version that you can start off with, and then there's also a paid version as well. So how does that kind of work?

00:31:27 - Simon Knott

Quirrel itself is MIT licensed. It's fully open source. I want to put that out there. If you want to host it yourself, you're totally free to host it yourself. But considering that it's for serverless use cases, I think people want to have a hosted version. So I offer one.

[00:31:58] On that hosted version, there is a free account version where you have 100 API calls a month for free. That would equate to about three cron jobs a day. Then there is a paid version which is 20 bucks a month, where you get 10,000 API calls. To be honest, you can even use more. It just says 10,000, I think to have some legal grounds there.

00:32:14 - Christopher Burns

Payment subscriptions are hard. It's what we're going through with our fund. It's like, how much do we price this? And we go, stick your finger in the air. Is it the same as yesterday? Do you feel the same as yesterday or not?

00:32:26 - Simon Knott

The thing is that I want to have a pay-per-use kind of payment model, but I am struggling to find a payment plan that really works. There are two different usage profiles of Quirrel. There are people that use it professionally, but with very few jobs, like having one cron job a day, maybe 30 a month. Or maybe 100 cron jobs a day, or 10 cron jobs.

00:32:55 - Anthony Campolo

Can we talk about what is the difference between a cron job and a background job, and how those two relate to each other? This has been brought up before. It's never quite stuck in my brain, but I know that these terms both get thrown around as if they're the same thing, but they're not.

00:33:07 - Simon Knott

Cron job is a term that stems back from a tool in Linux or Unix. It's an operating system tool where you give it a schedule and a command that it should run on that schedule. You could say run this command, any Linux command, every day at 4:05, or every Friday on the first of the month when the moon shines. You can do very ridiculous schedules. These are recurring jobs. Those are cron jobs.

[00:33:52] And then there are background jobs, or I call them queues, which are the imperative version of that. So you don't say I want to be called three times a month or something, but you say here I create a new job, have that execute in three days and have it repeat three times, or have it repeat along that cron schedule three times at maximum. But you create it imperatively.

[00:34:17] When you want to have some analytics thing, like Christopher, you would use cron jobs to have something run every day at 7 p.m. And when you want to have some jobs that come from certain events in your system, maybe you have a new user who you want to send an email out in five minutes, that would be a queue.

00:34:39 - Christopher Burns

So it would be very hard to say, when I, Chris, join the service, I'm going to attach a cron job for five days. But that's actually: when I, Chris, join the service, I'm going to be creating a background task job for five days.

00:35:01 - Simon Knott

And the difference is you can actually create background jobs that run on a cron schedule. Cron by itself is just the specification of that syntax of how to write out those schedules. It's not very human friendly, there's a lot of stars in it. But that is just a syntax to define when a job should run. You can create background jobs that repeat along a cron schedule. That is possible. But the difference is cron jobs are basically static, once per system, and then queues are invoked dynamically in your system. You can dynamically enqueue jobs into queues.

00:35:57 - Anthony Campolo

I can see why this has confused me in the past.

00:36:01 - Christopher Burns

To me, in my head, a cron job is something I would set up to do x amount of times, and a background job is something that I set to run once an event has happened.

00:36:15 - Simon Knott

That is correct. That's a really nice way to put it.

00:36:18 - Christopher Burns

I've got a few questions and they're all subjective. There's no easy answer because obviously this is how and why. But for example, we have logs, right? You have to have them in an enterprise system. But on the free tier you say you get them for three days. What in your eyes would be better: use a cron job that would run every midnight, deleting every log that is now expired, or a background task that runs individually on every log?

00:37:01 - Simon Knott

You should use a cron job because having a background task for every log — how many logs are there? A lot of them. You should just use a cron job and then you can batch delete your logs.

00:37:15 - Christopher Burns

Because this is a thing that in my head you have to think about background tasks and cron jobs as two different functionalities. And you need to understand the different use cases. The logs is a good example. There's going to be mass quantities of them. But if you set a background job, you can do that, but then it's going to be requiring your system to do a lot more work on every scale. But a cron job would require it to just do it once a day, for example. So my next use case is, say, like an invitation. You've just invited someone to your platform and they don't join within the first four days, right? So you decide to queue up an email for the fifth day. But on the fourth day, they join. So can you cancel cron jobs or background jobs?

00:38:19 - Simon Knott

You can't cancel cron jobs, but you can cancel background jobs. You can attach IDs to those background jobs, and then you can use those IDs to delete them afterwards. If you don't attach IDs, it just auto-generates them for you. There is full create, read, and delete functionality built in. You can enqueue new jobs, delete jobs, invoke jobs (which basically means ignore the desired running time, just execute it now). And then there's reading. You can either fetch one specific job by ID, or you can iterate through all the jobs in a queue.

00:39:07 - Christopher Burns

Now I have an interesting one. That's a scenario where you have, say, 900 users and you want to send them a notification, but you want to send it in three batches. You could say, I want to run a cron job that would send it to group A at 1:00, group B at 2:00, and group C at 3:00. You wouldn't use background jobs off the top of my head.

00:39:37 - Simon Knott

I think it depends. Imagine you have a back-office interface for creating those batches, like some in-house tooling, and you actually want to have those batch sending things be created dynamically. You would use background queues. But if the developers specify that once, then you'd use cron jobs.

00:40:10 - Christopher Burns

If anything, it's basically if you're going to do something on a schedule, it's a cron job. And if it's triggered by an event, then it's a background job.

00:40:27 - Simon Knott

That's one way to put it. Yeah.

00:40:29 - Christopher Burns

Does that help conceptualize it, Anthony, as use cases and...

00:40:34 - Anthony Campolo

This stuff is hard because it's so low level. It's really important to get some concrete use cases. I'm glad you went through all that. This will be one of those episodes I'll be enjoying going back and listening to and learning a lot myself.

00:40:49 - Simon Knott

One of the ways that I use Quirrel myself, just as another use case, is Quirrel uses Quirrel itself. It's full recursion, full meta. There are basically two parts of Quirrel. There's the Quirrel API and the Quirrel client, which is what I refer to when I say Quirrel. And then there's Quirrel.dev, which is the landing page of Quirrel and the whole dashboard. That is a whole other project, basically just an interface for the hosted Quirrel version.

00:41:22 - Anthony Campolo

This is probably what's confusing for people as well, because it's kind of a Rails app, but it's also built with Redwood. I think that's why in my brain that never really made any sense. But as you say, it's because it's really two things here.

00:41:35 - Simon Knott

It is. The .dev, the landing page — I also call it dashboard because it's basically a dashboard — once an hour it calls the API to get usage statistics about what the user used, how many calls, and that is implemented using a cron job, for example.

00:41:55 - Christopher Burns

I think we've really covered everything. But on your landing page you define background jobs in three separate categories. I guess these are just to help describe the use cases of them, but we could quickly explain the three to get the uses. A delayed job is, say, once a user signs up, email them a week later. A recurring job is every week, send that user a report through email. But a fan-out job, now please explain.

00:42:39 - Simon Knott

With serverless, you have a maximum execution time. I think on Vercel it's 40 seconds, after which they just kill your lambda.

00:42:51 - Anthony Campolo

This is why Netlify introduced background functions that run for 15 minutes for this specific reason. Yeah.

00:42:57 - Simon Knott

Correct. When you have one task that basically consists of a lot of subtasks — for example, I talked to a client that uses some third-party API that he synchronizes with. Whenever he synchronizes, he gets between 4,000 or 5,000 items back. For some of those items, he needs to do more calls. Imagine every ten minutes, he needs to do a couple of thousand network calls.

[00:43:47] That would not be possible if you just have those 40 seconds. It's very long running, but it's parallelizable. One thing you can do is enqueue those jobs into Quirrel. Quirrel will then call back the serverless function, but create 10,000 executions. Something like that — 10,000 is a bit much. But the idea is that you can split out work that occurs in one execution across a lot of lambdas, across a lot of functions. That's the idea.

00:44:18 - Anthony Campolo

We're getting close to the end of our time here. Simon, thank you so much for being here and talking with us and explaining all these projects you're working on. I'm so massively impressed by the stuff you've already done in your fairly short career. Thank you for all your open source contributions. Is there anything that you want to talk about in terms of things you're excited for, things you're working on?

00:44:43 - Simon Knott

I am very excited when I see how much stuff the last year brought, and I am even more excited to see how I will think about this year at the end of this year. There is already one project in the pipeline that I'm working on with a friend, where we are tackling the developer experience of event-based systems. The basic idea is to make real-time systems easier, simpler to work with for developers.

00:45:20 - Anthony Campolo

So you're going to make Meteor?

00:45:21 - Simon Knott

Kind of.

00:45:24 - Simon Knott

We'll see. It's not on the scale of Meteor. It's not going to be a full-fledged framework.

00:45:29 - Anthony Campolo

Ambitious, though. I like it. Am I on that?

00:45:32 - Simon Knott

Yeah, we'll see how that works out.

00:45:35 - Christopher Burns

The only service that I've heard like this is Pusher, maybe. I don't know if that's like...

00:45:41 - Simon Knott

It's a different thing. It's not a service. It's more like a library. The way you work with React, you don't need to think about real time. You just update your state and then everything updates down below. It will just update all the components it needs to update. As a developer, you don't need to work with all of that event-driven stuff.

[00:46:15] We're basically thinking about creating a similar kind of development experience for the back end. That's what we're thinking about. Actually, we're thinking about using a custom React renderer for that. But maybe that's not the right approach. We'll see — very much in the conceptual phase right now.

00:46:31 - Christopher Burns

Thank you for your time, Simon. I've loved having you on. I'm going to be checking out this and Repeater soon and making up my own mind on which one I need to use in my application.

00:46:47 - Simon Knott

Let me know what you decide on.

00:46:50 - Christopher Burns

I will. But also, how can the supporters help? Quirrel has a demo, a free tier, and a feedback loop. If you think, how can I help Simon grow Quirrel bigger? Your simplest way to help could be just providing a bit of feedback.

00:47:11 - Simon Knott

Correct. That would be awesome.

00:47:13 - Christopher Burns

That's my homework for the listeners. If you want to help see something grow, give some feedback or maybe even pay some money to grow bigger.

00:47:26 - Anthony Campolo

Where can people find you online?

00:47:28 - Simon Knott

On Twitter. It's @skn0tt. It's a zero, not an O. skn0tt, or simonknott.de, or just Googling my name. I think that works as well.

00:47:43 - Christopher Burns

Awesome. Thank you for your time, Simon.

00:47:46 - Simon Knott

Yeah, thanks for having me.