Does Remix Scale?

Zach Leatherman recently put out a provocative benchmark to compare different web frameworks on the speed at which they can generate a static site.

Outline

• Introduction
  • The Results in a Nutshell
• Why is this Contrived and Misleading?
• Benchmarks as Propaganda
• Open Source Your Benchmark or Shut Up

Introduction

Zach Leatherman recently put out a provocative benchmark to compare different web frameworks. It involves taking various different frameworks and using them to transform different numbers of markdown files into HTML files.

The Results in a Nutshell

Supposedly this is fair because Remix itself states that it can be used to build a blog. To me, this benchmark and how it is being applied to these specific frameworks is contrived and misleading. I’m not the only one who noticed.

However, I also think Zach is doing something very clever and subversive here. But first, what do I mean when I say this example is contrived and misleading?

Why is this Contrived and Misleading?

The biggest problem and what makes the benchmark contrived is Remix was never meant to build and deploy static assets. It fundamentally doesn’t make sense as a benchmark for Remix. You have to ignore the fact that you can build a blog without using a static site generator.

There will never be static files on a CDN if you use Remix the way it’s actually supposed to be used. You’ll have a server that is serving requests that will involve computation on every request. The actual Remix blog example uses Prisma and serves up a blog post with a persistently running backend or serverless function.

const posts = [
  {
    slug: "this-is-a-slug",
    title: "The Title of a Blog Post",
    markdown: `
      ## What a ridiculous thing to do

      - No one would ever write a blog post like this
      - But it works technically
    `.trim(),
  },
];

for (const post of posts) {
  await prisma.post.upsert({
    where: { slug: post.slug },
    update: post,
    create: post,
  });
}

This is not what Zach does in his benchmark, so he doesn’t actually build the Remix blog example. Now you might be thinking:

Okay, but having some sort of markdown parser shouldn’t make it choke so hard, right? This highlights an exponential curve in file processing.

If it’s not handling that much markdown well, then it’s probably going to have trouble with getting many pages out of a CMS.

This must mean there’s a bug in their build system. You still need to build all the endpoints to avoid serving JavaScript over the wire. It’s still a build step.

But why does that matter? Why does it need to be performant for a task it doesn’t ever need to do? Sure, it’s exposing a scaling issue, but where that scaling issue might reoccur isn’t entirely clear. It could become a problem or it could never become a problem.

I’d need to see more data in a relevant use case that a Remix developer actually cares about. For example, Gatsby builds took forever and everyone complained about them. That’s because it was a significant and consequential part of the normal, day-to-day Gatsby workflow.

Everyone who used Gatsby had to build their project every time it needed to be deployed so everyone hit that problem. This is a problem that only Zach is hitting because only Zach is a big enough Jamstack nerd to try and hack Remix into a static site generator.

Benchmarks as Propaganda

With all that said, the article is an absolutely brilliant piece of propaganda. 10/10. There’s a lot of interesting politics at play here. Zach works for Netlify. The Remix creators traditionally have trashed the Jamstack so it feels a bit like payback.

At first glance, the benchmark may not read as particularly provocative. “How long it takes to convert markdown into HTML” shouldn’t be considered an obscure or artificial benchmark. It’s provocative in the sense that it’s portraying Remix as being fundamentally broken.

The Remix curve continues above and over the title of the graph itself, making the visual of the result verge on the absurd. It doesn’t even give a result on the final data point, 4000 markdown files. It was shut down after an arbitrary amount of time with no explanation for why it was force quit after 30 minutes.

Most people who see this will only register that one, single detail. They probably won’t even read the rest. But because Zach is shoving a blatantly Jamstack shaped problem at the framework, the Remix team can easily say, “well obviously this doesn’t scale you would never do this with the framework.”

Open Source Your Benchmark or Shut Up

Ultimately though, the most important thing to me is that Zach open sourced the benchmark and made it available to anyone who wants to play around with it or modify it. Maybe Kent will decide that the sheer magnitude of Remix’s greatness and amazingness needs to be justified.

He’ll want to ensure that their glory cannot be in doubt so he’ll spend an entire week pounding Mountain Dew and optimizing a brand new build system that allows Remix to churn out 10,000 markdown pages in 17 milliseconds. That would be a win for everyone and I welcome it.

Special thanks to Emilia Zapata and Ben Myers, this blog post is abstracted from conversations I had with both of them.