More build tools: Nix 2024-10-29 After talking about Bazel in one of our previous episodes, we are now looking at Nix, a build system that has been getting increasing attention lately.
Resources:
Transcript Surma: I know there's like two camps of people. Surma: The people who like ASMR and the people who get freaked out by mouth sounds. Jake: Oh, yeah, a little... I don't like the little mouse sounds. No, no, no, no, no. In particular, Jake: podcasts, if you've got that... It's a lip-smacking thing. I'm like, no, no, no, no, no, no, no. Jake: Edit it out. Edit it out. Oh, hope we don't do that. Jake: Welcome to another episode of OTMT, a podcast about... What do we do here, Simon? Web things! Jake: Hi, I'm Jake, and Shopify lets us do this, and we're ever-grateful. Surma: And I'm Surma. Surma: Thank you, Shopify. Surma: I think it actually has been too long without a bathroom story, and there is a story that Surma: I need to share with you. Jake: Oh, I'm ready. I am in the receive position for your bathroom story. Jake: Thrown. Absolutely. Surma: Are you on the edge of your...? Surma: This was when I was on vacation, and I was in, like, a restaurant, and I wanted to wash Surma: my hands. Surma: I was not even, you know, not even bodily functions. Surma: Just wanted to wash my hands. Surma: Went into a really nice-looking bathroom, you know, like, they're going to have, you Surma: know, like, proper cotton towel stacks and those kind of things. Surma: And the sinks were really quite interesting. Surma: So it was like a big mirror on the wall, like, one... basically, the entire wall was just Surma: one mirror. Surma: And then the sinks were at a 45-degree angle going into the wall, so you had, like, one Jake: I see kind of trough sort of thing. Wait, hang on. This was this was definitely the Surma: super long sink in front of you, effectively, so multiple people could wash their hands Surma: next to each other. Surma: Yeah, okay. Surma: Like a trough. Surma: Yes, I did double-check. Jake: sink and not the urinal. OK. Oh, so you're going to get you're going to get all angles Surma: But also, wouldn't it be weird if you had a urinal with a big-ass mirror in front of Surma: you? Surma: And by the way, the trough itself was also a mirror. Surma: Yeah, you know, if you need a good checking out, that's the place to be. Jake: of your of your little lad. Oh, I see. Yes. OK, I'm with you. Surma: So you kind of hold your hands into this trough, and, like, the trough goes a bit under Surma: the wall, and then suddenly water comes out, like, you kind of know them from, like, airports, Surma: kind of, like, auto-sensing things. Surma: And so I was washing my hands, admiring the undersides of my hands in the mirror of the Surma: trough, and suddenly, from below the sink, from basically inside the wall, hands appear Jake: What? No, I want my money back. Surma: with really nicely done red nail polish. Surma: And I let out a little yelp, and that's when I understood that this wall was shared with Surma: the ladies' room, and from both sides, basically, the wall contained the taps. Surma: And then the trough came in from both sides with a cut-through in the wall, and you could Surma: watch each other's hands, how they wash. Jake: I don't know. No, I I disinvest in this idea. Jake: Like, yeah, because there's a chance of just, you know, two people going in at the same Surma: Yeah, like a, like a nice little post-bathroom break handshake, is that not what you're looking Jake: time and a hand collision. And I would have to get a flight back home. Surma: for? Jake: Bathroom break, handshake. I mean, now I hear it rhyming. I like it. Surma: Now there's, there's a niche, there's a brand, yeah, it's, it really has burned itself into Surma: my brain as a startling experience that, and then, when you start, when you keep watching, Jake: Does sound like the kind of thing like someone who works in finance in the city would say Surma: and you realize there's more hands, it's just like these disembodied hands kind of wringing Surma: around each other to wash themselves. Surma: Very Addams Family, it was surreal. Jake: is like, excuse me, let's just going off through a bathroom break, handshake. Jake: You know what I mean? I mean, I don't know what you mean. Jake: No, you know what I mean? I don't know what you mean. Jake: Yeah, you know what I mean? Yeah, that's. Surma: That's probably also the people who would enjoy this the most, and not be startled by Jake: Yeah, they built it. Yeah, absolutely. Surma: it. Surma: So instead of analyzing this further and re-traumatizing myself, should we talk about tech, I suppose? Jake: Oh, wait, wait, what? This I don't like change. Surma: Ah, well, no, I know. Surma: Well, it is, it is useful for web development, so, you know, I'm going to talk about Nix. Jake: I. Jake: Hmm. Surma: So a couple months ago, I think, I don't know how long ago it is now, actually, I did an Surma: episode about Bazel, which also, you know, not really web development, but kind of tangential Surma: because it can be used for web development, and Nix is the same. Surma: It's also a build system, and I have been learning it, and I have become a big old fan Jake: Oh. Surma: of it. Surma: And therefore, I'm now here to kind of like talk a bit about why, because I do think it Jake: Hmm. Surma: is really interesting to think about. Surma: It genuinely has made me question certain things about why our current incarnation of Surma: operating system, build systems work the way they do, because some of these things seem Surma: kind of like bad choices in retrospect. Jake: I do think that there's so much of the web-built systems that we use Jake: that kind of haven't maybe learned enough from some of these grander scale systems, Surma: It even made me question some of the choices that, for example, the current Rust compiler Jake: so I'm excited to hear more about this. Surma: toolchain makes, like not question it, but you kind of realize they're all built on assumptions Surma: that have been around for ages, and they kind of work, but we could be, we could be doing Surma: so much better. Jake: Before you dive into that, I feel, has Rust now switched into that thing Jake: where people are actually critically analysing it? Jake: I'm seeing a lot of this, like a lot of Rust criticism all of a sudden. Jake: It felt like it's been for five years or however long, Surma: Yes and no. Jake: it's like, wow, this is just revolutionising everything, Jake: we should rewrite everything in Rust. Jake: And it feels now that it's kind of got to, it's sort of past that hype peak, Jake: and people are like, huh, this is, you know, maybe we should be using Zig or... Surma: I feel like it is becoming cooler to not like Rust, to an extent that's a deserved counter Surma: movement because there was also some undeserved overhype, like this whole, you know, the Rust Jake: I don't know. Surma: community, I think is really great. Surma: But there are also a couple of traits that for some reason are very present in many people Surma: in the Rust community, like what you just said, the whole like, this should be rewritten Surma: in Rust and it would be so much better. Surma: Most of the time, that's just a shitty thing to say, like a piece of software has been Surma: around for ages and working well and has been written in C or C++. Surma: Will it get better just because it's now rewritten in Rust? Jake: Yeah. Surma: I doubt it. Surma: Like, I personally would love it because it makes the code much more approachable to me. Surma: Like, I feel much more comfortable jumping into a Rust code base than I do into a C or Surma: C++ code base. Surma: But that's just the nature of being fluent in a language versus a language that I'm not Surma: comfortable or fluent in. Surma: Like, yes, I know some C, I know some C++, but I not once really in the last, I don't Surma: know, 15 years have I written a side project in C++. Surma: I've written a whole bunch in Rust. Surma: So that's, you know, I think you're right. Jake: Yeah. Surma: There's more people being more vocal about things that are not great in Rust because, Surma: you know, it has introduced many new things to the ecosystem and to coding. Surma: But of course, it's not perfect. Surma: But I, as always. Jake: Okay, I derailed you. Jake: So, like... Surma: Yeah, let's talk about Rust. Surma: So how do I start? Jake: What is Nix? Surma: Well, it's a build system, innit? Surma: So I tried myself at making a YouTube video and I'm not sure if, so people seem to like Jake: What is Nix? Surma: it, but I'm not sure if they liked it because of the way I explained Nix or because I did Surma: nice animations, because I discovered this cool thing called Motion Canvas, which was Surma: really fun to use to build basically a video with. Surma: You can also use to build presentations. Jake: It is really nice. We'll link to that. Surma: Yeah, and I'm not going to, I literally feel bad like taking credit for how nice animations Surma: look because I just plugged things together. Surma: Anyway, my plan is not to just like rehash the video. Surma: In the video, I really like explain step by step how Nix was started by a guy named Alko Jake: It is really nice. We'll link to that. Jake: It is really nice. We'll link to that. Surma: Doltstra throughout the work on his PhD thesis and then the individual building blocks. Surma: This is a podcast. Surma: I don't have visuals. Surma: I want to focus a bit more or speed run a bit through that to get to where we could Surma: be using it in web development, potentially, because that's, I think, where it gets interesting. Jake: I have to say, when someone says, like, if you want to know about, you know, Jake: how this build system works, read my PhD thesis. Jake: Like, I cannot close the tab fast enough. Surma: It does sound scary, doesn't it? Surma: And I will say, sadly, that is kind of on brand for Nix a little bit. Surma: Like the documentation situation on Nix is not good. Surma: Like you don't have to read the PhD thesis, but you do have to dive into, you know, standard Jake: All right. Okay. Good start. Surma: library code to figure out how to use it sometimes. Surma: That's definite. Surma: And everybody I feel like that likes and loves Nix immediately acknowledges like, yeah, our Jake: All right. Surma: documentation is not good. Surma: It's not good, mate. Surma: So that's something that hopefully will get better. Surma: And also, yes, it is a PhD thesis, although it is the product of a PhD thesis, which means Surma: it has its origins in academia. Surma: And even that shows it is a purely it involves a purely functional programming language that Surma: looks a lot like or reminds one of Haskell. Surma: And that's also not everybody's cup of tea. Jake: All right. Surma: That being said, he designed a language for this build system. Surma: And I'm not sure it was 100% necessary, but the language is so small that I felt comfortable Surma: with it very, very quickly. Surma: But let me take a step back before we talk about the language itself. Surma: Basically, the problem that this guy, Alcadolstra, was looking at for his PhD thesis is the general Surma: problem of how do I get my code or my software on machine A to run on machine B? Jake: All right. Surma: And that actually isn't necessarily easy to do in a reliable way. Surma: You know, when you install Chrome, you download a binary and you have to make sure you download Surma: for your operating system. Surma: If you download the Linux one and you're on Mac, that will not work. Surma: That's the first instance already where this model of deployment can go wrong. Surma: You need to make sure that you compile the right binary for the target system. Jake: All right. Surma: And then often these binaries rely on certain libraries being present on the machine it's Surma: supposed to run on. Surma: And if these libraries are missing or even just have the wrong version, then it will Surma: likely not work. Surma: So this whole thing of just like copying binaries around is just it's not a good system. Surma: But we know people like you and me, we work in open source. Surma: That's very different where you have the source and you can just copy the source code over Surma: and compile that. Surma: And I think that actually works fairly well. Surma: But I think even there, most of us have experienced the problem of just because it compiles on Jake: Yes. Jake: Yes. Surma: my machine doesn't mean it will successfully compile on your machine. Jake: Yes. Jake: And we hit this a lot with, well, in Squoosh, Jake: because when we were dealing with the various image libraries, Surma: And I will talk about Squoosh later because that's exactly where I want to use Nix in Jake: and even two of us using Mac. Jake: Like, yeah. Jake: Works on one machine, but not the other. Surma: the future. Surma: But yeah, even with just when you have notes or NPM libraries like Sharp that pull in C++ Jake: Yeah. Surma: code or maybe even, you know, rely on certain files being present. Surma: But, you know, on Windows, everything is different. Surma: You can already tell just because something compiles on my machine doesn't mean it will Surma: compile on your machine. Surma: And even then, even if it compiles on both machines, doesn't mean that will run. Surma: Like if we have a JavaScript code base and we both run the exact same version of Vite Jake: And then Docker came along and solved the problem. Surma: to compile the exact same source code, they probably will spit out the exact same result. Surma: And yet, because you are on Node 20 and I'm on Node 22, it may not actually work because Surma: that code that was output relies on a specific version of the runtime or again on system Surma: libraries. Surma: Just if you think about one machine may have FFmpeg installed and the other one doesn't, Surma: but you rely on FFmpeg being there. Jake: Yeah. Surma: And everybody was happy. Surma: To an extent, yes, but also kind of no. Surma: I mean, so I think Docker is in its simplest terms, you can think about like it spins up Surma: a virtual machine. Surma: And so if everybody boots up the same virtual machine image, that means you can compile Surma: the software reliably in there because the entire environment is snapshotted basically Surma: in that image of the virtual machine effectively. Jake: Yes. Surma: But if you want to run the software outside of the virtual machine, you kind of back to Surma: the same problem. Surma: So unless you also tend to run the software inside the virtual machine, that problem is Surma: still kind of there because now you're back in the world outside of the virtual machine Surma: where you don't control what exactly is and isn't there. Surma: But you're right, like Docker is one of the things that's been around that kind of has Surma: made these things a lot more interesting. Surma: But also, and this is where and I think I'm basically I'm going to jump ahead here a little Jake: Yeah. Jake: Yeah. Surma: bit because we had a problem with or we use Docker in Squoosh. Jake: Yeah. Jake: Yeah. Surma: So in Squoosh, if you probably set the context here, we have loads of different image codecs. Jake: Yeah. Jake: Yeah. Jake: Yeah. Surma: So something that takes a bitmap and converts it to JPEG, another one to AVF, another one Surma: to JPEG XL. Surma: And those are all, you know, C++ or Rust programs that we have in there. Jake: Yeah. Jake: Yeah. Jake: Yeah. Surma: And we use Docker to turn those into WebAssembly because we did not want to force people to Surma: install all kinds of compiler tool chains and libraries to be able to work on these Jake: Yeah. Jake: Yeah. Jake: Yeah. Surma: image codecs. Surma: And for that, we use Docker. Surma: So inside the Docker container, we set up the basically the development environment Jake: Yeah. Surma: to work on the codecs and to also compile them. Jake: Yeah. Surma: And then we have WebAssembly, which kind of ensures that the binary works outside the Jake: Yes. Surma: virtual machine as well. Surma: But in there, in the inside the virtual machine, we still need to somehow formulate, codify Jake: Yes. Jake: Okay. Jake: Yes. Surma: how code is supposed to be built. Surma: And for that, we use makefiles. Jake: Yes. Surma: Basically, it's a bash script where every individual step, take these bunch of C files Jake: Yes. Surma: and compile them to a static library, take these bunch of C files and compile them to Jake: Yes. Surma: object files and then link them all together to form a WebAssembly file. Jake: Yes. Surma: And we kind of like wrote these steps out with a makefile, more or less. Jake: Yes. Surma: And makefiles have been around for ages, really well understood. Jake: Yes. Surma: And they work quite simply through the mechanism of timestamps. Jake: Yes. Surma: So you write your build instructions and you tell make these are the input files that this Jake: Yes. Jake: Yes. Surma: build step needs. Surma: And these are the expected output files. Jake: Yes. Surma: And if any one of the input files has a newer modify date than the output file that we have, Jake: Yes. Surma: we need to rerun the step. Jake: Yeah. Surma: That sounds pretty good on paper, but the reality is the amount of time that we lost Jake: Yes. Surma: hours to trying to figure out why something wasn't working until we realized our code Jake: Yes. Jake: Yes. Surma: changes are not actually being incorporated into the output, because somewhere along the Jake: Yes. Surma: stack of all these different build steps, there was a cache that just cached the output Jake: Yes. Jake: Yes. Surma: and didn't actually use the new output or something like that. Jake: Yes. Surma: We have lost so much time to that. Jake: Yes. Jake: Yes. Surma: No. Jake: Yes. Surma: And that's what I was praising so much about Bazel, where I felt like their caching is Jake: Yes. Surma: rock solid. Surma: Not once did I need to, you know, nuke a cache to fix a build error. Jake: Yes. Surma: It was always going to be the exact same build error because they got caching right. Jake: Yes. Jake: Yes. Surma: And I feel like Nix does the same. Surma: And that's, again, where then suddenly, to me, it becomes useful. Jake: Yes. Jake: Yes. Surma: So, you know, in Squoosh, we had this problem. Surma: We had basically used Docker to have a very consistent, shareable development environment. Jake: Yes. Jake: Yes. Surma: But then we used Makefile's glorified bash scripts to do the building. Jake: Yes. Surma: And going back to Nix and the PhD thesis, this is kind of the observation that I'll Jake: Yes. Surma: get as well. Jake: Yes. Surma: The problem with all of these tools is that on the one hand, they don't accurately necessarily Jake: Yes. Jake: Yes. Surma: track which steps need to be redone or rebuilt. Jake: Yes. Surma: And also that in many instances, it doesn't fully capture any variable that can affect Jake: Yes. Surma: the output. Surma: So, for example, in Makefile, you will just write, hey, take Clang as my C++ compiler, Jake: Yes. Jake: Yes. Surma: put these files in and take the output. Jake: Yes. Surma: But nowhere do you specify which exact version of Clang it is. Jake: Yes. Jake: Yes. Surma: Yes and no to an extent. Jake: Yes. Surma: I think it covers part of it, but it's still not, you know, bulletproof in that sense. Jake: Yes. Surma: And so what Alko did when he developed Nix is basically by tackling this problem from Jake: Yes. Jake: Yes. Surma: two sides. Surma: The one is to design a domain-specific language that is specifically designed to express how Jake: Yes. Jake: Yes. Surma: software is built. Surma: In that sense, it is like make, but on steroids. Jake: Yes. Surma: And these kind of like build recipes in Nix are called a derivation. Jake: Yes. Surma: And that's, again, that's probably academia leaking through a little bit, but it's a Jake: Yes. Surma: build recipe, right? Surma: And it is a bit like make in that you actually do end up writing bash scripts or scripts, Jake: Yes. Jake: Yes. Surma: abstraction of bash scripts, but basically just as a way for you to orchestrate these Jake: Yes. Surma: individual tools, similar to Makefile, where you invoke different tools and tell them which Jake: Yes. Surma: files to consume and where to put the output. Jake: Yes. Surma: So really what this is in that sense, again, it is still quite similar. Jake: Yes. Surma: These derivations, as they're called, they have inputs, which is usually source code Jake: Yes. Surma: and build tools. Surma: So the build tools are part of the input. Surma: They're exactly specified and they produce outputs. Jake: Yes. Surma: And here's where now the tracking of dependencies gets in. Jake: Yes. Surma: And I think it's really interesting that that build recipe, together with a list of the Jake: Yes. Surma: source files that it consumes and the tool that it needs, gets cryptographically hashed. Jake: Yes. Surma: I'm not going to go into details how, but basically I'm going to hand wave those details Jake: Yes. Surma: so that then this build recipe can be stored in what is called the Nix store. Jake: Yes. Jake: Yes. Surma: That's really just a folder on your disk, but that hash becomes part of the name. Jake: Yes. Surma: So in that sense, it's a bit like Git where it's content addressable. Jake: Yes. Surma: So if something changes in your derivation, it now gets a new name because the hash will Jake: Yes. Surma: be different. Surma: Yes, exactly. Jake: Yes. Surma: Because that's exactly where it gets kind of interesting and recursive with Nix, I feel Jake: Yes. Surma: like, because both the source files that you want to compile, as well as the tools, must Jake: Yes. Jake: Yes. Surma: come from the Nix store. Surma: You're basically not allowed to take something from anywhere else. Jake: Yes. Surma: So only things that have been hashed, you can use in your build recipe. Jake: Yes. Surma: And if you have this item in your Nix store, you can just use it. Jake: Yes. Surma: If you don't have it, because this hash also incorporates the build recipe that this item Jake: Yes. Jake: Yes. Surma: was built with, Nix can deduce how to create this item in the Nix store. Jake: Yes. Surma: So either you have this exact version of Clang on your system in the Nix store and you can Jake: Yes. Surma: use it. Jake: Yes. Surma: But if you don't, Nix can figure out how to build this exact version of Clang in the Nix Jake: Yes. Surma: store. Surma: How to build this exact version of Clang so that you have the exact same version of the Jake: Yes. Surma: compiler and the exact same version of all libraries and the exact correct source files Jake: Yes. Surma: to produce a specific output. Jake: Yes. Surma: And that sounds very tedious, but it is actually designed in a way where it's kind of natural Jake: Yes. Surma: to write this way. Jake: Yes. Surma: Yes, it is. Surma: In that sense, they bootstrap themselves up to the point where they are completely independent Jake: Yes. Jake: Yes. Surma: of the system. Surma: And that makes sense because the system is one of the biggest variables that differs Jake: Yes. Surma: between, well, systems where these kind of things, it works on my machine, but not yours. Jake: Yes. Surma: The origin often is differences in the system. Jake: Yes. Surma: And so he has designed this system in a way that you can use in your build recipe. Jake: Yes. Surma: Yes. Surma: So for example, the operating system is part of the hash. Jake: Yes. Surma: So that means if I'm building something on Windows, it will, by very definition, get Jake: Yes. Jake: Yes. Surma: a do... it's a do-it-yourself thing. Surma: It's not a do-it-yourself thing. Jake: Yes. Surma: It's a do-it-yourself thing. Surma: So if I'm building something on Windows, it will, by very definition, get a do... Jake: Yes. Jake: Yes. Surma: It's a different dependency chain all the way down than one on Darwin, because it is Jake: Yes. Surma: fundamentally a different operating system. Jake: Yes. Surma: It doesn't incorporate the version of the operating system, but I think that's all right Jake: Yes. Surma: because operating systems like kernels are stable in their APIs, or they're just assumed Jake: Yes. Surma: to be stable. Surma: And I think that's a realistic assumption. Jake: Yes. Surma: Exactly, exactly. Jake: Yes. Jake: Yes. Jake: Yes. Surma: Yeah. Surma: So they add stuff, but they barely break stuff. Jake: Yes. Surma: Breaking user land is a big no-no when it comes to operating system. Jake: Yes. Surma: So every piece of software that you express with Nix, you know, uses... needs tools to Jake: Yes. Surma: build it. Surma: And then these tools themselves must be buildable with Nix. Jake: Yes. Surma: Otherwise, they wouldn't exist in Nix. Surma: And this goes all the way down to like a very small... the smallest possible seed of kind Jake: Yes. Jake: Yes. Surma: of like impureness, where there's like one tiny, very basic, I don't know, maybe it's Jake: Yes. Jake: Yes. Surma: an assembler, I don't know, that is just pre-agreed to be accepted and will just be downloaded Jake: Yes. Surma: as a binary file or something. Surma: And everything else is bootstrapped from there. Jake: Yes. Surma: And that's because Nix is all about building software. Jake: Yes. Surma: And so it allows you to build the exact same version of this piece of code that you're Jake: Yes. Surma: looking at every single time. Jake: Yes. Surma: Of course, this sounds really, really slow. Surma: And it would be really, really slow if every time you'd have to like start from scratch Jake: Yes. Surma: and basically invent the universe. Jake: Yes. Surma: So the other thing that Alcott also looked at in his thesis is how you can do binary Jake: Yes. Jake: Yes. Surma: caching in a secure manner. Surma: So that basically, if one machine somewhere has already executed the derivation, like Jake: Yes. Jake: Yes. Surma: build the output, that you can just share that rather than having to rebuild it yourself. Jake: Yes. Surma: And there's all kinds of things you need to think about, you know, with like cache poisoning Jake: Yes. Surma: and other weird things. Surma: And he looks at all of it in the thesis. Jake: Yes. Surma: But basically, he found a solution that basically you could set up shared caches between systems Jake: Yes. Surma: that if one machine has already done the work, you don't need to redo the work, provided Jake: Yes. Surma: you have the same operating system. Jake: Yes. Jake: Yes. Surma: Yeah, it is in that sense. Jake: Yes. Surma: But the other part is that it is not just about setting up a system. Jake: Yes. Surma: It's also about executing build instructions, right? Jake: Yes. Surma: Like it's all about build recipe. Surma: So with Docker, you get your, I guess, you know, a Docker file in a sense, maybe. Jake: Yes. Surma: But even there, those are not, I guess, you know, if you think about even a Docker file Jake: Yes. Surma: is, it seems to be sufficiently reproducible considering we all use them in certain environments. Jake: Yes. Surma: But really, if you look at Docker files, they often just, you know, do apt-get install, Jake: Yes. Surma: which just downloads whatever is currently in that Debian package repository. Jake: Yes. Jake: Yes. Surma: And technically, it will definitely not be byte equivalent to what my Docker image looks Jake: Yes. Surma: like, necessarily, if we execute. Jake: Yes. Surma: Again, this is like the academic level of accuracy. Surma: And maybe that's not necessary. Jake: Yes. Surma: But I think in this case, the sharing these, these are these binary artifacts. Jake: Yes. Surma: There is stuff to think and to read about security. Surma: I'm not going to get into that. Surma: Because where it then gets interesting next is that, you know, now we have Nix, we can Surma: build software, we can bootstrap our way up from nothing and kind of. Surma: So it was originally and you still feel that kind of tailored for downloading tar balls Surma: of C or C++ source code and building that. Surma: But from there, you can kind of like continue building and build software packages like Jake: Yes. Surma: make or clang or whatever, but also build other language tool chains for like go or Jake: Yes. Surma: Dart or Rust. Surma: And that's what Nix packages is, which is an entire collection of build recipes for Jake: Yes. Jake: Yes. Surma: all kinds of software. Surma: And I think it is still currently, by a large margin, the biggest collection of packages Jake: Yes. Jake: Yes. Surma: and the most up to date ones. Surma: If you compare it to, you know, Debian's or Ubuntu's apt, or to Arch Linux's AUR, or Jake: Yes. Jake: Yes. Surma: even to Homebrew, these are all at about a third of the size than what Nix packages Jake: Yes. Surma: has. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Surma: There's two factors here. Surma: The one factor is that it's only achieved this place in the ranking in the last two Jake: Yes. Surma: years, I want to say. Surma: It was, I think, a proactive effort where they basically now have a system where each Jake: Yes. Jake: Yes. Surma: build recipe has a script that will automatically update the build recipe itself if there is Jake: Yes. Jake: Yes. Surma: a new version pushed to, you know, that piece of software GitHub repository. Jake: Yes. Surma: Previously, you would have to step in and update the thing. Surma: But most of the time, you know, software projects just like do minor fixes. Jake: Yes. Surma: So compiling a new version is kind of just, you know, bump the reference Git commit, and Jake: Yes. Jake: Yes. Surma: then the same build recipe will probably still work. Jake: Yes. Surma: So that's one thing they have done where most of these recipes will now be automatically Jake: Yes. Surma: updated on an almost daily basis via CI. Jake: Yes. Surma: And the other part is Nix is just weird to an extent. Surma: It is actually a bit scary to install. Jake: Yes. Surma: One of the implications of wanting to be able to share your build output with other users Jake: Yes. Jake: Yes. Surma: on your system, or even people on the internet, is that the building is actually executed Jake: Yes. Surma: by a different user on your system. Jake: Yes. Surma: So the Nix installer creates a bunch of dummy users. Jake: Yes. Surma: And on Mac OS, it even creates a separate partition to prevent being nuked from your Jake: Yes. Surma: disk because Mac OS is very precious about what gets to be in your, you know, in slash Jake: Yes. Jake: Yes. Surma: on your file system. Surma: And it's all legit. Surma: It's all not dangerous. Jake: Yes. Surma: But I think that is definitely the turn for many people if you, you know, you start the Jake: Yes. Surma: install and goes like, hey, I'm going to create 30 users, 30 groups and a new partition on Jake: Yes. Surma: your Mac hard drive. Jake: Yes. Surma: Yeah, I only do it now because, you know, I know it's worth it for me, but I can totally Jake: Yes. Surma: I think if you told me this, like half a year ago, I would have said nope, especially Jake: Yes. Surma: on my work machine. Surma: But even there, I've done it now. Jake: Yes. Jake: Yes. Jake: Yes. Surma: Oh, yeah, it prints it every time you boot. Surma: Gotta gotta do it. Surma: So Nix packages is this collection, huge amounts of software in there. Jake: Yes. Surma: And also there's stuff in there that lets you, if you were the author of your own Nix Jake: Yes. Jake: Yes. Surma: build recipe, makes your life easier. Jake: Yes. Surma: So, you know, instead of having to figure out how to compile, I don't know, Node or Jake: Yes. Surma: NPM yourself, because you want to build your own Node project, Nix packages is full of Jake: Yes. Surma: helper stuff for the Nix language to do Rust projects, Node projects, Go projects. Jake: Yes. Surma: It's all kind of in there. Jake: Yes. Surma: There's something you might even call documentation around it. Surma: But yeah, there is definitely support in there to help you build your own projects using Jake: Yes. Jake: Yes. Surma: Nix if you need these kind of kind of benefits. Surma: And that's exactly why I've been looking at it for Squoosh, because another problem Jake: Yes. Jake: Yes. Surma: we had in this thing that we are working at this, you know, often our codecs come at the Jake: Yes. Surma: very least as a two step compilation process. Jake: Yes. Surma: Once you compile the library that contains the codec, and then you compile our glue code Jake: Yes. Jake: Yes. Surma: that, you know, provides the standardized interface that the Squoosh app expects in Jake: Yes. Surma: a WebAssembly way to invoke that library. Jake: Yes. Surma: And often the work that happens is mostly on the glue code, because we don't develop Jake: Yes. Surma: the codecs ourselves. Surma: We just take those off of GitHub and compile them to a library. Jake: Yes. Surma: And then we write the glue code and have to figure out how do we invoke this library correctly. Jake: Yes. Surma: But in many cases, it seemed like we end up recompiling the library over and over, which Jake: Yes. Surma: takes a long time because those are, you know, quite big, quite complex. Jake: Yes. Surma: And similar to what I was a fan about in Bazel is that you can break out these individual Jake: Yes. Jake: Yes. Surma: build steps into singular units of execution, and therefore they're cached individually. Jake: Yes. Surma: So once they get built, they will never get rebuilt unless the inputs file change. Jake: Yes. Surma: And that's kind of what I've been looking at for both our Rust-based codecs as well Jake: Yes. Surma: as our C++ codecs in Squoosh. Jake: Yes. Surma: Yeah, I have a branch, and I think because, well, here's the reason why, right? Jake: Yes. Surma: Like so far, we have been committing the Wasm binaries to our repository in Squoosh, just Jake: Yes. Jake: Yes. Surma: because most of the time, if we are going to get contributions on Squoosh, it's going Jake: Yes. Surma: to be on the web app part, just because Wasm is just more niche, I guess, and probably Jake: Yes. Surma: also those things don't need, once they work, they work. Jake: Yes. Surma: Maybe an update to a newer version of the codec, but it's a lot rarer that something Jake: Yes. Surma: needs to be changing there. Jake: Yes. Surma: And so we just committed them. Surma: And I always feel iffy about committing binaries to a repository. Jake: Yes. Surma: But what is the alternative? Jake: Yes. Surma: Because building every codec every single time would, at the very least, blow our CI Jake: Yes. Surma: runtime. Surma: But also, it would just make it not fun for people to contribute. Jake: Yes. Surma: You clone the repository, you run npm install. Jake: Yes. Surma: And the first time you do npm start or run dev, whatever it's called, and then you have Jake: Yes. Surma: to wait for, I don't know, yeah, that's not cool. Jake: Yes. Surma: And like I said, even if we were to invoke Docker, then it's like, it works maybe great Jake: Yes. Surma: for a fresh build. Surma: But once you start working on the code, there's often these caching problems. Jake: Yes. Surma: So yeah, that's why I'm looking into that. Jake: Yes. Surma: Because on the one hand, with the hashing of the sources, if you had it built once on Jake: Yes. Surma: your system, it would be very fast afterwards. Jake: Yes. Surma: But also, we could set up our own remote cache. Jake: Yes. Surma: Like there's actually a whole website called cachehicks that does it for you and is free Jake: Yes. Surma: for open source projects. Jake: Yes. Surma: But you can also just use a GCS bucket. Surma: And basically, we could just allow people to clone the repository, which would also Jake: Yes. Surma: get quicker if the WASM binaries weren't in there anymore. Jake: Yes. Surma: And once they do npm run, we would invoke Nix. Jake: Yes. Surma: But then it would just end up downloading the result rather than actually doing the Jake: Yes. Surma: build. Jake: Yes. Surma: Now that presupposes that they have Nix installed, which probably would lead us back to Docker. Jake: Yes. Surma: So that's also not a perfect solution. Surma: I'm not sure yet. Surma: But I'm mostly exploring it as a means to address the problem of why am I not able to Jake: Yes. Jake: Yes. Surma: fix the bucket I'm investigating? Jake: Yes. Surma: And then an hour later, you find out, oh, there's some caching going on somewhere along Jake: Yes. Surma: the stack that maybe you didn't know about or that wasn't supposed to be caching, but Surma: it's caching for some reason anyway, which has just been a massive pain point. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Surma: So that's actually a really good point in general. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Surma: That's also what most of these helper functions for Rust projects, Go projects, JavaScript Jake: Yes. Jake: Yes. Surma: projects do for you, that they download your dependencies for you and also put them in Jake: Yes. Surma: the Nix cache. Surma: So it's an additional like, you know, projects like Rust or JavaScript nowadays have their Jake: Yes. Surma: package log file, which already protects against supply chains attacks. Jake: Yes. Surma: But if you were to like run your own cache, and then somebody goes ahead and unpublishes Jake: Yes. Jake: Yes. Surma: a package from npm, if you still have it in your cache, you would still be able to build Jake: Yes. Surma: it because it's in there. Surma: And the other part is if somebody if there was a supply chain attack, and somebody manages Jake: Yes. Surma: to somehow replace a version somewhere without doing a version bump with malicious code, Jake: Yes. Jake: Yes. Surma: all of that would be caught and it's implemented at the Nix level, it doesn't need to be implemented Jake: Yes. Surma: at the language level if you're using. Surma: So if you're experimenting with like a very new young language or some other build tool Jake: Yes. Jake: Yes. Surma: that doesn't have this kind of protection built in, Nix can provide it for you. Jake: Yes. Surma: So this kind of like security thing that you're mentioning, it provides it across the Jake: Yes. Surma: entire project. Surma: And I think that's actually also something really valuable, which is also one of the Jake: Yes. Surma: reasons why we are looking at Nix now for Shopify, to see if we can build our system Jake: Yes. Surma: with Nix. Surma: But that's early exploration, nothing I want to commit too much to publicly. Jake: Yes. Jake: Yes. Surma: Yeah, so that's something I'm, I've been working on. Jake: Yes. Surma: And I have to say like the Nix packages as a whole has replaced Homebrew for me on my Jake: Yes. Surma: system. Surma: Again, because there is a global shared cache for everything in Nix packages specifically, Jake: Yes. Jake: Yes. Surma: most of the time, it's just downloading. Surma: So it is really quick. Surma: It has so many packages in there. Jake: Yes. Surma: And not only that, it's very easy to install an older version or a newer version, depending Jake: Yes. Surma: on what you need, because each of these packages expresses their dependencies so accurately Jake: Yes. Jake: Yes. Surma: down to like a cryptographic hash that you don't run into this problem where you maybe Jake: Yes. Surma: have two pieces of software installed and one needs this new version of the FFmpeg library Jake: Yes. Jake: Yes. Surma: and the other version doesn't work with this new version of FFmpeg but needs the older Surma: one. Surma: But both versions of FFmpeg cannot coexist, which is something that has happened before. Jake: Yes. Jake: Yes. Surma: And that I just find really, really appealing. Jake: Yes. Jake: Yes. Surma: Yes. Jake: Yes. Surma: No, I know. Surma: So there's differences. Jake: Yes. Surma: One of the things actually, it is actually something I really appreciate about Nix. Jake: Yes. Surma: There is a command that you get by default when you install it called nix shell. Jake: Yes. Surma: And you can just say nix shell dash p for package and say, for example, FFmpeg. Jake: Yes. Surma: And then it will, you know, basically create a new bash shell for you. Jake: Yes. Surma: And in the shell, FFmpeg is suddenly available for you and you can use it. Jake: Yes. Surma: And once you go out of the shell, FFmpeg is gone again. Jake: Yes. Surma: More or less like it was never on your system. Jake: Yes. Surma: So it's not like it has installed all kinds of shit to, you know, your slash op slash Jake: Yes. Surma: homebrew or anything like there's nothing that has been on your system outside of the Jake: Yes. Jake: Yes. Surma: Nix stores. Surma: There will be a Nix store item that is specifically for FFmpeg, but everything is contained within Jake: Yes. Surma: there. Surma: So that way you can actually try out something without having to commit to actually fully Jake: Yes. Surma: installing on your system. Jake: Yes. Surma: And then there's a command called nix collect garbage, which will remove everything from Jake: Yes. Surma: a Nix store that you currently don't use anymore. Surma: You can also install it persistently with a command called nixenv. Jake: Yes. Jake: Yes. Surma: These are kind of like, I guess, not the best practices anymore, but they're the easiest Surma: to explain. Jake: Yes. Surma: So there's, you know, every user gets their own so-called Nix environments, which you Jake: Yes. Surma: manage with the command called nixenv. Surma: And you can add Nix packages to this environment. Jake: Yes. Surma: And then they're just there until you remove them again. Jake: Yes. Surma: So that's how installing would work. Jake: Yes. Jake: Yes. Jake: Yes. Surma: With a Nix shell commands, it knows that the second you close the shell with nixenv, when Jake: Yes. Surma: you install a package, it installs it into your profile, which sounds very abstract, Jake: Yes. Jake: Yes. Surma: but basically, like I said, every user gets their own environment. Jake: Yes. Surma: And an environment is just a collection of symlinks. Surma: So basically, there will be a .nixprofile folder that is full of symlinks with different Jake: Yes. Surma: packages that you have installed. Jake: Yes. Surma: And as long as these symlinks exist, it knows it's still needed. Jake: Yes. Surma: And then it can obviously, you know, track the dependencies all the way down and basically Jake: Yes. Surma: mark what is still needed in Nix store and remove the rest. Jake: Yes. Surma: And that's kind of how it works. Surma: And if you use nixenv to uninstall something, technically, I think you could even just remove Jake: Yes. Surma: the symlink. Surma: I wouldn't recommend that, but you probably can do that. Jake: Yes. Surma: If you then run nix collect garbage again, now the symlink isn't there. Surma: Now this package will not get marked as still in use, and then it will also get removed. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Surma: Yeah, it is weird, but I haven't been told of it. Jake: Yes. Surma: I mean, you can, as always, run it in Docker first if you want to play around with it and Jake: Yes. Surma: get familiar with the commands. Surma: To hint at something, there is a thing in Nix project called Home Manager. Jake: Yes. Jake: Yes. Surma: And I don't want to get too much into NixOS, which is a whole Linux distribution that put Jake: Yes. Surma: Nix at its heart, but I think it's more of a distraction. Surma: In Home Manager, basically, you kind of declare the setup of your machine. Jake: Yes. Jake: Yes. Surma: You basically say, these are my config files that I have in my home folder, you know, like, Jake: Yes. Surma: like, you know, like people used to have .file repositories. Jake: Yes. Surma: But it also lists, I want to have these packages installed. Jake: Yes. Surma: And then you can use Home Manager to just, like, set up your machine and continually Jake: Yes. Surma: maintain the state. Surma: So when you want to install a new package, you just add it to your Home Manager config, Jake: Yes. Surma: and suddenly this package is now there and installed and set up. Jake: Yes. Surma: And if you get a new machine, you just run this config file on a new machine, and it Jake: Yes. Surma: will be set up the exact same way. Surma: And I've done this a couple of times now. Jake: Yes. Surma: I've been playing around with a couple of Raspberry Pis at home, and basically being Jake: Yes. Surma: able to go on a freshly installed system and run two commands. Jake: Yes. Surma: And suddenly my entire environment is there. Surma: My Vim config, my SSH and GPG keys, all the tools I use are installed. Jake: Yes. Jake: Yes. Surma: There is something about that. Surma: And again, because it's all, you know, hashed and tracked, it knows exactly whether your Jake: Yes. Jake: Yes. Surma: environment is currently out of date or not. Surma: It doesn't come without its drawbacks. Jake: Yes. Surma: But if I have NerdSnipe people, they should take a look at that because it's, yeah, that's Jake: Yes. Surma: what I'm using now. Surma: I have my Home Manager config where every machine has its own little configuration file. Jake: Yes. Jake: Yes. Surma: And there's a couple of shared files when, you know, because obviously I want the same Jake: Yes. Surma: Vim config on every machine. Surma: So I don't want to duplicate that. Surma: And yeah, that's what I use to kind of like make sure my system is set up. Jake: Yes. Surma: And you can even install, just like with Vue, proper applications. Jake: Yes. Surma: So if you want to install Arc or Obsidian or Audacity, all these things are available, Jake: Yes. Jake: Yes. Surma: even if they're closed source. Surma: Some people have added them to Nix packages and it's just, you know, it's a form that Jake: Yes. Surma: just downloads a binary. Surma: But even there, you will be sure it will always be the exact same version to make sure your Jake: Yes. Jake: Yes. Surma: system is in exactly the same state. Surma: And I'm not sure that's always desirable, but for certain things that is definitely Jake: Yes. Jake: Yes. Surma: helpful to know that if this config file is applied to my system, I know that my tools Jake: Yes. Surma: will work because they have worked on these exact versions for the previous past four Jake: Yes. Surma: weeks. Surma: There's lots more to talk about, but I think I've fanboyed enough for this episode and Jake: Yes. Surma: I will link to a couple of interesting things to look at for folks who got successfully Jake: Yes. Jake: Yes. Surma: nerd sniped by this. Jake: Yes. Surma: It does feel a bit like it, right? Surma: Because I did Bazel, now I did Nix. Jake: Yes. Surma: I did feel a bit bad, but at the same time, you know what? Jake: Yes. Surma: No shame. Jake: Yes. Jake: Yes. Surma: It is, it's the best part of it. Jake: Yes. Jake: Yes. Surma: I feel like sometimes, well, you know, after your episode, we had to digest. Surma: I definitely now need to digest my own episode. Jake: Yes. Surma: So I think with all that being said, with all that being shared, I think there's nothing Jake: Yes. Surma: left to say except for happy next time. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes. Jake: Yes.