Off The Main Thread Podcast

Why Source Maps don’t always work

In this episode, Surma shares with Jake all the things he learned reading up on source maps and DWARF.

Resources:

Transcript
  1. Jake:Welcome to another episode of... Sorry, I went in too strong there, and I felt like I couldn't sustain it, so I'm backing out.
  2. Jake:Let's maybe start softer, and we can build up from there.
  3. Surma:Starts off to, like, hello, hi. And me, Surma.
  4. Jake:Hello. Hello. Welcome to another edition of OTMT, with me, Jake Archibald.
  5. Jake:Excellent. Okay.
  6. Surma:Welcome to the Most Mindful podcast in web development.
  7. Jake:Oh, it's a little bit NPR now, isn't it? We need to pick it up from here.
  8. Jake:I should say, thank you to Shopify for letting us do this, and hosting it, and that kind of stuff. Thank you very much.
  9. Surma:Yeah.
  10. Surma:Thank you, appreciate it.
  11. Jake:So, I was in Newcastle recently, visiting some friends, and I did something I've been wanting to do for a while.
  12. Jake:I did the Chris Jericho Triangle. Have you heard of this? Are you familiar with the Chris...?
  13. Surma:No, I'm assuming it's not a musical instrument?
  14. Jake:No. Chris Jericho is a Canadian wrestler. He was in the WWF when I was a teenager, and then obviously WWE afterwards.
  15. Surma:You say obviously. I don't even know what those letters stand for. But I'm guessing it's different wrestling leagues?
  16. Jake:What was the WWF?
  17. Jake:The World Wrestling Federation was the WWF, and then they had a bit of a dispute with the World Wildlife Fund, and I assume they lost.
  18. Jake:No, there was a cage match, but the problem is, like, the WWF, as in the wildlife one, they brought tigers and bears.
  19. Surma:Was this dispute real or was it just them acting?
  20. Jake:And, you know, man, like, they were just not... there was so, so much blood. So much blood.
  21. Surma:Many people died.
  22. Jake:And now the Wrestling Federation are called WWE instead, and that was many, many years ago.
  23. Surma:And the E stands for?
  24. Jake:Entertainment. Come on.
  25. Jake:How did you get into wrestling?
  26. Surma:World Wrestling Entertainment. It kind of gives it away. I thought the whole point... I remember when I went to school, like, primary school.
  27. Surma:I don't think it was particularly popular in Germany, wrestling overall.
  28. Jake:How did you get into wrestling?
  29. Surma:But it was like, it's all just acting. And other people were like, no, it's not. It's real. Look at them. They're bleeding.
  30. Surma:And it's just like big, big mystery and conspiracy on the primary school playground.
  31. Jake:At school, I remember being sort of made fun of for watching wrestling by a bunch of girls who were really into, like, boy bands.
  32. Jake:Like, they were really into Westlife, and I was proud of my...
  33. Surma:Well, clearly, people in glass houses.
  34. Jake:Well, that was my response. I was like, well, what's the difference between a wrestling fan and a Westlife fan?
  35. Jake:The wrestling fan knows it's fake. You know, I was quite pleased with myself.
  36. Jake:But yeah, no, no, we knew it was just soap opera, you know?
  37. Jake:Anyway, Chris Jericho Triangle.
  38. Jake:This is, oh, I guess a good few years now ago, Chris Jericho visited Newcastle.
  39. Jake:And he posted on Instagram, great day wandering around beautiful hashtag Newcastle getting ready for the sold out show.
  40. Surma:Well, it's just three different sides of the same building.
  41. Jake:And he posted three pictures of his, you know, wandering around beautiful Newcastle.
  42. Jake:And so I, you know, I wanted to walk in his footsteps.
  43. Jake:And I went to the same three places and took the same three photos of myself.
  44. Jake:All three photos were within a 10 meter radius of each other.
  45. Surma:Well, I mean, that's right up your street.
  46. Jake:Not just that, outside the Wetherspoons.
  47. Jake:Yeah, well, I don't know.
  48. Surma:Did you have an honorary pint in each Weatherspoons?
  49. Jake:Oh, no, no, one Wetherspoon.
  50. Jake:Look, they're all in front of the same Wetherspoons.
  51. Surma:Oh.
  52. Jake:Like, they just, just...
  53. Surma:Okay, let me do it again. Did you have three honorary pints in that Weatherspoons?
  54. Jake:No, I did, I did have one though.
  55. Jake:And how would you explain Wetherspoons to international listeners?
  56. Surma:Like the simple way to put it is a pub chain. If you want to go deeper into it, it's owned by a bigot. And technically, you should not be giving them money.
  57. Jake:Oh, yes.
  58. Jake:I would say it's like the McDonald's of pubs.
  59. Surma:Yeah, that's a good way to put it.
  60. Jake:And in the same way that they are both kind of like figureheaded by a clown.
  61. Jake:Yeah, Tim Spoon, who owns Wetherspoons is a bit of a wolf.
  62. Jake:Yeah, a bit of a problem person.
  63. Surma:That name. You really don't have much choice in life, I feel like.
  64. Jake:Oh, that's not his real name.
  65. Jake:I can't remember his real name, but he looks a little...
  66. Jake:Well, and I'm stealing a joke here.
  67. Jake:He looks a bit like a lino from the Thundercats.
  68. Jake:It was if lino had a racist uncle, he looks like.
  69. Jake:We'll put a link to a picture of, I think it's Tim Martin.
  70. Jake:But I don't know, Tim Spoon.
  71. Surma:I think I prefer that. He shall henceforth be known.
  72. Surma:We've probably mentioned this before, but we have the inverse law of effort. The more effort you put into a blog post, a library or anything really, the less people are going to give a shit and understand what it's about.
  73. Jake:Have you ever thought about blogging about things you've just learned?
  74. Surma:And if you do a quick throwaway article, tweet, mini hack library, people go nuts on average. Less effort, more interest.
  75. Jake:Yes, a thing of this has been happening to me recently
  76. Jake:that I used to blog about things I had just learned.
  77. Jake:And because I was excited, I've just learned this thing.
  78. Jake:I'm excited about learning this thing.
  79. Jake:I'm going to write it down.
  80. Jake:And yeah, like very, very variable results on that.
  81. Jake:Whereas recently I've done a couple of posts
  82. Jake:where it's kind of just stuff I know from being around so long.
  83. Jake:And it's like, oh, now that's a thing.
  84. Jake:That's an unusual piece of entertainment
  85. Jake:that I've just been around so long.
  86. Jake:To sort of know how these things started.
  87. Jake:And it kind of makes me feel very old.
  88. Surma:That is something that many people do not have. The origins of weird attribute quirks.
  89. Jake:But, you know, people have been reading them, which is nice.
  90. Jake:Have you ever thought about age?
  91. Jake:Age.
  92. Surma:So I had this thing where I needed to scratch an itch. I've been working with a lot of WebAssembly stuff, mostly from C and Rust.
  93. Surma:And, you know, in JavaScript, we have a couple of ways to figure out what a big JavaScript bundle consists of.
  94. Jake:Oh, yes, you get those lovely little diagrams.
  95. Jake:There's plug-ins for Webpack and Rollup.
  96. Surma:Yeah, my go-to is usually Source Map Explorer, which is on NPM. And you just give it your source map. And then it gives you, I think it's called a tree map, which is a rectangle.
  97. Jake:Yes.
  98. Surma:And it partitions that rectangle into areas of proportional size to how much any given entity consumes.
  99. Surma:So if you have 20% is your code, 80% is React, that rectangle is going to be divided into 80% for a rectangle called React and 20% for a rectangle called your code.
  100. Jake:Yes.
  101. Surma:And it does this by folder structure. So there will be a big rectangle for node modules, but that in itself is going to be subdivided.
  102. Jake:Yes.
  103. Surma:And it's quite a nice way to figure out why is this JavaScript bundle 1.2 megabytes.
  104. Jake:Historically an amazing way to find out
  105. Jake:where all of your hard drive space is going.
  106. Jake:That's what these tools use.
  107. Jake:And it works exactly the same way for a JavaScript bundle.
  108. Surma:Oh, true.
  109. Surma:I feel like Source Map Explorer hasn't been maintained super well. Like, over time, it seems that it's not... I wonder if it doesn't support the newer source map versions.
  110. Surma:Because, like, there's more recent bundles, like where the Vite spits out just has, like, a big unknown area, but then other tools get it right.
  111. Jake:Yes.
  112. Surma:Anyway, what I needed is that, but for my WebAssembly binary, because it got bigger.
  113. Jake:Hmm.
  114. Surma:And I wanted to, you know, in Rust, I just wrote my code, but I also pulled in some crates, as libraries are called in Rust.
  115. Surma:And I was like, well, I wonder if it's this or that library that is actually making this WebAssembly binary bigger.
  116. Surma:And I went on a journey to try and figure out where can I get this data and can I somehow visualize it.
  117. Jake:Yes.
  118. Surma:And I want to tell you about it, Jake.
  119. Surma:Well, actually, I also want to tell you about source maps, because that was a whole parallel journey.
  120. Jake:Yes.
  121. Surma:So, I guess, actually, now that I say all this, what I want to call this episode is debugging on the web, because both of these things are related to that.
  122. Jake:Yes.
  123. Surma:But, yeah, so that's what we're going to talk about.
  124. Surma:If you take a step back, you know, we have JavaScript and it gets bundled, it gets minified, it gets all kinds of...
  125. Jake:Yes.
  126. Jake:I think your title is correct.
  127. Jake:I just don't think it's going to do well on the socials.
  128. Surma:Oh, that's true.
  129. Jake:I think we need to think of, like,
  130. Jake:I looked at what was in my WebAssembly binary
  131. Surma:I could also... I have beef with source maps.
  132. Jake:and I couldn't believe what I found.
  133. Jake:Something like that, right?
  134. Jake:Come on.
  135. Surma:There you go.
  136. Jake:Oh, yeah.
  137. Jake:So do I.
  138. Surma:Oh, maybe I should have graded with source maps.
  139. Jake:No, I have gravy with beef.
  140. Jake:But, yeah.
  141. Surma:So, I don't know which way to start, but I think I'm going to start with source maps,
  142. Jake:Maybe.
  143. Surma:because I'm assuming our audience is, on average, slightly more familiar with their existence and what they're commonly used for.
  144. Jake:Maybe.
  145. Surma:And what I learned while doing the research for this is that source maps were kind of like an ad hoc invention by an engineer at Google working on Gmail.
  146. Jake:Oh, okay.
  147. Surma:Gmail, probably using GWT at the time or whatever, but the JavaScript in the end was compiled by Closure Compiler,
  148. Surma:which, you know, for a long time was deemed to be the strongest JavaScript optimizer tool out there, until, I think, it was superseded.
  149. Jake:And even then, like, you know,
  150. Jake:sometimes I'll say on Twitter,
  151. Surma:But there was a time where you would add Closure Compiler to your Gulp toolchain.
  152. Jake:I'm importing a function from a file
  153. Jake:and then another file is importing
  154. Jake:another function from that file.
  155. Surma:I think Closure Compiler is not to take away from all our bundlers and minifiers,
  156. Jake:I wish bundlers were clever enough
  157. Jake:to split that out.
  158. Jake:You know, as in, you know,
  159. Jake:each function goes separately
  160. Jake:into the individual bundles.
  161. Jake:And then someone will go,
  162. Jake:um, actually, um,
  163. Jake:Closure Compiler can do it.
  164. Jake:So, yeah.
  165. Jake:Closure Compiler can do it.
  166. Jake:So I still think there's lots of stuff in there
  167. Jake:that, you know.
  168. Surma:but it's a proper compiler.
  169. Jake:Mmm.
  170. Surma:Like, it has a lot more of these techniques built in that we usually associate with compilers.
  171. Surma:And at the time, I think it was all kinds of stuff.
  172. Surma:You could add extra annotations, and it was really quite good.
  173. Surma:But the problem was that at the time, and we're talking like 2009 here,
  174. Surma:this engineer working on Gmail was trying to debug Gmail and the JavaScript that Closure Compiler had spat out,
  175. Jake:Mmm.
  176. Surma:which also, because Google at the time was already aware that every byte counts,
  177. Surma:had minified variable names, all kinds of minification tricks.
  178. Jake:Mmm.
  179. Surma:Really hard to look at and debug in Firebug at the time.
  180. Jake:Oh, wow, yeah.
  181. Surma:So this guy just sat down and basically implemented a little thing into Closure
  182. Surma:that would keep track of the code that's just being emitted.
  183. Surma:What does it correspond to in the original input files?
  184. Jake:Mmm.
  185. Surma:And that's kind of what source maps are.
  186. Surma:Like, it just gives you for every point in your generated code,
  187. Jake:Mmm.
  188. Surma:it has a way to tell you what is the position in the original source that corresponds to it.
  189. Jake:Mmm.
  190. Jake:Mmm.
  191. Surma:And there's some extra bits involved.
  192. Surma:But basically, he just did this to help himself debug.
  193. Jake:Mmm.
  194. Surma:And it kind of picked up steam.
  195. Surma:And here we are with source maps now being the...
  196. Jake:Mmm.
  197. Jake:Mmm.
  198. Surma:It's not a standard.
  199. Surma:It has a spec-ish that's just a Google doc that has been living for over 13 years now
  200. Surma:and has been worked on last year still.
  201. Surma:But it is kind of a de facto standard of JavaScript tooling, isn't it?
  202. Jake:Yes.
  203. Jake:And I do think there is now an effort
  204. Jake:to do it better, right,
  205. Jake:to actually sort of take it into TC39.
  206. Surma:Oh, is there?
  207. Jake:Now, I'm kind of going from memory here,
  208. Jake:but I remember that being some chatter
  209. Jake:that the DevTools teams were wanting
  210. Jake:because, you know what,
  211. Jake:source maps aren't perfect.
  212. Jake:The amount of projects where I just go into DevTools
  213. Surma:Yeah.
  214. Jake:and turn them off
  215. Jake:because I feel like I'm being lied to
  216. Jake:by DevTools, you know.
  217. Jake:And it's like, look,
  218. Jake:I just pressed, you know,
  219. Jake:skip the next line
  220. Jake:and something else happened.
  221. Jake:You know what I mean?
  222. Jake:That's not what just happened.
  223. Jake:And I'm trying to debug this properly.
  224. Jake:I want to debug what's actually running,
  225. Jake:because I'm looking at...
  226. Surma:Exactly.
  227. Jake:So I think there is some appetite
  228. Jake:to kind of revisit it
  229. Jake:and solve the problems that exist
  230. Surma:Yeah, I wanted to try to explain a little bit what is in a source map.
  231. Jake:and actually standardize it properly.
  232. Surma:On the one hand, because it helped me understand what it can and cannot do
  233. Jake:Mmm.
  234. Surma:and also explain why these weird behaviors sometimes happen.
  235. Surma:So I guess the first thing we need to figure out is,
  236. Surma:how does a browser, when you debug code, even know that a source map is available?
  237. Surma:And usually, that's a special comment.
  238. Jake:Yes.
  239. Surma:I think typically at the end of the file,
  240. Surma:which is like a slash slash hash bang,
  241. Surma:and it says the source map URL equals,
  242. Jake:And it could be inline as well, right?
  243. Surma:and it's a URL to the actual source map file,
  244. Jake:Like inline is an option.
  245. Jake:Hmm.
  246. Surma:which typically has the same file name with .map at the end.
  247. Jake:Ah.
  248. Jake:And it could be inline as well, right?
  249. Jake:Like inline is an option.
  250. Surma:That's what I was going to say.
  251. Surma:It is a URL.
  252. Jake:Hmm.
  253. Surma:So a data URL is completely fine.
  254. Jake:Ah.
  255. Surma:And it's not uncommon.
  256. Surma:And that's usually called an inline source map.
  257. Surma:But whatever that file is,
  258. Surma:whether it's an inline base64 encoded data URL
  259. Surma:or the other file that you actually have to fetch,
  260. Surma:that file is just a JSON file with a handful of properties, really.
  261. Jake:Hmm.
  262. Surma:So what is in there?
  263. Surma:One thing I found is that Evan, who did ESBuild,
  264. Jake:Hmm.
  265. Surma:also wrote a source map visualizer.
  266. Surma:This is not like a bundle analyzer to tell you
  267. Surma:this source file is consuming this many bytes in a bundle,
  268. Surma:but it actually visualizes the original source code
  269. Surma:next to the minified source code.
  270. Surma:So when you hover over a bit,
  271. Surma:it shows an arrow to where it ended up on the other side,
  272. Jake:Oh, nice.
  273. Surma:and you can literally see which files have been mangled together
  274. Surma:in different ways.
  275. Jake:Oh, we'll link to that, because I want to play with that.
  276. Surma:Yeah, absolutely.
  277. Surma:That will, of course, be in the description.
  278. Surma:There's also going to be a link to the spec
  279. Jake:Oh.
  280. Surma:and to a Hacker News comment from the inventor of source maps,
  281. Surma:that Gmail engineer, which has a bit of details on the history.
  282. Surma:But yeah, it is just a JSON object,
  283. Surma:which really, it is quite straightforward, I want to say.
  284. Surma:So it has the name of the file that this source map is for.
  285. Surma:So the generated file links to the source map.
  286. Surma:The source map links to the generated file.
  287. Surma:So you just know how to map from one to the other.
  288. Surma:You have a list of the original source file names
  289. Surma:that were incorporated.
  290. Jake:Hmm.
  291. Surma:That's mostly because you don't want to repeat the string
  292. Surma:multiple times later on, but just like, say,
  293. Surma:this is source file one, source file two,
  294. Surma:which makes the whole source map smaller,
  295. Surma:even though they're already quite big.
  296. Surma:There is sometimes a list of symbols.
  297. Surma:This helps you unminify variable or function names.
  298. Surma:So it tells you, ah, this function is called O in the output,
  299. Surma:but the original name was delete all files on hard drive
  300. Jake:Hmm.
  301. Surma:or whatever, like it tells you that.
  302. Surma:And then the big part is the mapping property,
  303. Jake:Hmm.
  304. Surma:which is a very long string.
  305. Surma:And looking at it now, I actually am surprised it's a string
  306. Jake:Hmm.
  307. Surma:because it is really a two-dimensional array.
  308. Surma:That's for some reason has been string encoded.
  309. Surma:I'm not sure what the reasoning behind it is.
  310. Surma:I wonder if it just compresses better. I don't know.
  311. Surma:So basically it's a very long string.
  312. Surma:And if you were to split it by semicolon,
  313. Surma:each entry in that resulting array represents one line
  314. Surma:in the output file.
  315. Jake:OK.
  316. Surma:And then each of these lines you can split by comma,
  317. Surma:which gives you an array of segments in that line.
  318. Jake:Hmm.
  319. Surma:And segments are these little blobs in the output file
  320. Surma:that can be mapped to a source file.
  321. Surma:Now, I'm not going to try to explain the actual encoding,
  322. Surma:but it's a bit like UTF-8.
  323. Jake:Hmm.
  324. Surma:It's called VLQ.
  325. Surma:Basically, it's just up to five numbers,
  326. Jake:Hmm.
  327. Surma:which is like what file is it,
  328. Surma:what line in the original source,
  329. Jake:Hmm.
  330. Surma:what column in the original source,
  331. Surma:and potentially what original name for this variable.
  332. Jake:Hmm.
  333. Surma:But it really just maps a point to a point.
  334. Surma:And this is one of the things I realized
  335. Jake:Hmm.
  336. Surma:where things get iffy with source maps
  337. Surma:because it doesn't really tell you the range that it maps to.
  338. Jake:Hmm.
  339. Surma:So like in the generated code,
  340. Surma:you obviously just have these list of segments
  341. Surma:and they are going to be,
  342. Surma:if you say one segment starts in column five
  343. Surma:and the next segment starts in column eight,
  344. Surma:you know the first segment is three characters long
  345. Surma:or something.
  346. Surma:But what it doesn't tell you of where it maps to,
  347. Surma:how long that is.
  348. Surma:And I think that's where the problem sometimes comes in
  349. Surma:where you're technically setting a break point
  350. Jake:Hmm.
  351. Surma:in the middle of a segment,
  352. Surma:but DevTools can only set the break point to the start
  353. Surma:or jump to the start.
  354. Surma:And especially with minifiers,
  355. Surma:putting multiple statements into one line
  356. Surma:and using commas instead of semicolons
  357. Surma:and potentially skipping certain things,
  358. Surma:I don't think the mapping it can express is powerful enough
  359. Surma:to handle all modern minification techniques.
  360. Surma:But I'm going to get a bit more into that
  361. Jake:Oh, interesting.
  362. Surma:because it's a very different story in WebAssembly
  363. Surma:that's quite interesting.
  364. Surma:So yeah, basically what the browsers do,
  365. Jake:Hmm.
  366. Surma:you know, stepping through the generated code,
  367. Surma:the minified code is quite easy for a JavaScript engine.
  368. Jake:Hmm.
  369. Surma:And then whenever it wants to figure out
  370. Jake:Hmm.
  371. Surma:where it currently is,
  372. Surma:it just basically does a binary search
  373. Jake:Hmm.
  374. Surma:through this mapping data
  375. Surma:and figure out what is the correct corresponding source file.
  376. Jake:Hmm.
  377. Surma:And then it can either fetch the source file
  378. Jake:Hmm.
  379. Surma:or sometimes the contents of the source files
  380. Jake:Hmm.
  381. Surma:are also embedded in the source map itself,
  382. Surma:which makes them even bigger.
  383. Surma:But that allows you to debug in production as well.
  384. Jake:I think that's the more common way of doing it, right?
  385. Jake:Like that's, I think...
  386. Surma:I think so, yeah.
  387. Jake:Yeah.
  388. Surma:I guess sometimes if you want to debug in production,
  389. Surma:it might be good to strip it
  390. Jake:Hmm.
  391. Surma:because you don't necessarily want to leak your source code.
  392. Surma:But if you have the source code locally available,
  393. Surma:you can give it to DevTools
  394. Surma:so it can resolve it while debugging
  395. Surma:while people who are out there in the world
  396. Jake:Hmm.
  397. Surma:can't really deduce anything meaningful from it.
  398. Jake:I was going to say, does that
  399. Jake:actually matter
  400. Jake:because, you know, your minification
  401. Jake:shouldn't be a security step, but
  402. Surma:Yes.
  403. Jake:in the sort of
  404. Jake:newer world of remix
  405. Jake:and sort of
  406. Jake:the React server components stuff
  407. Jake:where you have, like,
  408. Jake:your database code in the same
  409. Jake:file as your client code
  410. Jake:I guess that's where it would become
  411. Jake:potentially difficult, right?
  412. Surma:I would agree that minification,
  413. Surma:all this stuff is not a security primitive.
  414. Surma:But for certain things.
  415. Surma:It's always the same.
  416. Surma:On my screen, technically, I can download it.
  417. Jake:Hmm.
  418. Surma:But sometimes you don't want that,
  419. Surma:while still allowing people to look at stuff on screen.
  420. Jake:Right.
  421. Surma:And so I think you just want to make it hard.
  422. Surma:And so debugging could also make it harder
  423. Jake:Yeah, okay.
  424. Surma:for people to actually inject code in the right place
  425. Surma:if they want to write a malicious browser extension.
  426. Surma:So I think, yeah, it's not a good protection.
  427. Surma:It's just a hurdle.
  428. Surma:But it shouldn't be at all...
  429. Surma:It should be the only thing you do
  430. Surma:to try and keep your users safe.
  431. Jake:Oh, but you definitely wouldn't want your server
  432. Jake:code crossing that boundary, which technically
  433. Surma:For sure.
  434. Jake:in a build system
  435. Surma:No, that's actually a good point.
  436. Surma:So there, in that world,
  437. Surma:it's probably more important to do that.
  438. Surma:To not necessarily have your source maps
  439. Surma:contain the full source code of all your source files.
  440. Jake:you wouldn't want.
  441. Surma:This is already the end of what source maps contain.
  442. Surma:So it's really just a list of segments
  443. Surma:that map to your original source.
  444. Surma:And so what Source Map Explorer does,
  445. Surma:and what I now did as well,
  446. Surma:in the tool I wrote that I'm going to talk about in a little bit,
  447. Surma:you basically just iterate over all the segments
  448. Surma:in the generated code.
  449. Surma:You figure out how long they are
  450. Surma:by looking where does the next segment start.
  451. Surma:And then map that,
  452. Jake:Hmm.
  453. Surma:like figure out what does the segment
  454. Surma:correspond to my original source.
  455. Surma:Actually, not really what,
  456. Surma:but just like which file did the segment come from.
  457. Jake:Mm-hmm.
  458. Surma:And so I just built a dictionary
  459. Surma:where I have counters for each source file.
  460. Surma:And when I find a segment that is eight bytes long,
  461. Surma:I'm going to increase the count of eight.
  462. Surma:And when I go through the entire source map,
  463. Surma:I now have a dictionary where for each file,
  464. Surma:I know how many bytes it contributed in the bundle.
  465. Surma:And now I can visualize it.
  466. Surma:So with source maps,
  467. Surma:it actually was quite easy to like build
  468. Surma:almost like a clone of Source Map Explorer.
  469. Jake:Hmm.
  470. Jake:Nice.
  471. Surma:However, I told the story historically a bit reversed
  472. Surma:because originally, as I kind of,
  473. Surma:I guess already hit it at the start,
  474. Surma:is I needed this for WebAssembly.
  475. Surma:And WebAssembly doesn't use source maps,
  476. Surma:which is now very understandable to me
  477. Surma:because like it's a compile target.
  478. Surma:And most compilers already have a way to allow debugging.
  479. Surma:And that is with DWARF,
  480. Surma:which is I guess the de facto standard
  481. Jake:And this is a cross
  482. Surma:or an actual standard for debugging symbols
  483. Surma:in native binaries.
  484. Surma:Yeah, so the DWARF, the spec is very much explicitly
  485. Jake:difference, like, it's not a rust
  486. Jake:thing, it's not a...
  487. Surma:language and architecture agnostic.
  488. Surma:So it's not just for Intel processors or ARM processors.
  489. Surma:It doesn't care.
  490. Surma:And it doesn't care about the source language.
  491. Surma:And it's been very longstanding.
  492. Surma:It's now in the fifth version.
  493. Surma:And basically that is what GDB or LLDB use
  494. Jake:Hmm.
  495. Surma:when you want to step through a binary.
  496. Jake:Hmm.
  497. Surma:And the whole design around DWARF
  498. Surma:is that it's very quick to parse,
  499. Surma:very quick to access or to find
  500. Surma:and access the things you're looking for.
  501. Surma:It is very capable and flexible.
  502. Jake:Hmm.
  503. Jake:So presumably this is working
  504. Jake:very similar to source maps, so your binary
  505. Jake:now has a bit of extra
  506. Jake:metadata that is able to
  507. Surma:Yeah.
  508. Surma:Yeah.
  509. Jake:map a certain
  510. Jake:bit of binary to
  511. Jake:essentially a sort of assembly-esque
  512. Jake:kind of stuff at the end, like map
  513. Jake:that to some source
  514. Surma:There is a lot of similarity.
  515. Jake:code.
  516. Surma:DWARF actually does even more,
  517. Surma:a lot more than source maps.
  518. Surma:And it's actually not easy to wrap your head around it
  519. Surma:because of how much it needs to be able to express.
  520. Jake:Hmm.
  521. Surma:And so like, I'm not an expert in DWARF.
  522. Surma:I started skimming the spec,
  523. Surma:but it is quite a lot.
  524. Surma:But I did learn a little bit about it
  525. Surma:while I was building my WebAssembly bundle analysis tool.
  526. Surma:And so I thought I would give, like,
  527. Surma:a really quick summary of what is in there
  528. Jake:Hmm.
  529. Surma:to just see where native land, I guess,
  530. Surma:has ended up to do these kind of debugging procedures.
  531. Surma:And I'm guessing at the time,
  532. Surma:because it was like born out of an ad hoc idea,
  533. Surma:it wouldn't have made sense to use DWARF.
  534. Surma:But now that I look at it,
  535. Surma:DWARF, because it is so widespread and supported
  536. Surma:in so many compilers,
  537. Surma:there's so much tooling out there
  538. Surma:that it might actually have been beneficial to use DWARF.
  539. Jake:Oh.
  540. Surma:But that's, you know, like hindsight is 20-20.
  541. Jake:Well, they might be
  542. Surma:So maybe...
  543. Jake:looking at it again, so yeah.
  544. Surma:So like DWARF itself is...
  545. Surma:I don't want to go too deep,
  546. Surma:but it's basically a tree data structure
  547. Surma:of what they call DIEs,
  548. Surma:debugging information entries.
  549. Surma:And those are really a bit like JavaScript objects.
  550. Surma:They have attributes,
  551. Surma:and these attributes have values.
  552. Surma:And there's a whole bunch of predefined attributes
  553. Surma:with specific meaning.
  554. Surma:But as far as I can tell,
  555. Surma:you can also add arbitrary custom attributes
  556. Surma:for certain reasons.
  557. Jake:Hmm.
  558. Surma:So in the end,
  559. Surma:what you kind of get is just debugging entries,
  560. Surma:and they form kind of a graph.
  561. Surma:And I could go a bit more into it,
  562. Surma:but I think it's actually not that interesting.
  563. Surma:If you have a WebAssembly binary,
  564. Surma:or actually any native binary with debugging symbol,
  565. Surma:there is a tool from LLVM.
  566. Surma:So that's, you know, Clang, Rust compiler,
  567. Surma:all LLVM-based.
  568. Surma:So usually you have this on your system
  569. Surma:called LLVM-DWARFDUMP.
  570. Surma:And if you give a binary to that,
  571. Surma:you can just get a big console dump,
  572. Jake:Hmm.
  573. Surma:and you can kind of scroll through
  574. Surma:and see what it means.
  575. Surma:But basically each debugging entry,
  576. Surma:each of these entries in the graph,
  577. Surma:has a source name and a source file,
  578. Surma:and a range or a list of ranges
  579. Surma:of where this debugging entry
  580. Surma:is occupying the binary.
  581. Surma:So it's actually a bit the other way around.
  582. Surma:You don't go through the binary
  583. Surma:and have mappings from segments to source file,
  584. Surma:but rather you have a debug entry
  585. Surma:that is like a function or a block or a variable.
  586. Jake:Hmm.
  587. Surma:And it tells you all the locations in the binary
  588. Surma:where this entity is appearing in different ways.
  589. Surma:So for example, if there was a function called main,
  590. Jake:Oh.
  591. Surma:that would usually just have, you know,
  592. Surma:source file, the name is main,
  593. Surma:and then probably just start at an endpoint
  594. Surma:because the main has been mapped to here.
  595. Surma:Then inside this main,
  596. Surma:there could, because it's a tree,
  597. Surma:there would be sub entries,
  598. Surma:whereas like in the main function,
  599. Jake:Hmm.
  600. Surma:I have a variable,
  601. Surma:and that is used here, here, and here.
  602. Jake:Hmm.
  603. Surma:And so that is already something
  604. Jake:Hmm.
  605. Surma:that Dwarf can express at source maps.
  606. Surma:Not necessarily could,
  607. Jake:Hmm.
  608. Surma:but you have this tie back to what is the same entity.
  609. Surma:And so I was like, oh cool, this looks easy enough.
  610. Surma:I'll just iterate over all the entries in this graph
  611. Surma:and count the bytes by the ranges they have.
  612. Surma:But the result was that my total byte count
  613. Jake:Hmm.
  614. Surma:was a multiple of the actual file I was trying to analyze.
  615. Surma:So clearly I was double, triple, quadruple counting bytes.
  616. Jake:Hmm.
  617. Surma:And that actually makes a lot of sense
  618. Surma:because we are now in compiled language land
  619. Jake:Hmm.
  620. Surma:and a bit like what Clojure did,
  621. Jake:Hmm.
  622. Surma:they do a lot more than what just bundlers
  623. Jake:Hmm.
  624. Surma:and minifiers mostly do.
  625. Surma:They do stuff like function inlining.
  626. Surma:So what used to be a function call,
  627. Surma:rather than doing a call,
  628. Jake:Hmm.
  629. Surma:they just copy the code from the function
  630. Jake:Hmm.
  631. Surma:at that place where you call the function
  632. Surma:to avoid the overhead of jumping to another function.
  633. Surma:Then Rust has generics or C++ has templates.
  634. Jake:Oh!
  635. Surma:There's this thing called monomorphization
  636. Surma:or specialization where you create an instance
  637. Surma:of a generic function because you're using it
  638. Surma:with a specific type.
  639. Surma:So if I have a function called add
  640. Surma:and it takes any type, when I call it with a float
  641. Surma:and then I call it with an integer,
  642. Surma:this function will be duplicated once for floats
  643. Surma:and once for integers.
  644. Surma:So this function now exists twice,
  645. Jake:Oh, interesting.
  646. Surma:even though the source code, it only exists once.
  647. Surma:And then, you know, compilers in general
  648. Surma:are allowed to like reorder and fragments.
  649. Jake:And I guess this is a little bit like when, well, in Emscripten,
  650. Jake:which I presume comes from the C compilers,
  651. Jake:where you can compile for performance or you can compile for file size,
  652. Jake:and I presume that's one of the differences,
  653. Jake:like the compile for file size is not going to do function in Lightning,
  654. Jake:but that is at the expense of performance.
  655. Jake:Right! Okay.
  656. Surma:Exactly.
  657. Surma:Yeah, so these flags with like what you optimize for
  658. Surma:is a stiff and optimization passes.
  659. Surma:And that's exactly it.
  660. Surma:If you optimize for size, you're saying,
  661. Surma:I'm okay having slower code if the binary is small.
  662. Jake:Mm-hmm.
  663. Surma:And let's go, okay.
  664. Surma:Not only am I not going to inline,
  665. Surma:but if I find the same code in different places,
  666. Surma:I'm going to create a new function and call it
  667. Jake:Oh, nice.
  668. Surma:rather than leave it.
  669. Surma:Like, I don't know if that is exactly what's happening,
  670. Surma:but I would assume that would be quite a trivial optimization
  671. Surma:to add to save file size.
  672. Surma:So that's what these flags usually decide,
  673. Jake:Mm-hmm.
  674. Surma:which kind of passes are enabled or disabled
  675. Surma:and how are they configured.
  676. Surma:So luckily there is, I want to give a shout out to Gimli,
  677. Surma:which is a rust crate to read dwarf files,
  678. Surma:but even more so that they have, right, right.
  679. Jake:Oh, I get it. I get it.
  680. Jake:Yeah. Yeah, very good. Very good.
  681. Surma:And they have an example program.
  682. Surma:Like here's how you use Gimli correctly.
  683. Surma:And an example program they have is add to line,
  684. Surma:address to line, which you give it a dwarf file
  685. Surma:or a binary with a dwarf file,
  686. Surma:and you give it an address,
  687. Jake:Mm-hmm.
  688. Surma:basically a memory address of where a function
  689. Surma:or whatever is.
  690. Surma:And it tells you which source file
  691. Surma:and which line did this come from?
  692. Surma:So I was basically able to use most of that code
  693. Surma:to now iterate through all the memory addresses
  694. Surma:that the program would be loaded to in memory
  695. Surma:and figure out which source file does that come from.
  696. Surma:So now we're kind of back to the previous approach
  697. Surma:I took with source maps,
  698. Surma:which byte corresponds to which source file,
  699. Surma:but still it gets tricky with this monomorphization stuff.
  700. Jake:Mm-hmm.
  701. Surma:Like when you create a new version of a function
  702. Surma:for a specific type,
  703. Surma:because this could be a generic function from a library,
  704. Surma:even the standard library,
  705. Surma:but now it gets instantiated
  706. Surma:for one of your types specifically.
  707. Surma:So do you attribute that to your library, your code?
  708. Surma:Do you attribute it to the core library,
  709. Surma:consumer price?
  710. Surma:And then after all the specialization kicks in,
  711. Jake:Right!
  712. Surma:now optimization comes.
  713. Surma:There was a really interesting example I found.
  714. Surma:Like imagine you have a function
  715. Jake:Mm-hmm.
  716. Surma:in your standard library to iterate over a 2D grid.
  717. Surma:You give it like a 2D grid
  718. Surma:and it calls a callback for every cell in this grid.
  719. Surma:And this function, quite sensibly,
  720. Surma:could be used for making each pixel in an image brighter,
  721. Surma:but also could give each unit
  722. Surma:in a specific region more health.
  723. Surma:It is very generic and universal.
  724. Surma:But let's say that me using this function
  725. Surma:to make each pixel brighter gets inlined.
  726. Surma:So rather than calling a function
  727. Surma:and having a nested for loop in that function,
  728. Jake:Mm-hmm.
  729. Surma:now this nested for loop is in my code.
  730. Surma:And now my callback is not a callback,
  731. Surma:but it's inlined there as well.
  732. Surma:And after optimizing it,
  733. Surma:it figures out that part of my callback code
  734. Surma:is actually independent of the inner loop
  735. Surma:because there's some pre-calculation per row
  736. Surma:that I could do.
  737. Jake:Because it feels like we've got a similar problem
  738. Jake:in sort of JavaScript standard stuff in that, like,
  739. Jake:because of tree shaking and dead code elimination,
  740. Jake:the size of, you know, some library's impact on your project
  741. Surma:Right.
  742. Jake:can be attributed or can be impacted by the other things that are calling it,
  743. Jake:which, you know, you can say, like, this library is 300K,
  744. Jake:but it's only 300K because it's being called by this, you know,
  745. Jake:or it could even be, like, you know, an import star
  746. Jake:and then an iteration over that,
  747. Jake:or something, like, has actually caused the size of that to explode.
  748. Surma:Yeah.
  749. Jake:Is that similar, or am I kind of way off?
  750. Surma:Somewhat.
  751. Jake:Way off.
  752. Surma:Somewhat.
  753. Surma:Like, that's why I didn't find bundle-phobia super useful, because it tells you this npm
  754. Jake:Hmm.
  755. Surma:package is this big.
  756. Surma:I was like, but, well, there's like three different versions in there, and hopefully
  757. Surma:it's three shakables if I only use one function.
  758. Surma:Like that's fine.
  759. Surma:So that's why I think actually analyzing your bundle, your output, is a lot more useful
  760. Surma:as a data point.
  761. Surma:But obviously it's, you can't say in general this library is big or this library is small.
  762. Surma:And you can't do that in native land either, because exactly that, things get removed or
  763. Jake:Hmm.
  764. Surma:reordered.
  765. Surma:The problem here, just what I realized, and this is for example I think not something
  766. Surma:that JavaScript bundlers or minifiers do, nested loop has been inlined, my callback
  767. Surma:has been inlined, and now some of my code has been moved in between the loops, but not
  768. Surma:all of it.
  769. Surma:So what used to be a very clear call stack, I call a function, that function calls my
  770. Surma:function back, now these quotes have been interleaved.
  771. Jake:Hmm.
  772. Surma:So there's not just a clear mapping of range to range, but now you have an interleaving.
  773. Jake:Hmm.
  774. Surma:And DWARF can express that as well.
  775. Surma:And that's something that source maps I don't think would be able to, apart from the fact
  776. Surma:that I don't think this kind of optimization happens very commonly in JavaScript lands.
  777. Surma:So yeah, that was just something I found very interesting.
  778. Surma:And also, when you ask for, in this code that I've used, what is the originating source
  779. Surma:file for this range, it doesn't just give you one file, it gives you a whole list of
  780. Surma:files, because inlining may have happened, it actually does happen quite commonly.
  781. Surma:So it will say, yeah, this address has four source files, because it was your main function,
  782. Jake:Hmm.
  783. Surma:but in this specific POC spot, it was originally calling a different function, has been inlined,
  784. Surma:and that function, also called a function, has been inlined.
  785. Surma:So you get this whole stack.
  786. Surma:So it makes it a bit unclear how you attribute the file size, like I just did whatever address
  787. Jake:Oh, that's fun. Yeah.
  788. Surma:to file decided to do, and it seems to be good enough.
  789. Surma:But it's quite interesting to look at WASM files and see that sometimes code from a different
  790. Surma:library is attributed to my code or the other way around, and it's just something you have
  791. Jake:Yeah.
  792. Surma:to deal with.
  793. Surma:There is, I don't think there's a very clear, correct answer in every case.
  794. Jake:I had a similar problem once when I was, like, trying to...
  795. Jake:Because a lot of these bundle analyzer things,
  796. Jake:like, you're dealing in source bytes, right?
  797. Jake:And sometimes, you know, something can look really big
  798. Jake:compared to another thing, but one of them gzips really well,
  799. Jake:and that's what we really care about, because that's the bytes over the wire.
  800. Jake:To some extent, you know, obviously gzip bombs are bad.
  801. Jake:But, yeah, you get that similar attribution problem
  802. Jake:because the order of the modules in a single file,
  803. Jake:like, you know, once they've been bundled together,
  804. Jake:that matters, because suddenly it's like,
  805. Jake:oh, this first one is massive, and this second one is really small,
  806. Surma:And then, you know, then there's stuff where the compiler has to inject stuff, it's just
  807. Jake:but you put them the other way around, and the back reference is, you know...
  808. Jake:It's kind of like, who owns the data being back referenced, right?
  809. Jake:It's kind of... It depends on the order.
  810. Surma:like in JavaScript land, where, example, a pure annotation, that the function is pure
  811. Surma:if you build a library, that has no source file, that's like a compiler-emitted part.
  812. Jake:Thank you.
  813. Surma:So those bytes are often without any mappings, and there's similar stuff in native land as
  814. Surma:well.
  815. Surma:So I was not really giving a good understanding of how DWARF works, but it is definitely a
  816. Surma:lot more capable.
  817. Surma:And looking back at it, I can see how DWARF would have been overkill for JavaScript, but
  818. Surma:at the same time, I do wonder, with all the tooling that already exists, maybe it would
  819. Surma:have been useful.
  820. Surma:Although I have to say, I could not find a tool that I can give a DWARF file or a binary
  821. Surma:with DWARF in it, that tells me which files have contributed how many bytes to this native
  822. Jake:You
  823. Surma:binary.
  824. Surma:And so I had to build that myself, like this kind of, I guess, in native land, this kind
  825. Surma:of bundle analysis isn't common, because only web is a streaming platform, right?
  826. Surma:I guess the one thing, it sets it apart from all other platforms.
  827. Surma:And so that's, in the end, what I wrote.
  828. Surma:It's called Wasmphobia, because even though I don't like bundle phobia, I thought the
  829. Jake:Nice
  830. Surma:name was quite funny.
  831. Surma:And so I published it.
  832. Surma:I also added source map support, because that was, after I was done with it, actually comparatively
  833. Surma:easy.
  834. Surma:And the fun fact is that, you know, the WebAssembly binaries can have DWARF embedded, so you can
  835. Jake:Oh
  836. Surma:just drag and drop a DWARF binary into Wasmphobia, but DevTools doesn't support DWARF.
  837. Surma:So when you step through a WebAssembly binary, it has its own format for the name of a function.
  838. Surma:So you see sometimes the name of the function, but you don't know what the source file is
  839. Surma:for this function.
  840. Surma:And, you know, Rust and C++ mangle function names quite excessively to encode what the
  841. Surma:module is that it came from and what the parameter types are.
  842. Jake:Oh
  843. Surma:Because in C++, you can have the function with the same name, but with different parameter
  844. Surma:types for dynamic dispatch and stuff like that.
  845. Surma:But luckily, at the time, Inga, who was on our Squoosh team and was a WebAssembly advocate
  846. Surma:on the Chrome DevRel team, wrote a Chrome extension that teaches DevTools about DWARF.
  847. Surma:So when you install that and you debug a WebAssembly binary, you now actually can step through
  848. Surma:your original source code.
  849. Jake:Oh, that's good.
  850. Surma:And when you throw an exception or something, you get an actual stack with your source files
  851. Jake:Oh, why is that not just in the browser?
  852. Surma:and lines rather than just random WebAssembly verses.
  853. Jake:That seems essential, right?
  854. Jake:But, yeah, very good.
  855. Surma:I know, right?
  856. Surma:I don't think variable inspection works.
  857. Surma:You know, in JavaScript, when you step through, you can hover over variables and see what
  858. Surma:the values are.
  859. Surma:I don't think that works because it is kind of language specific.
  860. Surma:Because like basically DWARF tells, yeah, this is a variable.
  861. Surma:The original name is whatever.
  862. Surma:And its type is Rust colon colon string.
  863. Surma:So unless you know what the exact data structure of Rust colon colon string is, you won't know
  864. Surma:how to visualize that.
  865. Jake:Of course.
  866. Surma:But just being able to step through and figuring out where an exception is being thrown or
  867. Surma:something like that is already incredibly useful.
  868. Surma:So yeah.
  869. Surma:And I do not know why they kept it an extension and not just looped it into DevTools itself.
  870. Surma:But that just deserves a massive shout out because I didn't realize it would actually
  871. Surma:make my exceptions nicer.
  872. Surma:And that's really valuable.
  873. Jake:At the same time, I just did a sneaky bit of searching,
  874. Jake:and, yeah, source maps is now in TC39.
  875. Surma:Oh, cool.
  876. Surma:We should link to that.
  877. Jake:Maybe has been for a couple of years. Yes, we will.
  878. Jake:It's stage zero, but they're definitely working on it.
  879. Jake:Like, the last commit was three weeks ago,
  880. Jake:so it's sort of being actively worked on, which is really nice.
  881. Surma:That's more activities than some other stage zero proposals.
  882. Surma:Yeah.
  883. Surma:So this was my overview of what I wanted to call debugging on the web and how does DevTools
  884. Jake:Absolutely.
  885. Surma:know what code to show you and also why I guess we both have beef with source maps.
  886. Surma:We both have beef with source maps and I really hope that in the standardization that
  887. Jake:Oh
  888. Surma:they learn from where source maps are falling short and then either make something that
  889. Surma:is long lasting and doesn't have these problems, or maybe they should use DWARF.
  890. Surma:I don't know.
  891. Surma:I mean, source maps are already big.
  892. Surma:DWARF is also big.
  893. Surma:I don't think you win or lose a lot either way.
  894. Surma:And using existing format is usually much more beneficial than inventing something new.
  895. Jake:Oh, absolutely.
  896. Jake:It's had more hours of work put into it.
  897. Surma:Yeah.
  898. Surma:DWARF is extremely battle tested.
  899. Surma:So I feel like it.
  900. Surma:I don't know.
  901. Surma:I'll guess we'll see what they do.
  902. Jake:Excellent. Well, we'll link to your Wasmphobia thing,
  903. Jake:which is, yeah, it's a lovely little project.
  904. Jake:It is impressive, because my assumption was always, like,
  905. Jake:well, once it's a binary, then your ability to really understand
  906. Jake:where things in there came from,
  907. Jake:I just assumed that that was impossible now.
  908. Jake:But it's, yeah, and especially on the web,
  909. Jake:where we do care about those bytes,
  910. Surma:Yeah.
  911. Jake:a lot of the image encoders and decoders we were dealing with
  912. Jake:would have, like, except PNG as an input,
  913. Surma:Yeah.
  914. Jake:but we knew, like, we were just able to pass in raw bytes of an image
  915. Jake:and sort of determining, like, have I definitely created a build
  916. Jake:that doesn't have the PNG encoder and decoder in there?
  917. Surma:Yeah.
  918. Jake:And I was just like, I'm going to assume,
  919. Jake:because I added these flags and the bundle is now smaller,
  920. Jake:that I have succeeded.
  921. Jake:But I don't know, it feels like it should have got more smaller
  922. Jake:than it did, but I don't know.
  923. Surma:You know what?
  924. Surma:And now we could actually probably figure this out a bit more because we could because
  925. Jake:Yeah.
  926. Surma:Emscripten emits DWARF as well.
  927. Jake:Oh, nice.
  928. Surma:If you add the dash G full flag, I think I have it in the readme of isomphobia and we
  929. Jake:Ooh.
  930. Surma:could drop it in and we could actually take a look of like, is it a PNG dot header file
  931. Surma:or PNG dot C files that they still include it somewhere.
  932. Jake:Yeah.
  933. Surma:I actually would be more confident that I can inspect what's in there and we could figure
  934. Surma:out whether there's more that we need to do.
  935. Surma:And I guess it's again, it's the same thing in native land.
  936. Surma:This kind of tree shaking is just not a requirement, not something that is being thought of when
  937. Jake:Yeah.
  938. Surma:you write code, because even in Android land, right, like whether your Android app is eight
  939. Surma:megabytes or 12 megabytes, nobody cares.
  940. Surma:I mean, there is a limit at some point where you go like, wait, why is this app to take
  941. Surma:notes?
  942. Surma:120 megs.
  943. Surma:But if you below a certain threshold, nobody cares really about squeezing out the last
  944. Surma:couple of bytes.
  945. Surma:But on the web, it's very, very different.
  946. Jake:And I think we were both impressed, like,
  947. Jake:somewhere where it did happen differently,
  948. Jake:we were both impressed by Animal Well,
  949. Surma:Yes.
  950. Jake:the kind of Metroidvania game, which I haven't played yet,
  951. Jake:but I've watched videos of it and it looks really good
  952. Jake:and I really want to play it.
  953. Jake:But this is, like, when you get a game on the PlayStation,
  954. Jake:it's, yeah, it's 100 gigabytes or whatever,
  955. Jake:and if you dare install it from a CD,
  956. Surma:Yes, easily.
  957. Jake:because you think that, a CD, a Blu-ray, whatever,
  958. Surma:Yes, easily.
  959. Jake:because you think that's going to be quicker,
  960. Jake:it's still going to be hit with that 50 gigabyte update
  961. Jake:afterwards, you know?
  962. Jake:But Animal Well is 34 megabytes,
  963. Jake:which is, like, for a game these days,
  964. Jake:that's smaller than Minesweeper is on Windows these days, so...
  965. Surma:That's more than some web apps that are out there.
  966. Jake:Oh, many, many.
  967. Jake:Funny detail of it is, like, on the PS5, it's not 34 megabytes,
  968. Jake:it's 101 megabytes.
  969. Jake:But the theory is that, you know,
  970. Jake:when you highlight the game's icon on the PlayStation,
  971. Jake:you get, like, a background image shows up,
  972. Surma:Yes.
  973. Surma:Yes.
  974. Surma:Yes.
  975. Jake:which would be, like, I don't know, 4K or 8K or whatever,
  976. Surma:Yes.
  977. Surma:Yes.
  978. Surma:Yes.
  979. Surma:Yes.
  980. Jake:and the theory is that's most of the size increase.
  981. Surma:Yes.
  982. Surma:Yes.
  983. Jake:So that background image that appears on the PlayStation menu
  984. Surma:Yes.
  985. Surma:Yes.
  986. Surma:Yes.
  987. Surma:Yes.
  988. Jake:is actually bigger, bigger than the game itself, yeah.
  989. Surma:Yes.
  990. Surma:Yes.
  991. Jake:Which is, you know, that is the right way around.
  992. Jake:Like, there are so many websites where a video of the website
  993. Jake:ends up being smaller than the code to make it run.
  994. Surma:It always makes me think of the demo scene in the mid-2000s.
  995. Jake:Oh, yeah.
  996. Surma:There was this game called KKrieger, I think, which was 100K and had, for the time, stunning
  997. Surma:visual 3D graphic.
  998. Surma:And it was basically like a Doom first-person shooter.
  999. Surma:And, you know, obviously the enemies were quite dumb, but it was incredible, it was
  1000. Surma:super impressive.
  1001. Surma:Actually, I'm going to link to that as well.
  1002. Jake:And things like Frontier on the Amiga,
  1003. Jake:where it's seemingly the universe fit on a floppy disk somehow.
  1004. Jake:There was one...
  1005. Jake:I mean, we talked about it on the show before,
  1006. Jake:but there was a few games on the Commodore 64
  1007. Jake:where they would give you a game to play
  1008. Jake:while the game was being loaded,
  1009. Jake:which I always thought was, like,
  1010. Jake:well, that's a lovely little bit of progressive loading,
  1011. Jake:but there was another reason that they did that,
  1012. Jake:and the reason was the hardware code on the Commodore 64
  1013. Jake:for loading data from the tape or from floppy disks
  1014. Jake:into memory was bad.
  1015. Jake:Like, they hit a bug close to shipping the machine out,
  1016. Jake:and it was like, this bug's so bad,
  1017. Surma:Wow.
  1018. Jake:we've got to do the whole loading thing in it
  1019. Jake:in a different and terrible way just for it to not be buggy.
  1020. Jake:And then so after it shipped, people found,
  1021. Jake:it's like, well, we could load in a loader that can do it better,
  1022. Jake:because better ways of doing it on software were found later on.
  1023. Jake:So while you were actually playing this little game,
  1024. Jake:it was actually also loading its own loader,
  1025. Jake:which could load the rest of the game into memory faster.
  1026. Jake:Yeah, which is brilliant, isn't it? I love that.
  1027. Surma:Some of these tricks at the time, I remember, I don't know which game it was, I'll look
  1028. Surma:it up, but there was a game which, again, shortly before shipping, was crashing when
  1029. Surma:you finished the game, like when you beat it.
  1030. Surma:And they couldn't figure out why it crashed.
  1031. Surma:So basically when you finish the game, it's just like, segfault, blah, something.
  1032. Jake:Oh, yes.
  1033. Surma:And so what they did, because at the time this was possible, when they loaded the game,
  1034. Surma:they would find that string in memory, like the string that contains the error message
  1035. Surma:for a segfault, and overwrite it with a message, thank you for playing whatever the game was
  1036. Jake:It was Wing Commander.
  1037. Jake:I'm sure we've talked about that on the show before.
  1038. Surma:called.
  1039. Jake:Yeah, thank you for playing. Wing Commander was the game.
  1040. Surma:And then they just ship it.
  1041. Jake:Yeah, yeah, I love that game as well.
  1042. Surma:Really?
  1043. Surma:Brilliant.
  1044. Jake:Nah, well, whatever.
  1045. Surma:Well, then we can cut this.
  1046. Jake:No, look, look, people have said,
  1047. Jake:like, one of the things they like about the show
  1048. Jake:is it's kind of like, it's almost like, you know,
  1049. Jake:they're just sort of down the pub with us.
  1050. Jake:And if there's one thing I do down the pub,
  1051. Jake:it's tell the same story over and over again.
  1052. Jake:Absolutely.
  1053. Jake:Oh, absolutely.
  1054. Jake:Well, with that, then, we shall say,
  1055. Jake:happy next time!
  1056. Surma:What a brilliant point to end this episode, I think.
  1057. Surma:We've told the same stories, so before we lose more credibility, we can call it.
  1058. Surma:Happy next time!
  1. Why Source Maps don’t always work

    Why Source Maps don’t always work

  2. Putting React In The Browser

    Putting React In The Browser

  3. Canvas-based Web Apps

    Canvas-based Web Apps

  4. The Apple PWA Ban

    The Apple PWA Ban

  5. TC39 Roundup and Bevy’s ECS

    TC39 Roundup and Bevy’s ECS

  6. The Big Build Bool Bonanaza II

    The Big Build Bool Bonanaza II

  7. WebGPU and Browser Ideologies

    WebGPU and Browser Ideologies

  8. From the Archive — Changing jobs, Deno, and optimizing animations

    From the Archive — Changing jobs, Deno, and optimizing animations