Why Source Maps don’t always work

  1. Jake:Welcome to another episode of... Sorry, I went in too strong there, and I felt like I couldn't sustain it, so I'm backing out.
  2. Jake:Let's maybe start softer, and we can build up from there.
  3. Surma:Starts off to, like, hello, hi. And me, Surma.
  4. Jake:Hello. Hello. Welcome to another edition of OTMT, with me, Jake Archibald.
  5. Jake:Excellent. Okay.
  6. Surma:Welcome to the Most Mindful podcast in web development.
  7. Jake:Oh, it's a little bit NPR now, isn't it? We need to pick it up from here.
  8. Jake:I should say, thank you to Shopify for letting us do this, and hosting it, and that kind of stuff. Thank you very much.
  9. Surma:Yeah.
  10. Surma:Thank you, appreciate it.
  11. Jake:So, I was in Newcastle recently, visiting some friends, and I did something I've been wanting to do for a while.
  12. Jake:I did the Chris Jericho Triangle. Have you heard of this? Are you familiar with the Chris...?
  13. Surma:No, I'm assuming it's not a musical instrument?
  14. Jake:No. Chris Jericho is a Canadian wrestler. He was in the WWF when I was a teenager, and then obviously WWE afterwards.
  15. Surma:You say obviously. I don't even know what those letters stand for. But I'm guessing it's different wrestling leagues?
  16. Jake:What was the WWF?
  17. Jake:The World Wrestling Federation was the WWF, and then they had a bit of a dispute with the World Wildlife Fund, and I assume they lost.
  18. Jake:No, there was a cage match, but the problem is, like, the WWF, as in the wildlife one, they brought tigers and bears.
  19. Surma:Was this dispute real or was it just them acting?
  20. Jake:And, you know, man, like, they were just not... there was so, so much blood. So much blood.
  21. Surma:Many people died.
  22. Jake:And now the Wrestling Federation are called WWE instead, and that was many, many years ago.
  23. Surma:And the E stands for?
  24. Jake:Entertainment. Come on.
  25. Jake:How did you get into wrestling?
  26. Surma:World Wrestling Entertainment. It kind of gives it away. I thought the whole point... I remember when I went to school, like, primary school.
  27. Surma:I don't think it was particularly popular in Germany, wrestling overall.
  28. Jake:How did you get into wrestling?
  29. Surma:But it was like, it's all just acting. And other people were like, no, it's not. It's real. Look at them. They're bleeding.
  30. Surma:And it's just like big, big mystery and conspiracy on the primary school playground.
  31. Jake:At school, I remember being sort of made fun of for watching wrestling by a bunch of girls who were really into, like, boy bands.
  32. Jake:Like, they were really into Westlife, and I was proud of my...
  33. Surma:Well, clearly, people in glass houses.
  34. Jake:Well, that was my response. I was like, well, what's the difference between a wrestling fan and a Westlife fan?
  35. Jake:The wrestling fan knows it's fake. You know, I was quite pleased with myself.
  36. Jake:But yeah, no, no, we knew it was just soap opera, you know?
  37. Jake:Anyway, Chris Jericho Triangle.
  38. Jake:This is, oh, I guess a good few years now ago, Chris Jericho visited Newcastle.
  39. Jake:And he posted on Instagram, great day wandering around beautiful hashtag Newcastle getting ready for the sold out show.
  40. Surma:Well, it's just three different sides of the same building.
  41. Jake:And he posted three pictures of his, you know, wandering around beautiful Newcastle.
  42. Jake:And so I, you know, I wanted to walk in his footsteps.
  43. Jake:And I went to the same three places and took the same three photos of myself.
  44. Jake:All three photos were within a 10 meter radius of each other.
  45. Surma:Well, I mean, that's right up your street.
  46. Jake:Not just that, outside the Wetherspoons.
  47. Jake:Yeah, well, I don't know.
  48. Surma:Did you have an honorary pint in each Weatherspoons?
  49. Jake:Oh, no, no, one Wetherspoon.
  50. Jake:Look, they're all in front of the same Wetherspoons.
  51. Surma:Oh.
  52. Jake:Like, they just, just...
  53. Surma:Okay, let me do it again. Did you have three honorary pints in that Weatherspoons?
  54. Jake:No, I did, I did have one though.
  55. Jake:And how would you explain Wetherspoons to international listeners?
  56. Surma:Like the simple way to put it is a pub chain. If you want to go deeper into it, it's owned by a bigot. And technically, you should not be giving them money.
  57. Jake:Oh, yes.
  58. Jake:I would say it's like the McDonald's of pubs.
  59. Surma:Yeah, that's a good way to put it.
  60. Jake:And in the same way that they are both kind of like figureheaded by a clown.
  61. Jake:Yeah, Tim Spoon, who owns Wetherspoons is a bit of a wolf.
  62. Jake:Yeah, a bit of a problem person.
  63. Surma:That name. You really don't have much choice in life, I feel like.
  64. Jake:Oh, that's not his real name.
  65. Jake:I can't remember his real name, but he looks a little...
  66. Jake:Well, and I'm stealing a joke here.
  67. Jake:He looks a bit like a lino from the Thundercats.
  68. Jake:It was if lino had a racist uncle, he looks like.
  69. Jake:We'll put a link to a picture of, I think it's Tim Martin.
  70. Jake:But I don't know, Tim Spoon.
  71. Surma:I think I prefer that. He shall henceforth be known.
  72. Surma:We've probably mentioned this before, but we have the inverse law of effort. The more effort you put into a blog post, a library or anything really, the less people are going to give a shit and understand what it's about.
  73. Jake:Have you ever thought about blogging about things you've just learned?
  74. Surma:And if you do a quick throwaway article, tweet, mini hack library, people go nuts on average. Less effort, more interest.
  75. Jake:Yes, a thing of this has been happening to me recently
  76. Jake:that I used to blog about things I had just learned.
  77. Jake:And because I was excited, I've just learned this thing.
  78. Jake:I'm excited about learning this thing.
  79. Jake:I'm going to write it down.
  80. Jake:And yeah, like very, very variable results on that.
  81. Jake:Whereas recently I've done a couple of posts
  82. Jake:where it's kind of just stuff I know from being around so long.
  83. Jake:And it's like, oh, now that's a thing.
  84. Jake:That's an unusual piece of entertainment
  85. Jake:that I've just been around so long.
  86. Jake:To sort of know how these things started.
  87. Jake:And it kind of makes me feel very old.
  88. Surma:That is something that many people do not have. The origins of weird attribute quirks.
  89. Jake:But, you know, people have been reading them, which is nice.
  90. Jake:Have you ever thought about age?
  91. Jake:Age.
  92. Surma:So I had this thing where I needed to scratch an itch. I've been working with a lot of WebAssembly stuff, mostly from C and Rust.
  93. Surma:And, you know, in JavaScript, we have a couple of ways to figure out what a big JavaScript bundle consists of.
  94. Jake:Oh, yes, you get those lovely little diagrams.
  95. Jake:There's plug-ins for Webpack and Rollup.
  96. Surma:Yeah, my go-to is usually Source Map Explorer, which is on NPM. And you just give it your source map. And then it gives you, I think it's called a tree map, which is a rectangle.
  97. Jake:Yes.
  98. Surma:And it partitions that rectangle into areas of proportional size to how much any given entity consumes.
  99. Surma:So if you have 20% is your code, 80% is React, that rectangle is going to be divided into 80% for a rectangle called React and 20% for a rectangle called your code.
  100. Jake:Yes.
  101. Surma:And it does this by folder structure. So there will be a big rectangle for node modules, but that in itself is going to be subdivided.
  102. Jake:Yes.
  103. Surma:And it's quite a nice way to figure out why is this JavaScript bundle 1.2 megabytes.
  104. Jake:Historically an amazing way to find out
  105. Jake:where all of your hard drive space is going.
  106. Jake:That's what these tools use.
  107. Jake:And it works exactly the same way for a JavaScript bundle.
  108. Surma:Oh, true.
  109. Surma:I feel like Source Map Explorer hasn't been maintained super well. Like, over time, it seems that it's not... I wonder if it doesn't support the newer source map versions.
  110. Surma:Because, like, there's more recent bundles, like where the Vite spits out just has, like, a big unknown area, but then other tools get it right.
  111. Jake:Yes.
  112. Surma:Anyway, what I needed is that, but for my WebAssembly binary, because it got bigger.
  113. Jake:Hmm.
  114. Surma:And I wanted to, you know, in Rust, I just wrote my code, but I also pulled in some crates, as libraries are called in Rust.
  115. Surma:And I was like, well, I wonder if it's this or that library that is actually making this WebAssembly binary bigger.
  116. Surma:And I went on a journey to try and figure out where can I get this data and can I somehow visualize it.
  117. Jake:Yes.
  118. Surma:And I want to tell you about it, Jake.
  119. Surma:Well, actually, I also want to tell you about source maps, because that was a whole parallel journey.
  120. Jake:Yes.
  121. Surma:So, I guess, actually, now that I say all this, what I want to call this episode is debugging on the web, because both of these things are related to that.
  122. Jake:Yes.
  123. Surma:But, yeah, so that's what we're going to talk about.
  124. Surma:If you take a step back, you know, we have JavaScript and it gets bundled, it gets minified, it gets all kinds of...
  125. Jake:Yes.
  126. Jake:I think your title is correct.
  127. Jake:I just don't think it's going to do well on the socials.
  128. Surma:Oh, that's true.
  129. Jake:I think we need to think of, like,
  130. Jake:I looked at what was in my WebAssembly binary
  131. Surma:I could also... I have beef with source maps.
  132. Jake:and I couldn't believe what I found.
  133. Jake:Something like that, right?
  134. Jake:Come on.
  135. Surma:There you go.
  136. Jake:Oh, yeah.
  137. Jake:So do I.
  138. Surma:Oh, maybe I should have graded with source maps.
  139. Jake:No, I have gravy with beef.
  140. Jake:But, yeah.
  141. Surma:So, I don't know which way to start, but I think I'm going to start with source maps,
  142. Jake:Maybe.
  143. Surma:because I'm assuming our audience is, on average, slightly more familiar with their existence and what they're commonly used for.
  144. Jake:Maybe.
  145. Surma:And what I learned while doing the research for this is that source maps were kind of like an ad hoc invention by an engineer at Google working on Gmail.
  146. Jake:Oh, okay.
  147. Surma:Gmail, probably using GWT at the time or whatever, but the JavaScript in the end was compiled by Closure Compiler,
  148. Surma:which, you know, for a long time was deemed to be the strongest JavaScript optimizer tool out there, until, I think, it was superseded.
  149. Jake:And even then, like, you know,
  150. Jake:sometimes I'll say on Twitter,
  151. Surma:But there was a time where you would add Closure Compiler to your Gulp toolchain.
  152. Jake:I'm importing a function from a file
  153. Jake:and then another file is importing
  154. Jake:another function from that file.
  155. Surma:I think Closure Compiler is not to take away from all our bundlers and minifiers,
  156. Jake:I wish bundlers were clever enough
  157. Jake:to split that out.
  158. Jake:You know, as in, you know,
  159. Jake:each function goes separately
  160. Jake:into the individual bundles.
  161. Jake:And then someone will go,
  162. Jake:um, actually, um,
  163. Jake:Closure Compiler can do it.
  164. Jake:So, yeah.
  165. Jake:Closure Compiler can do it.
  166. Jake:So I still think there's lots of stuff in there
  167. Jake:that, you know.
  168. Surma:but it's a proper compiler.
  169. Jake:Mmm.
  170. Surma:Like, it has a lot more of these techniques built in that we usually associate with compilers.
  171. Surma:And at the time, I think it was all kinds of stuff.
  172. Surma:You could add extra annotations, and it was really quite good.
  173. Surma:But the problem was that at the time, and we're talking like 2009 here,
  174. Surma:this engineer working on Gmail was trying to debug Gmail and the JavaScript that Closure Compiler had spat out,
  175. Jake:Mmm.
  176. Surma:which also, because Google at the time was already aware that every byte counts,
  177. Surma:had minified variable names, all kinds of minification tricks.
  178. Jake:Mmm.
  179. Surma:Really hard to look at and debug in Firebug at the time.
  180. Jake:Oh, wow, yeah.
  181. Surma:So this guy just sat down and basically implemented a little thing into Closure
  182. Surma:that would keep track of the code that's just being emitted.
  183. Surma:What does it correspond to in the original input files?
  184. Jake:Mmm.
  185. Surma:And that's kind of what source maps are.
  186. Surma:Like, it just gives you for every point in your generated code,
  187. Jake:Mmm.
  188. Surma:it has a way to tell you what is the position in the original source that corresponds to it.
  189. Jake:Mmm.
  190. Jake:Mmm.
  191. Surma:And there's some extra bits involved.
  192. Surma:But basically, he just did this to help himself debug.
  193. Jake:Mmm.
  194. Surma:And it kind of picked up steam.
  195. Surma:And here we are with source maps now being the...
  196. Jake:Mmm.
  197. Jake:Mmm.
  198. Surma:It's not a standard.
  199. Surma:It has a spec-ish that's just a Google doc that has been living for over 13 years now
  200. Surma:and has been worked on last year still.
  201. Surma:But it is kind of a de facto standard of JavaScript tooling, isn't it?
  202. Jake:Yes.
  203. Jake:And I do think there is now an effort
  204. Jake:to do it better, right,
  205. Jake:to actually sort of take it into TC39.
  206. Surma:Oh, is there?
  207. Jake:Now, I'm kind of going from memory here,
  208. Jake:but I remember that being some chatter
  209. Jake:that the DevTools teams were wanting
  210. Jake:because, you know what,
  211. Jake:source maps aren't perfect.
  212. Jake:The amount of projects where I just go into DevTools
  213. Surma:Yeah.
  214. Jake:and turn them off
  215. Jake:because I feel like I'm being lied to
  216. Jake:by DevTools, you know.
  217. Jake:And it's like, look,
  218. Jake:I just pressed, you know,
  219. Jake:skip the next line
  220. Jake:and something else happened.
  221. Jake:You know what I mean?
  222. Jake:That's not what just happened.
  223. Jake:And I'm trying to debug this properly.
  224. Jake:I want to debug what's actually running,
  225. Jake:because I'm looking at...
  226. Surma:Exactly.
  227. Jake:So I think there is some appetite
  228. Jake:to kind of revisit it
  229. Jake:and solve the problems that exist
  230. Surma:Yeah, I wanted to try to explain a little bit what is in a source map.
  231. Jake:and actually standardize it properly.
  232. Surma:On the one hand, because it helped me understand what it can and cannot do
  233. Jake:Mmm.
  234. Surma:and also explain why these weird behaviors sometimes happen.
  235. Surma:So I guess the first thing we need to figure out is,
  236. Surma:how does a browser, when you debug code, even know that a source map is available?
  237. Surma:And usually, that's a special comment.
  238. Jake:Yes.
  239. Surma:I think typically at the end of the file,
  240. Surma:which is like a slash slash hash bang,
  241. Surma:and it says the source map URL equals,
  242. Jake:And it could be inline as well, right?
  243. Surma:and it's a URL to the actual source map file,
  244. Jake:Like inline is an option.
  245. Jake:Hmm.
  246. Surma:which typically has the same file name with .map at the end.
  247. Jake:Ah.
  248. Jake:And it could be inline as well, right?
  249. Jake:Like inline is an option.
  250. Surma:That's what I was going to say.
  251. Surma:It is a URL.
  252. Jake:Hmm.
  253. Surma:So a data URL is completely fine.
  254. Jake:Ah.
  255. Surma:And it's not uncommon.
  256. Surma:And that's usually called an inline source map.
  257. Surma:But whatever that file is,
  258. Surma:whether it's an inline base64 encoded data URL
  259. Surma:or the other file that you actually have to fetch,
  260. Surma:that file is just a JSON file with a handful of properties, really.
  261. Jake:Hmm.
  262. Surma:So what is in there?
  263. Surma:One thing I found is that Evan, who did ESBuild,
  264. Jake:Hmm.
  265. Surma:also wrote a source map visualizer.
  266. Surma:This is not like a bundle analyzer to tell you
  267. Surma:this source file is consuming this many bytes in a bundle,
  268. Surma:but it actually visualizes the original source code
  269. Surma:next to the minified source code.
  270. Surma:So when you hover over a bit,
  271. Surma:it shows an arrow to where it ended up on the other side,
  272. Jake:Oh, nice.
  273. Surma:and you can literally see which files have been mangled together
  274. Surma:in different ways.
  275. Jake:Oh, we'll link to that, because I want to play with that.
  276. Surma:Yeah, absolutely.
  277. Surma:That will, of course, be in the description.
  278. Surma:There's also going to be a link to the spec
  279. Jake:Oh.
  280. Surma:and to a Hacker News comment from the inventor of source maps,
  281. Surma:that Gmail engineer, which has a bit of details on the history.
  282. Surma:But yeah, it is just a JSON object,
  283. Surma:which really, it is quite straightforward, I want to say.
  284. Surma:So it has the name of the file that this source map is for.
  285. Surma:So the generated file links to the source map.
  286. Surma:The source map links to the generated file.
  287. Surma:So you just know how to map from one to the other.
  288. Surma:You have a list of the original source file names
  289. Surma:that were incorporated.
  290. Jake:Hmm.
  291. Surma:That's mostly because you don't want to repeat the string
  292. Surma:multiple times later on, but just like, say,
  293. Surma:this is source file one, source file two,
  294. Surma:which makes the whole source map smaller,
  295. Surma:even though they're already quite big.
  296. Surma:There is sometimes a list of symbols.
  297. Surma:This helps you unminify variable or function names.
  298. Surma:So it tells you, ah, this function is called O in the output,
  299. Surma:but the original name was delete all files on hard drive
  300. Jake:Hmm.
  301. Surma:or whatever, like it tells you that.
  302. Surma:And then the big part is the mapping property,
  303. Jake:Hmm.
  304. Surma:which is a very long string.
  305. Surma:And looking at it now, I actually am surprised it's a string
  306. Jake:Hmm.
  307. Surma:because it is really a two-dimensional array.
  308. Surma:That's for some reason has been string encoded.
  309. Surma:I'm not sure what the reasoning behind it is.
  310. Surma:I wonder if it just compresses better. I don't know.
  311. Surma:So basically it's a very long string.
  312. Surma:And if you were to split it by semicolon,
  313. Surma:each entry in that resulting array represents one line
  314. Surma:in the output file.
  315. Jake:OK.
  316. Surma:And then each of these lines you can split by comma,
  317. Surma:which gives you an array of segments in that line.
  318. Jake:Hmm.
  319. Surma:And segments are these little blobs in the output file
  320. Surma:that can be mapped to a source file.
  321. Surma:Now, I'm not going to try to explain the actual encoding,
  322. Surma:but it's a bit like UTF-8.
  323. Jake:Hmm.
  324. Surma:It's called VLQ.
  325. Surma:Basically, it's just up to five numbers,
  326. Jake:Hmm.
  327. Surma:which is like what file is it,
  328. Surma:what line in the original source,
  329. Jake:Hmm.
  330. Surma:what column in the original source,
  331. Surma:and potentially what original name for this variable.
  332. Jake:Hmm.
  333. Surma:But it really just maps a point to a point.
  334. Surma:And this is one of the things I realized
  335. Jake:Hmm.
  336. Surma:where things get iffy with source maps
  337. Surma:because it doesn't really tell you the range that it maps to.
  338. Jake:Hmm.
  339. Surma:So like in the generated code,
  340. Surma:you obviously just have these list of segments
  341. Surma:and they are going to be,
  342. Surma:if you say one segment starts in column five
  343. Surma:and the next segment starts in column eight,
  344. Surma:you know the first segment is three characters long
  345. Surma:or something.
  346. Surma:But what it doesn't tell you of where it maps to,
  347. Surma:how long that is.
  348. Surma:And I think that's where the problem sometimes comes in
  349. Surma:where you're technically setting a break point
  350. Jake:Hmm.
  351. Surma:in the middle of a segment,
  352. Surma:but DevTools can only set the break point to the start
  353. Surma:or jump to the start.
  354. Surma:And especially with minifiers,
  355. Surma:putting multiple statements into one line
  356. Surma:and using commas instead of semicolons
  357. Surma:and potentially skipping certain things,
  358. Surma:I don't think the mapping it can express is powerful enough
  359. Surma:to handle all modern minification techniques.
  360. Surma:But I'm going to get a bit more into that
  361. Jake:Oh, interesting.
  362. Surma:because it's a very different story in WebAssembly
  363. Surma:that's quite interesting.
  364. Surma:So yeah, basically what the browsers do,
  365. Jake:Hmm.
  366. Surma:you know, stepping through the generated code,
  367. Surma:the minified code is quite easy for a JavaScript engine.
  368. Jake:Hmm.
  369. Surma:And then whenever it wants to figure out
  370. Jake:Hmm.
  371. Surma:where it currently is,
  372. Surma:it just basically does a binary search
  373. Jake:Hmm.
  374. Surma:through this mapping data
  375. Surma:and figure out what is the correct corresponding source file.
  376. Jake:Hmm.
  377. Surma:And then it can either fetch the source file
  378. Jake:Hmm.
  379. Surma:or sometimes the contents of the source files
  380. Jake:Hmm.
  381. Surma:are also embedded in the source map itself,
  382. Surma:which makes them even bigger.
  383. Surma:But that allows you to debug in production as well.
  384. Jake:I think that's the more common way of doing it, right?
  385. Jake:Like that's, I think...
  386. Surma:I think so, yeah.
  387. Jake:Yeah.
  388. Surma:I guess sometimes if you want to debug in production,
  389. Surma:it might be good to strip it
  390. Jake:Hmm.
  391. Surma:because you don't necessarily want to leak your source code.
  392. Surma:But if you have the source code locally available,
  393. Surma:you can give it to DevTools
  394. Surma:so it can resolve it while debugging
  395. Surma:while people who are out there in the world
  396. Jake:Hmm.
  397. Surma:can't really deduce anything meaningful from it.
  398. Jake:I was going to say, does that
  399. Jake:actually matter
  400. Jake:because, you know, your minification
  401. Jake:shouldn't be a security step, but
  402. Surma:Yes.
  403. Jake:in the sort of
  404. Jake:newer world of remix
  405. Jake:and sort of
  406. Jake:the React server components stuff
  407. Jake:where you have, like,
  408. Jake:your database code in the same
  409. Jake:file as your client code
  410. Jake:I guess that's where it would become
  411. Jake:potentially difficult, right?
  412. Surma:I would agree that minification,
  413. Surma:all this stuff is not a security primitive.
  414. Surma:But for certain things.
  415. Surma:It's always the same.
  416. Surma:On my screen, technically, I can download it.
  417. Jake:Hmm.
  418. Surma:But sometimes you don't want that,
  419. Surma:while still allowing people to look at stuff on screen.
  420. Jake:Right.
  421. Surma:And so I think you just want to make it hard.
  422. Surma:And so debugging could also make it harder
  423. Jake:Yeah, okay.
  424. Surma:for people to actually inject code in the right place
  425. Surma:if they want to write a malicious browser extension.
  426. Surma:So I think, yeah, it's not a good protection.
  427. Surma:It's just a hurdle.
  428. Surma:But it shouldn't be at all...
  429. Surma:It should be the only thing you do
  430. Surma:to try and keep your users safe.
  431. Jake:Oh, but you definitely wouldn't want your server
  432. Jake:code crossing that boundary, which technically
  433. Surma:For sure.
  434. Jake:in a build system
  435. Surma:No, that's actually a good point.
  436. Surma:So there, in that world,
  437. Surma:it's probably more important to do that.
  438. Surma:To not necessarily have your source maps
  439. Surma:contain the full source code of all your source files.
  440. Jake:you wouldn't want.
  441. Surma:This is already the end of what source maps contain.
  442. Surma:So it's really just a list of segments
  443. Surma:that map to your original source.
  444. Surma:And so what Source Map Explorer does,
  445. Surma:and what I now did as well,
  446. Surma:in the tool I wrote that I'm going to talk about in a little bit,
  447. Surma:you basically just iterate over all the segments
  448. Surma:in the generated code.
  449. Surma:You figure out how long they are
  450. Surma:by looking where does the next segment start.
  451. Surma:And then map that,
  452. Jake:Hmm.
  453. Surma:like figure out what does the segment
  454. Surma:correspond to my original source.
  455. Surma:Actually, not really what,
  456. Surma:but just like which file did the segment come from.
  457. Jake:Mm-hmm.
  458. Surma:And so I just built a dictionary
  459. Surma:where I have counters for each source file.
  460. Surma:And when I find a segment that is eight bytes long,
  461. Surma:I'm going to increase the count of eight.
  462. Surma:And when I go through the entire source map,
  463. Surma:I now have a dictionary where for each file,
  464. Surma:I know how many bytes it contributed in the bundle.
  465. Surma:And now I can visualize it.
  466. Surma:So with source maps,
  467. Surma:it actually was quite easy to like build
  468. Surma:almost like a clone of Source Map Explorer.
  469. Jake:Hmm.
  470. Jake:Nice.
  471. Surma:However, I told the story historically a bit reversed
  472. Surma:because originally, as I kind of,
  473. Surma:I guess already hit it at the start,
  474. Surma:is I needed this for WebAssembly.
  475. Surma:And WebAssembly doesn't use source maps,
  476. Surma:which is now very understandable to me
  477. Surma:because like it's a compile target.
  478. Surma:And most compilers already have a way to allow debugging.
  479. Surma:And that is with DWARF,
  480. Surma:which is I guess the de facto standard
  481. Jake:And this is a cross
  482. Surma:or an actual standard for debugging symbols
  483. Surma:in native binaries.
  484. Surma:Yeah, so the DWARF, the spec is very much explicitly
  485. Jake:difference, like, it's not a rust
  486. Jake:thing, it's not a...
  487. Surma:language and architecture agnostic.
  488. Surma:So it's not just for Intel processors or ARM processors.
  489. Surma:It doesn't care.
  490. Surma:And it doesn't care about the source language.
  491. Surma:And it's been very longstanding.
  492. Surma:It's now in the fifth version.
  493. Surma:And basically that is what GDB or LLDB use
  494. Jake:Hmm.
  495. Surma:when you want to step through a binary.
  496. Jake:Hmm.
  497. Surma:And the whole design around DWARF
  498. Surma:is that it's very quick to parse,
  499. Surma:very quick to access or to find
  500. Surma:and access the things you're looking for.
  501. Surma:It is very capable and flexible.
  502. Jake:Hmm.
  503. Jake:So presumably this is working
  504. Jake:very similar to source maps, so your binary
  505. Jake:now has a bit of extra
  506. Jake:metadata that is able to
  507. Surma:Yeah.
  508. Surma:Yeah.
  509. Jake:map a certain
  510. Jake:bit of binary to
  511. Jake:essentially a sort of assembly-esque
  512. Jake:kind of stuff at the end, like map
  513. Jake:that to some source
  514. Surma:There is a lot of similarity.
  515. Jake:code.
  516. Surma:DWARF actually does even more,
  517. Surma:a lot more than source maps.
  518. Surma:And it's actually not easy to wrap your head around it
  519. Surma:because of how much it needs to be able to express.
  520. Jake:Hmm.
  521. Surma:And so like, I'm not an expert in DWARF.
  522. Surma:I started skimming the spec,
  523. Surma:but it is quite a lot.
  524. Surma:But I did learn a little bit about it
  525. Surma:while I was building my WebAssembly bundle analysis tool.
  526. Surma:And so I thought I would give, like,
  527. Surma:a really quick summary of what is in there
  528. Jake:Hmm.
  529. Surma:to just see where native land, I guess,
  530. Surma:has ended up to do these kind of debugging procedures.
  531. Surma:And I'm guessing at the time,
  532. Surma:because it was like born out of an ad hoc idea,
  533. Surma:it wouldn't have made sense to use DWARF.
  534. Surma:But now that I look at it,
  535. Surma:DWARF, because it is so widespread and supported
  536. Surma:in so many compilers,
  537. Surma:there's so much tooling out there
  538. Surma:that it might actually have been beneficial to use DWARF.
  539. Jake:Oh.
  540. Surma:But that's, you know, like hindsight is 20-20.
  541. Jake:Well, they might be
  542. Surma:So maybe...
  543. Jake:looking at it again, so yeah.
  544. Surma:So like DWARF itself is...
  545. Surma:I don't want to go too deep,
  546. Surma:but it's basically a tree data structure
  547. Surma:of what they call DIEs,
  548. Surma:debugging information entries.
  549. Surma:And those are really a bit like JavaScript objects.
  550. Surma:They have attributes,
  551. Surma:and these attributes have values.
  552. Surma:And there's a whole bunch of predefined attributes
  553. Surma:with specific meaning.
  554. Surma:But as far as I can tell,
  555. Surma:you can also add arbitrary custom attributes
  556. Surma:for certain reasons.
  557. Jake:Hmm.
  558. Surma:So in the end,
  559. Surma:what you kind of get is just debugging entries,
  560. Surma:and they form kind of a graph.
  561. Surma:And I could go a bit more into it,
  562. Surma:but I think it's actually not that interesting.
  563. Surma:If you have a WebAssembly binary,
  564. Surma:or actually any native binary with debugging symbol,
  565. Surma:there is a tool from LLVM.
  566. Surma:So that's, you know, Clang, Rust compiler,
  567. Surma:all LLVM-based.
  568. Surma:So usually you have this on your system
  569. Surma:called LLVM-DWARFDUMP.
  570. Surma:And if you give a binary to that,
  571. Surma:you can just get a big console dump,
  572. Jake:Hmm.
  573. Surma:and you can kind of scroll through
  574. Surma:and see what it means.
  575. Surma:But basically each debugging entry,
  576. Surma:each of these entries in the graph,
  577. Surma:has a source name and a source file,
  578. Surma:and a range or a list of ranges
  579. Surma:of where this debugging entry
  580. Surma:is occupying the binary.
  581. Surma:So it's actually a bit the other way around.
  582. Surma:You don't go through the binary
  583. Surma:and have mappings from segments to source file,
  584. Surma:but rather you have a debug entry
  585. Surma:that is like a function or a block or a variable.
  586. Jake:Hmm.
  587. Surma:And it tells you all the locations in the binary
  588. Surma:where this entity is appearing in different ways.
  589. Surma:So for example, if there was a function called main,
  590. Jake:Oh.
  591. Surma:that would usually just have, you know,
  592. Surma:source file, the name is main,
  593. Surma:and then probably just start at an endpoint
  594. Surma:because the main has been mapped to here.
  595. Surma:Then inside this main,
  596. Surma:there could, because it's a tree,
  597. Surma:there would be sub entries,
  598. Surma:whereas like in the main function,
  599. Jake:Hmm.
  600. Surma:I have a variable,
  601. Surma:and that is used here, here, and here.
  602. Jake:Hmm.
  603. Surma:And so that is already something
  604. Jake:Hmm.
  605. Surma:that Dwarf can express at source maps.
  606. Surma:Not necessarily could,
  607. Jake:Hmm.
  608. Surma:but you have this tie back to what is the same entity.
  609. Surma:And so I was like, oh cool, this looks easy enough.
  610. Surma:I'll just iterate over all the entries in this graph
  611. Surma:and count the bytes by the ranges they have.
  612. Surma:But the result was that my total byte count
  613. Jake:Hmm.
  614. Surma:was a multiple of the actual file I was trying to analyze.
  615. Surma:So clearly I was double, triple, quadruple counting bytes.
  616. Jake:Hmm.
  617. Surma:And that actually makes a lot of sense
  618. Surma:because we are now in compiled language land
  619. Jake:Hmm.
  620. Surma:and a bit like what Clojure did,
  621. Jake:Hmm.
  622. Surma:they do a lot more than what just bundlers
  623. Jake:Hmm.
  624. Surma:and minifiers mostly do.
  625. Surma:They do stuff like function inlining.
  626. Surma:So what used to be a function call,
  627. Surma:rather than doing a call,
  628. Jake:Hmm.
  629. Surma:they just copy the code from the function
  630. Jake:Hmm.
  631. Surma:at that place where you call the function
  632. Surma:to avoid the overhead of jumping to another function.
  633. Surma:Then Rust has generics or C++ has templates.
  634. Jake:Oh!
  635. Surma:There's this thing called monomorphization
  636. Surma:or specialization where you create an instance
  637. Surma:of a generic function because you're using it
  638. Surma:with a specific type.
  639. Surma:So if I have a function called add
  640. Surma:and it takes any type, when I call it with a float
  641. Surma:and then I call it with an integer,
  642. Surma:this function will be duplicated once for floats
  643. Surma:and once for integers.
  644. Surma:So this function now exists twice,
  645. Jake:Oh, interesting.
  646. Surma:even though the source code, it only exists once.
  647. Surma:And then, you know, compilers in general
  648. Surma:are allowed to like reorder and fragments.
  649. Jake:And I guess this is a little bit like when, well, in Emscripten,
  650. Jake:which I presume comes from the C compilers,
  651. Jake:where you can compile for performance or you can compile for file size,
  652. Jake:and I presume that's one of the differences,
  653. Jake:like the compile for file size is not going to do function in Lightning,
  654. Jake:but that is at the expense of performance.
  655. Jake:Right! Okay.
  656. Surma:Exactly.
  657. Surma:Yeah, so these flags with like what you optimize for
  658. Surma:is a stiff and optimization passes.
  659. Surma:And that's exactly it.
  660. Surma:If you optimize for size, you're saying,
  661. Surma:I'm okay having slower code if the binary is small.
  662. Jake:Mm-hmm.
  663. Surma:And let's go, okay.
  664. Surma:Not only am I not going to inline,
  665. Surma:but if I find the same code in different places,
  666. Surma:I'm going to create a new function and call it
  667. Jake:Oh, nice.
  668. Surma:rather than leave it.
  669. Surma:Like, I don't know if that is exactly what's happening,
  670. Surma:but I would assume that would be quite a trivial optimization
  671. Surma:to add to save file size.
  672. Surma:So that's what these flags usually decide,
  673. Jake:Mm-hmm.
  674. Surma:which kind of passes are enabled or disabled
  675. Surma:and how are they configured.
  676. Surma:So luckily there is, I want to give a shout out to Gimli,
  677. Surma:which is a rust crate to read dwarf files,
  678. Surma:but even more so that they have, right, right.
  679. Jake:Oh, I get it. I get it.
  680. Jake:Yeah. Yeah, very good. Very good.
  681. Surma:And they have an example program.
  682. Surma:Like here's how you use Gimli correctly.
  683. Surma:And an example program they have is add to line,
  684. Surma:address to line, which you give it a dwarf file
  685. Surma:or a binary with a dwarf file,
  686. Surma:and you give it an address,
  687. Jake:Mm-hmm.
  688. Surma:basically a memory address of where a function
  689. Surma:or whatever is.
  690. Surma:And it tells you which source file
  691. Surma:and which line did this come from?
  692. Surma:So I was basically able to use most of that code
  693. Surma:to now iterate through all the memory addresses
  694. Surma:that the program would be loaded to in memory
  695. Surma:and figure out which source file does that come from.
  696. Surma:So now we're kind of back to the previous approach
  697. Surma:I took with source maps,
  698. Surma:which byte corresponds to which source file,
  699. Surma:but still it gets tricky with this monomorphization stuff.
  700. Jake:Mm-hmm.
  701. Surma:Like when you create a new version of a function
  702. Surma:for a specific type,
  703. Surma:because this could be a generic function from a library,
  704. Surma:even the standard library,
  705. Surma:but now it gets instantiated
  706. Surma:for one of your types specifically.
  707. Surma:So do you attribute that to your library, your code?
  708. Surma:Do you attribute it to the core library,
  709. Surma:consumer price?
  710. Surma:And then after all the specialization kicks in,
  711. Jake:Right!
  712. Surma:now optimization comes.
  713. Surma:There was a really interesting example I found.
  714. Surma:Like imagine you have a function
  715. Jake:Mm-hmm.
  716. Surma:in your standard library to iterate over a 2D grid.
  717. Surma:You give it like a 2D grid
  718. Surma:and it calls a callback for every cell in this grid.
  719. Surma:And this function, quite sensibly,
  720. Surma:could be used for making each pixel in an image brighter,
  721. Surma:but also could give each unit
  722. Surma:in a specific region more health.
  723. Surma:It is very generic and universal.
  724. Surma:But let's say that me using this function
  725. Surma:to make each pixel brighter gets inlined.
  726. Surma:So rather than calling a function
  727. Surma:and having a nested for loop in that function,
  728. Jake:Mm-hmm.
  729. Surma:now this nested for loop is in my code.
  730. Surma:And now my callback is not a callback,
  731. Surma:but it's inlined there as well.
  732. Surma:And after optimizing it,
  733. Surma:it figures out that part of my callback code
  734. Surma:is actually independent of the inner loop
  735. Surma:because there's some pre-calculation per row
  736. Surma:that I could do.
  737. Jake:Because it feels like we've got a similar problem
  738. Jake:in sort of JavaScript standard stuff in that, like,
  739. Jake:because of tree shaking and dead code elimination,
  740. Jake:the size of, you know, some library's impact on your project
  741. Surma:Right.
  742. Jake:can be attributed or can be impacted by the other things that are calling it,
  743. Jake:which, you know, you can say, like, this library is 300K,
  744. Jake:but it's only 300K because it's being called by this, you know,
  745. Jake:or it could even be, like, you know, an import star
  746. Jake:and then an iteration over that,
  747. Jake:or something, like, has actually caused the size of that to explode.
  748. Surma:Yeah.
  749. Jake:Is that similar, or am I kind of way off?
  750. Surma:Somewhat.
  751. Jake:Way off.
  752. Surma:Somewhat.
  753. Surma:Like, that's why I didn't find bundle-phobia super useful, because it tells you this npm
  754. Jake:Hmm.
  755. Surma:package is this big.
  756. Surma:I was like, but, well, there's like three different versions in there, and hopefully
  757. Surma:it's three shakables if I only use one function.
  758. Surma:Like that's fine.
  759. Surma:So that's why I think actually analyzing your bundle, your output, is a lot more useful
  760. Surma:as a data point.
  761. Surma:But obviously it's, you can't say in general this library is big or this library is small.
  762. Surma:And you can't do that in native land either, because exactly that, things get removed or
  763. Jake:Hmm.
  764. Surma:reordered.
  765. Surma:The problem here, just what I realized, and this is for example I think not something
  766. Surma:that JavaScript bundlers or minifiers do, nested loop has been inlined, my callback
  767. Surma:has been inlined, and now some of my code has been moved in between the loops, but not
  768. Surma:all of it.
  769. Surma:So what used to be a very clear call stack, I call a function, that function calls my
  770. Surma:function back, now these quotes have been interleaved.
  771. Jake:Hmm.
  772. Surma:So there's not just a clear mapping of range to range, but now you have an interleaving.
  773. Jake:Hmm.
  774. Surma:And DWARF can express that as well.
  775. Surma:And that's something that source maps I don't think would be able to, apart from the fact
  776. Surma:that I don't think this kind of optimization happens very commonly in JavaScript lands.
  777. Surma:So yeah, that was just something I found very interesting.
  778. Surma:And also, when you ask for, in this code that I've used, what is the originating source
  779. Surma:file for this range, it doesn't just give you one file, it gives you a whole list of
  780. Surma:files, because inlining may have happened, it actually does happen quite commonly.
  781. Surma:So it will say, yeah, this address has four source files, because it was your main function,
  782. Jake:Hmm.
  783. Surma:but in this specific POC spot, it was originally calling a different function, has been inlined,
  784. Surma:and that function, also called a function, has been inlined.
  785. Surma:So you get this whole stack.
  786. Surma:So it makes it a bit unclear how you attribute the file size, like I just did whatever address
  787. Jake:Oh, that's fun. Yeah.
  788. Surma:to file decided to do, and it seems to be good enough.
  789. Surma:But it's quite interesting to look at WASM files and see that sometimes code from a different
  790. Surma:library is attributed to my code or the other way around, and it's just something you have
  791. Jake:Yeah.
  792. Surma:to deal with.
  793. Surma:There is, I don't think there's a very clear, correct answer in every case.
  794. Jake:I had a similar problem once when I was, like, trying to...
  795. Jake:Because a lot of these bundle analyzer things,
  796. Jake:like, you're dealing in source bytes, right?
  797. Jake:And sometimes, you know, something can look really big
  798. Jake:compared to another thing, but one of them gzips really well,
  799. Jake:and that's what we really care about, because that's the bytes over the wire.
  800. Jake:To some extent, you know, obviously gzip bombs are bad.
  801. Jake:But, yeah, you get that similar attribution problem
  802. Jake:because the order of the modules in a single file,
  803. Jake:like, you know, once they've been bundled together,
  804. Jake:that matters, because suddenly it's like,
  805. Jake:oh, this first one is massive, and this second one is really small,
  806. Surma:And then, you know, then there's stuff where the compiler has to inject stuff, it's just
  807. Jake:but you put them the other way around, and the back reference is, you know...
  808. Jake:It's kind of like, who owns the data being back referenced, right?
  809. Jake:It's kind of... It depends on the order.
  810. Surma:like in JavaScript land, where, example, a pure annotation, that the function is pure
  811. Surma:if you build a library, that has no source file, that's like a compiler-emitted part.
  812. Jake:Thank you.
  813. Surma:So those bytes are often without any mappings, and there's similar stuff in native land as
  814. Surma:well.
  815. Surma:So I was not really giving a good understanding of how DWARF works, but it is definitely a
  816. Surma:lot more capable.
  817. Surma:And looking back at it, I can see how DWARF would have been overkill for JavaScript, but
  818. Surma:at the same time, I do wonder, with all the tooling that already exists, maybe it would
  819. Surma:have been useful.
  820. Surma:Although I have to say, I could not find a tool that I can give a DWARF file or a binary
  821. Surma:with DWARF in it, that tells me which files have contributed how many bytes to this native
  822. Jake:You
  823. Surma:binary.
  824. Surma:And so I had to build that myself, like this kind of, I guess, in native land, this kind
  825. Surma:of bundle analysis isn't common, because only web is a streaming platform, right?
  826. Surma:I guess the one thing, it sets it apart from all other platforms.
  827. Surma:And so that's, in the end, what I wrote.
  828. Surma:It's called Wasmphobia, because even though I don't like bundle phobia, I thought the
  829. Jake:Nice
  830. Surma:name was quite funny.
  831. Surma:And so I published it.
  832. Surma:I also added source map support, because that was, after I was done with it, actually comparatively
  833. Surma:easy.
  834. Surma:And the fun fact is that, you know, the WebAssembly binaries can have DWARF embedded, so you can
  835. Jake:Oh
  836. Surma:just drag and drop a DWARF binary into Wasmphobia, but DevTools doesn't support DWARF.
  837. Surma:So when you step through a WebAssembly binary, it has its own format for the name of a function.
  838. Surma:So you see sometimes the name of the function, but you don't know what the source file is
  839. Surma:for this function.
  840. Surma:And, you know, Rust and C++ mangle function names quite excessively to encode what the
  841. Surma:module is that it came from and what the parameter types are.
  842. Jake:Oh
  843. Surma:Because in C++, you can have the function with the same name, but with different parameter
  844. Surma:types for dynamic dispatch and stuff like that.
  845. Surma:But luckily, at the time, Inga, who was on our Squoosh team and was a WebAssembly advocate
  846. Surma:on the Chrome DevRel team, wrote a Chrome extension that teaches DevTools about DWARF.
  847. Surma:So when you install that and you debug a WebAssembly binary, you now actually can step through
  848. Surma:your original source code.
  849. Jake:Oh, that's good.
  850. Surma:And when you throw an exception or something, you get an actual stack with your source files
  851. Jake:Oh, why is that not just in the browser?
  852. Surma:and lines rather than just random WebAssembly verses.
  853. Jake:That seems essential, right?
  854. Jake:But, yeah, very good.
  855. Surma:I know, right?
  856. Surma:I don't think variable inspection works.
  857. Surma:You know, in JavaScript, when you step through, you can hover over variables and see what
  858. Surma:the values are.
  859. Surma:I don't think that works because it is kind of language specific.
  860. Surma:Because like basically DWARF tells, yeah, this is a variable.
  861. Surma:The original name is whatever.
  862. Surma:And its type is Rust colon colon string.
  863. Surma:So unless you know what the exact data structure of Rust colon colon string is, you won't know
  864. Surma:how to visualize that.
  865. Jake:Of course.
  866. Surma:But just being able to step through and figuring out where an exception is being thrown or
  867. Surma:something like that is already incredibly useful.
  868. Surma:So yeah.
  869. Surma:And I do not know why they kept it an extension and not just looped it into DevTools itself.
  870. Surma:But that just deserves a massive shout out because I didn't realize it would actually
  871. Surma:make my exceptions nicer.
  872. Surma:And that's really valuable.
  873. Jake:At the same time, I just did a sneaky bit of searching,
  874. Jake:and, yeah, source maps is now in TC39.
  875. Surma:Oh, cool.
  876. Surma:We should link to that.
  877. Jake:Maybe has been for a couple of years. Yes, we will.
  878. Jake:It's stage zero, but they're definitely working on it.
  879. Jake:Like, the last commit was three weeks ago,
  880. Jake:so it's sort of being actively worked on, which is really nice.
  881. Surma:That's more activities than some other stage zero proposals.
  882. Surma:Yeah.
  883. Surma:So this was my overview of what I wanted to call debugging on the web and how does DevTools
  884. Jake:Absolutely.
  885. Surma:know what code to show you and also why I guess we both have beef with source maps.
  886. Surma:We both have beef with source maps and I really hope that in the standardization that
  887. Jake:Oh
  888. Surma:they learn from where source maps are falling short and then either make something that
  889. Surma:is long lasting and doesn't have these problems, or maybe they should use DWARF.
  890. Surma:I don't know.
  891. Surma:I mean, source maps are already big.
  892. Surma:DWARF is also big.
  893. Surma:I don't think you win or lose a lot either way.
  894. Surma:And using existing format is usually much more beneficial than inventing something new.
  895. Jake:Oh, absolutely.
  896. Jake:It's had more hours of work put into it.
  897. Surma:Yeah.
  898. Surma:DWARF is extremely battle tested.
  899. Surma:So I feel like it.
  900. Surma:I don't know.
  901. Surma:I'll guess we'll see what they do.
  902. Jake:Excellent. Well, we'll link to your Wasmphobia thing,
  903. Jake:which is, yeah, it's a lovely little project.
  904. Jake:It is impressive, because my assumption was always, like,
  905. Jake:well, once it's a binary, then your ability to really understand
  906. Jake:where things in there came from,
  907. Jake:I just assumed that that was impossible now.
  908. Jake:But it's, yeah, and especially on the web,
  909. Jake:where we do care about those bytes,
  910. Surma:Yeah.
  911. Jake:a lot of the image encoders and decoders we were dealing with
  912. Jake:would have, like, except PNG as an input,
  913. Surma:Yeah.
  914. Jake:but we knew, like, we were just able to pass in raw bytes of an image
  915. Jake:and sort of determining, like, have I definitely created a build
  916. Jake:that doesn't have the PNG encoder and decoder in there?
  917. Surma:Yeah.
  918. Jake:And I was just like, I'm going to assume,
  919. Jake:because I added these flags and the bundle is now smaller,
  920. Jake:that I have succeeded.
  921. Jake:But I don't know, it feels like it should have got more smaller
  922. Jake:than it did, but I don't know.
  923. Surma:You know what?
  924. Surma:And now we could actually probably figure this out a bit more because we could because
  925. Jake:Yeah.
  926. Surma:Emscripten emits DWARF as well.
  927. Jake:Oh, nice.
  928. Surma:If you add the dash G full flag, I think I have it in the readme of isomphobia and we
  929. Jake:Ooh.
  930. Surma:could drop it in and we could actually take a look of like, is it a PNG dot header file
  931. Surma:or PNG dot C files that they still include it somewhere.
  932. Jake:Yeah.
  933. Surma:I actually would be more confident that I can inspect what's in there and we could figure
  934. Surma:out whether there's more that we need to do.
  935. Surma:And I guess it's again, it's the same thing in native land.
  936. Surma:This kind of tree shaking is just not a requirement, not something that is being thought of when
  937. Jake:Yeah.
  938. Surma:you write code, because even in Android land, right, like whether your Android app is eight
  939. Surma:megabytes or 12 megabytes, nobody cares.
  940. Surma:I mean, there is a limit at some point where you go like, wait, why is this app to take
  941. Surma:notes?
  942. Surma:120 megs.
  943. Surma:But if you below a certain threshold, nobody cares really about squeezing out the last
  944. Surma:couple of bytes.
  945. Surma:But on the web, it's very, very different.
  946. Jake:And I think we were both impressed, like,
  947. Jake:somewhere where it did happen differently,
  948. Jake:we were both impressed by Animal Well,
  949. Surma:Yes.
  950. Jake:the kind of Metroidvania game, which I haven't played yet,
  951. Jake:but I've watched videos of it and it looks really good
  952. Jake:and I really want to play it.
  953. Jake:But this is, like, when you get a game on the PlayStation,
  954. Jake:it's, yeah, it's 100 gigabytes or whatever,
  955. Jake:and if you dare install it from a CD,
  956. Surma:Yes, easily.
  957. Jake:because you think that, a CD, a Blu-ray, whatever,
  958. Surma:Yes, easily.
  959. Jake:because you think that's going to be quicker,
  960. Jake:it's still going to be hit with that 50 gigabyte update
  961. Jake:afterwards, you know?
  962. Jake:But Animal Well is 34 megabytes,
  963. Jake:which is, like, for a game these days,
  964. Jake:that's smaller than Minesweeper is on Windows these days, so...
  965. Surma:That's more than some web apps that are out there.
  966. Jake:Oh, many, many.
  967. Jake:Funny detail of it is, like, on the PS5, it's not 34 megabytes,
  968. Jake:it's 101 megabytes.
  969. Jake:But the theory is that, you know,
  970. Jake:when you highlight the game's icon on the PlayStation,
  971. Jake:you get, like, a background image shows up,
  972. Surma:Yes.
  973. Surma:Yes.
  974. Surma:Yes.
  975. Jake:which would be, like, I don't know, 4K or 8K or whatever,
  976. Surma:Yes.
  977. Surma:Yes.
  978. Surma:Yes.
  979. Surma:Yes.
  980. Jake:and the theory is that's most of the size increase.
  981. Surma:Yes.
  982. Surma:Yes.
  983. Jake:So that background image that appears on the PlayStation menu
  984. Surma:Yes.
  985. Surma:Yes.
  986. Surma:Yes.
  987. Surma:Yes.
  988. Jake:is actually bigger, bigger than the game itself, yeah.
  989. Surma:Yes.
  990. Surma:Yes.
  991. Jake:Which is, you know, that is the right way around.
  992. Jake:Like, there are so many websites where a video of the website
  993. Jake:ends up being smaller than the code to make it run.
  994. Surma:It always makes me think of the demo scene in the mid-2000s.
  995. Jake:Oh, yeah.
  996. Surma:There was this game called KKrieger, I think, which was 100K and had, for the time, stunning
  997. Surma:visual 3D graphic.
  998. Surma:And it was basically like a Doom first-person shooter.
  999. Surma:And, you know, obviously the enemies were quite dumb, but it was incredible, it was
  1000. Surma:super impressive.
  1001. Surma:Actually, I'm going to link to that as well.
  1002. Jake:And things like Frontier on the Amiga,
  1003. Jake:where it's seemingly the universe fit on a floppy disk somehow.
  1004. Jake:There was one...
  1005. Jake:I mean, we talked about it on the show before,
  1006. Jake:but there was a few games on the Commodore 64
  1007. Jake:where they would give you a game to play
  1008. Jake:while the game was being loaded,
  1009. Jake:which I always thought was, like,
  1010. Jake:well, that's a lovely little bit of progressive loading,
  1011. Jake:but there was another reason that they did that,
  1012. Jake:and the reason was the hardware code on the Commodore 64
  1013. Jake:for loading data from the tape or from floppy disks
  1014. Jake:into memory was bad.
  1015. Jake:Like, they hit a bug close to shipping the machine out,
  1016. Jake:and it was like, this bug's so bad,
  1017. Surma:Wow.
  1018. Jake:we've got to do the whole loading thing in it
  1019. Jake:in a different and terrible way just for it to not be buggy.
  1020. Jake:And then so after it shipped, people found,
  1021. Jake:it's like, well, we could load in a loader that can do it better,
  1022. Jake:because better ways of doing it on software were found later on.
  1023. Jake:So while you were actually playing this little game,
  1024. Jake:it was actually also loading its own loader,
  1025. Jake:which could load the rest of the game into memory faster.
  1026. Jake:Yeah, which is brilliant, isn't it? I love that.
  1027. Surma:Some of these tricks at the time, I remember, I don't know which game it was, I'll look
  1028. Surma:it up, but there was a game which, again, shortly before shipping, was crashing when
  1029. Surma:you finished the game, like when you beat it.
  1030. Surma:And they couldn't figure out why it crashed.
  1031. Surma:So basically when you finish the game, it's just like, segfault, blah, something.
  1032. Jake:Oh, yes.
  1033. Surma:And so what they did, because at the time this was possible, when they loaded the game,
  1034. Surma:they would find that string in memory, like the string that contains the error message
  1035. Surma:for a segfault, and overwrite it with a message, thank you for playing whatever the game was
  1036. Jake:It was Wing Commander.
  1037. Jake:I'm sure we've talked about that on the show before.
  1038. Surma:called.
  1039. Jake:Yeah, thank you for playing. Wing Commander was the game.
  1040. Surma:And then they just ship it.
  1041. Jake:Yeah, yeah, I love that game as well.
  1042. Surma:Really?
  1043. Surma:Brilliant.
  1044. Jake:Nah, well, whatever.
  1045. Surma:Well, then we can cut this.
  1046. Jake:No, look, look, people have said,
  1047. Jake:like, one of the things they like about the show
  1048. Jake:is it's kind of like, it's almost like, you know,
  1049. Jake:they're just sort of down the pub with us.
  1050. Jake:And if there's one thing I do down the pub,
  1051. Jake:it's tell the same story over and over again.
  1052. Jake:Absolutely.
  1053. Jake:Oh, absolutely.
  1054. Jake:Well, with that, then, we shall say,
  1055. Jake:happy next time!
  1056. Surma:What a brilliant point to end this episode, I think.
  1057. Surma:We've told the same stories, so before we lose more credibility, we can call it.
  1058. Surma:Happy next time!