Holy cow, I'm a very casual gamer, I was excited about the game but when it came out I decided I don't want to wait that long and I'll wait until they sort it out. 2 years later it still sucked. So I abandoned it. But.. this... ?! This is unbelievable. I'm certain that many people left this game because of the waiting time. Then there are man-years wasted (in a way different than desired).
Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020.
Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.
I don’t think the lesson here is “be careful when parsing json” so much as it’s “stop writing quadratic code.” The json quadratic algorithm was subtle. I think most people’s mental model of sscanf is that it would be linear in the number of bytes it scans, not that it would be linear in the length of the input. With smaller test data this may have been harder to catch. The linear search was also an example of bad quadratic code that works fine for small inputs.
Some useful lessons might be:
- try to make test more like prod.
- actually measure performance and try to improve it
- it’s very easy to write accidentally quadratic code and the canonical example is this sort of triangular computation where you do some linear amount of work processing all the finished or remaining items on each item you process.
As I read the article, my guess was that it was some terrible synchronisation bug (eg download a bit of data -> hand off to two sub tasks in parallel -> each tries to take out the same lock on something (eg some shared data or worse, a hash bucket but your hash function is really bad so collisions are frequent) -> one process takes a while doing something, the other doesn’t take long but more data can’t be downloaded until it’s done -> the slow process consistently wins the race on some machines -> downloads get blocked and only 1 cpu is being used)
> actually measure performance and try to improve it
This really rings truest to me: I find it hard to believe nobody ever plays their own game but I’d easily believe that the internal culture doesn’t encourage anyone to do something about it. It’s pretty easy to imagine a hostile dev-QA relationship or management keeping everyone busy enough that it’s been in the backlog since it’s not causing crashes. After all, if you cut “overhead” enough you might turn a $1B game into a $1.5B one, right?
I could easily imagine variations of that where this is in maintenance mode with a couple of junior programmers because the senior ones either burnt out or moved on to another project. I’ve definitely gotten the impression that most games studios have roughly the same attitude towards their employees as a strip-miner has towards an Appalachian hilltop.
True, but they could still be understaffing and have their senior people working on the next big version rather than maintenance. It’s definitely penny wise, pound foolish no matter the exact details.
- do not implement your own JSON parser (I mean, really?).
- if you do write a parser, do not use scanf (which is complex and subtle) for parsing, write a plain loop that dispatches on characters in a switch. But really, don't.
I think sscanf is subtle because when you think about what it does (for a given format string), it’s reasonably straightforward. The code in question did sscanf("%d", ...), which you read as “parse the digits at the start of the string into a number,” which is obviously linear. The subtlety is that sscanf doesn’t do what you expect. I think that “don’t use library functions that don’t do what you expect” is impossible advice.
I don’t use my own json parser but I nearly do. If this were some custom format rather than json and the parser still used sscanf, the bug would still happen. So I think json is somewhat orthogonal to the matter.
> The code in question did sscanf("%d", ...), which you read as “parse the digits at the start of the string into a number,” which is obviously linear.
I think part of the problem is that scanf has a very broad API and many features via its format string argument. I assume that's where the slowdown comes from here - scanf needs to implement a ton of features, some of which need the input length, and the implementor expected it to be run on short strings.
> The subtlety is that sscanf doesn’t do what you expect. I think that “don’t use library functions that don’t do what you expect” is impossible advice.
I don't know, at face value it seems reasonable to expect programmers to carefully check whether the library function they use does what they want it to do? How would you otherwise ever be sure what your program does?
There might be an issue that scanf doesn't document it's performance well. But using a more appropriate and tighter function (atoi?) would have avoided the issue as well.
Or, you know, don't implement your own parser. JSON is deceptively simple, but there's still enough subtlety to screw things up, qed.
But sscanf does do what they want it to do by parsing numbers. The problem is that it also calls strlen. I’m still not convinced that it’s realistically possible to have people very carefully understand the performance characteristics of every function they use.
Every programmer I know thinks about performance of functions either by thinking about what the function is doing and guessing linear/constant, or by knowing what the data structure is and guessing (eg if you know you’re doing some insert operation on a binary tree, guess that it’s logarithmic), or by knowing that the performance is subtle (eg “you would guess that this is log but it needs to update some data on every node so it’s linear”). When you write your own library you can hopefully avoid having functions with subtle performance and make sure things are documented well (but then you also don’t think they should be writing their own library). When you use the C stdlib you’re a bit stuck. Maybe most of the functions there should just be banned from the codebase, but I would guess that would be hard.
> I assume that's where the slowdown comes from here - scanf needs to implement a ton of features, some of which need the input length, and the implementor expected it to be run on short strings.
I didn't get that impression. It sounded like the slowdown comes from the fact that someone expected sscanf to terminate when all directives were successfully matched, whereas it actually terminates when either (1) the input is exhausted; or (2) a directive fails. There is no expectation that you run sscanf on short strings; it works just as well on long ones. The expectation is that you're intentionally trying to read all of the input you have. (This expectation makes a little more sense for scanf than it does for sscanf.)
The scanf man page isn't very clear, but it looks to me like replacing `sscanf("%d", ...)` with `sscanf("%d\0", ...)` would solve the problem. "%d" will parse an integer and then dutifully read and discard the rest of the input. "%d\0" will parse an integer and immediately fail to match '\0', forcing a termination.
EDIT: on my xubuntu install, scanf("%d") does not clear STDIN when it's called, which conflicts with my interpretation here.
No it would not. Think about what the function would see as its format string in both cases.
The root cause here isn't formatting or scanned items. It is C library implementations that implement the "s" versions of these functions by turning the input string into a nonce FILE object on every call, which requires an initial call to strlen() to set up the end of read buffer point. (C libraries do not have to work this way. Neither P.J. Plauger's Standard C library nor mine implement sscanf() this way. I haven't checked Borland's or Watcom's.)
Yes, it looks that way. On the unix/linux side of things, glibc also implements scanf() by converting to a FILE* object, as does the OpenBSD implementation.
It looks like this approach is taken by the majority of sscanf() implementations!
I honestly would not personally have expected sscanf() to implicitly call strlen() on every call.
> If this were some custom format rather than json and the parser still used sscanf, the bug would still happen. So I think json is somewhat orthogonal to the matter.
What's the point of using standard formats if you're not taking advantage of off-the-shelf software for handling it?
This is probably good advice but not even relevant. It's down one level from the real problem: when your game spends 6 minutes on a loading screen, *profile* the process first. You can't optimize what you haven't measured. Now, if you've identified that JSON parsing is slow, you can start worrying about how to fix that (which, obviously, should be "find and use a performant and well-tested library".)
It could be, and the article acknowledges that possibility. For example, a cursory check of the musl sscanf [0] suggests that it does not (though I may have missed something). However, whichever implementation Rockstar used apparently does.
A lot of libc implementations seem to implement sscanf() this way: As well as the windows libc ones mentioned above, I checked the OpenBSD & glibc implemtations & they worked the same way.
Part of this is that game companies are notorious for re-implementing standard libraries for "performance". I suspect both shitty implementations of sscanf and the not-a-hashmap stem from this.
> actually measure performance and try to improve it
This reminds me that I used to do that all the time when programming with Matlab. I have stopped investigating performance bottlenecks after switching to Python. It is as if I traded performance profiling with unit testing in my switch from Matlab to Python.
I wonder if there are performance profilers which I could easily plug into PyCharm to do what I used to do with Matlab's default IDE (with a built-in profiler) and catch up with good programming practices. Or maybe PyCharm does that already and I was not curious enough to investigate.
The JSON parsing is forgivable (I actually didn't know that scanf computed the length of the string for every call) but the deduplication code is a lot less so, especially in C++ where maps are available in the STL.
It also comforts me into my decision of never using scanf, instead preferring manual parsing with strtok_r and strtol and friends. It's just not robust and flexible enough.
Many developers I have spoken to out there in the wild in my role as a consultant have wildly distorted mental models of performance, often many orders of magnitude incorrect.
They hear somewhere that "JSON is slow", which it is, but you and I know that it's not this slow. But "slow" can encompass something like 10 orders of magnitude, depending on context. Is it slow relative to a non-validating linear binary format? Yes. Is it minutes slow for a trivial amount of data? No. But in their mind... it is, and there's "nothing" that can be done about it.
Speaking of which: A HTTPS REST API call using JSON encoding between two PaaS web servers in Azure is about 3-10ms. A local function call is 3-10ns. In other words, a lightweight REST call is one million times slower than a local function call, yet many people assume that a distributed mesh microservices architecture has only "small overheads"! Nothing could be further from the truth.
Similarly, a disk read on a mechanical drive is hundreds of thousands of times slower than local memory, which is a thousand times slower than L1 cache.
With ratios like that being involved on a regular basis, it's no wonder that programmers make mistakes like this...
Tbe funny thing is, as a long time SDET, I had to give up trying to get people to write or architect in a more "local first" manner.
Everyone thinks the network is free... Until it isn't. Every bit move in a computer has a time cost, and yes, it's small... But... When you have processors as fast as what exist today, it seems a sin that we delegate so much functionality out to some other machine across a network boundary when the same work could be done locally. The reason why though?
Monetizability and trust. All trivial computation must be done on my services so they can be metered and charged for.
We're hamstringing the programs we run for the sole reason that we don't want to make tools. We want to make invoices.
> But how could R* not do this? GTAV is so full of great engineering
I assume there were different people working on the core game engine and mechanics VS loading. It could as well be some modular system, where someone just implemented the task "load items during online mode loading screen screen".
Me and my twice-a-week gaming group enjoyed GTA V but abandoned it years ago simply because of the load times. We have 2x short slots (90-120 minutes) each week to play and don't want to waste them in loading screens.
We all would have picked this game back up in a second if the load times were reduced. Although I must say even with the same results as this author, 2 minutes is still too long. But I'll bet that, given the source code, there are other opportunities to improve.
I wonder if a paid subscription would have fixed this? If you left a paid MMO, they'd probably ask you to fill out an exit survey, and you could say "I'm canceling because load times are terrible", which would (hopefully) raise the priority of reducing load times. But since GTA online is "free", there's not a single exit point where they can ask "why did you stop playing".
Gta has made billions off of its shark card micro transaction system. So the incentives are probably pretty similar for player retention. Granted, the players leaving over load times are probably not the players who are invested enough to spend thousands in micro transactions.
That's my point though, you don't get to survey people not buying the microtransaction because they quit due to terrible load times, whereas you could survey people who cancel a subscription. I guess they could still gather data by reading reviews, looking through forums / Reddit / whatever, and tallying up complaints, though.
This is why I come to HN, I was going to skip this because I thought it was about video games, but really glad to have read it, and loved every line of the article.
So much to get from this.
Even if you don't have the source, you can make a change if you are annoyed enough.
If you don't like something, and the source code is out there, really go contribute.
Performance matters, know how to profile and if using an external dependency, then figure out their implementation details.
Algorithms & Data structures matter, often I see devs talking about how it doesn't matter much but the difference between using a hashmap vs array is evident.
Attentive code reviews matter, chances are they gave this to a junior dev/intern, and it worked with a small dataset and no one noticed.
I think this is a perfect example of “algorithms and data structures emphasis is overblown.” Real world performance problems don’t look like LeetCode Hard, they look like doing obviously stupid, wasteful work in tight loops.
... that's the exact opposite of what I took from this.
The obviously stupid, wasteful work is at heart an algorithmic problem. And it cropped up even in the simplest of data structures. A constant amount of wasteful work often isn't a problem even in tight loops. A linear amount of wasted work, per loop, absolutely is.
It's not something that requires deep algorithms/data structures knowledge, is the point. Knowing how to invert a binary tree won't move the needle on whether you can spot this kind of problem. Knowing how to operate a profiler is a lot more useful.
True that it's rare that you need to pull out obscure algorithms or data structures, but in many projects you'll be _constantly_ constructing, composing, and walking data structures, and it only takes one or two places that are accidentally quadratic to make something that should take milliseconds take minutes.
The mindset of constantly considering the big-O category of the code you're writing and reviewing pays off big. And neglecting it costs big as well.
Except that you need to test your software and if you see performance problems, profile them to identify the cause. It's not like you have one single chance to get everything right.
The later in development a problem is caught, the more expensive it is. The farther it gets along the pipeline of concept -> prototype -> testing -> commit -> production, the longer it's going to take to notice, repro, identify the responsible code, and fix.
It's true that you don't just have one shot to get it right, but you can't afford to be littering the codebase with accidentally quadratic algorithms.
I fairly regularly encounter code that performed all right when it was written, then something went from X0 cases to X000 cases and now this bit of N^2 code is taking minutes when it should take milliseconds.
And trying to optimize them gets you stink eye at code review time. Someone quotes Knuth, they replace your fast 200 lines with slow-as-molasses 10 lines and head to the bar.
And here what matters is not your programming skills, it’s your profiling skills. Every dev writes code that’s not the most optimized piece from the start, hell we even say “don’t optimise prematurely”. But good devs know how to profile and flamegraph their app, not leetcode their app.
actually, "don't optimize prematurely" is a poor advice.
just recently I was doing a good review that had the same issue where they were counting the size of an array in a loop, when stuff was being added to the array in the loop too. obvious solution was to track the length and
arr = []
while ...:
if something:
arr.append(foo)
...
if count(arr) == x:
stuff
...
changed to
arr=[]
arr_size = 0
while ...:
if something:
arr.append(foo)
arr_size++;
...
if arr_size == x:
stuff
...
This is clearly optimization, but it's not premature. The original might just pass code review, but when it wrecks havoc, the amount of time it will cost will not be worth it, jira tickets, figuring out why the damn thing is slower, then having to recreate it in dev, fixing it, reopening another pull request, review, deploy, etc. Sometimes "optimizing prematurely" is the right thing to do if it doesn't cost much time to do or overly completely the initial solution. Of course, this depends on the language, some languages will track the length of the array so checking the size is o(1), but not all languages do, so checking the length can be expensive, knowing the implementation detail matters.
I'm not sure I would prefer the second version in a code review. I find the first version is conceptually nicer because it's easy to see that you will always get the correct count. In the second version you have to enforce that invariant yourself and future code changes could break it.
If this is premature optimization or not depends on the size of the array, number of loop iterations and how often that procedure is called. If that's an optimization you decide to do, I think it would be nice to extract this into an "ArrayWithLength" data structure that encapsulates the invariant.
> In the second version you have to enforce that invariant yourself and future code changes could break it.
Yes, that's a real issue. But we've been given two options:
- Does the correct thing, and will continue to do the correct thing regardless of future changes to the code. Will break if the use case changes, even if the code never does.
- Does the correct thing, but will probably break if changes are made to the code. Will work on any input.
It actually seems a lot more likely to me that the input given to the code might change than that the code itself might change. (That's particularly the case for the original post, where the code serves to read a configuration file, but it's true in general.)
Yes, I absolutely see the reasoning and I think if one does go the route of encapsulating the more efficient array logic one can have the best of both options.
> if one does go the route of encapsulating the more efficient array logic one can have the best of both options.
Do you see a way to do this that doesn't involve rolling your own array-like or list-like data type and replacing all uses of ordinary types with the new one? (This is actually already the implementation of standard Python types, but if you're encountering the problem, it isn't the implementation of your types.)
I guess it depends on the language and library you are using but I have the feeling that in most cases one would probably need to replace the usage of the old data type with the new one.
With these things, I have always had the hope that an optimizing compiler would catch this. I think it is an allowed optimization if the count function is considered `const` in C or C++ at least.
Exactly - though to add a little nuance to your post, it’s about having a million loops in a 10M line code base and exactly one of them is operating maximally slowly. So preventing the loop from entering the code base is tough - finding it is key.
I always tell a story about an application we ran, it generated its own interface based on whatever was in inventory. Someone did something really stupid and duplicated each inventory item for each main unit we sold...so you had recursive mess. Assigning 100,000 items when previously it was 100-ish
Anyway, everyone just rolled their eyes and blamed the fact that the app was written in Java.
It ended up generating an XML file during that minute long startup....so we just saved the file to the network and loaded it on startup. If inventory changed, we’d re-generate the file once and be done with it.
There’s also a confound between a language and its communities. I’ve seen so many cases where a “slow” language like Python or (older) Perl smoked Java or C++ because the latter developers were trying to follow cultural norms which said that Real Decelopers™ don’t write simple code and they had huge memory churn with dense object hierarchies and indirection so performance ended up being limited by O(n) XML property lookups for a config setting which nobody had ever changed whereas the “slow” language developer had just implemented a simple algorithm directly and most of the runtime was in highly-optimized stdlib native code, a fast regex instead of a naive textbook parser which devolved into an object churn benchmark, etc.
Languages like Java get a lot of bad reputation for that because of popularity: not just that many people are hired into broken-by-design environments (or ones where they’re using some framework from a big consultancy or a vendor who makes most of their revenue from consulting services) but also because many people learn the language as their first language and often are deeply influenced by framework code without realizing the difference between widely used long-term reusable code and what most projects actually need and are staffed for. It’s easy to see the style of the Java standard frameworks or one of the major Apache projects and think that everyone is supposed to write code like that, forgetting that they have to support a greater number of far more diverse projects over a longer timeframe than your in-house business app nobody else works on. Broader experience helps moderate this but many places choose poor metrics and neglect career development.
Java is RAM guzzler with some a small inability to optimize with value types. In its class (managed programming language without value types) it is pretty much as fast as it gets.
The two performance flaws that exist are:
1. Old Java frameworks were not written with performance in mind
2. Your entire app is written in Java so you don't benefit from C++ libraries
Getting banned for modifying the game process seems very commonplace and likely? It'll be a part of any anti-cheat system, it's basically table stakes.
Honestly, while this horrible code is mildly offensive to me, I'm pretty impressed by this person's persistence. It's one thing to identify a bug in a compiled program, but it's another to patch it without fully understanding what's going on. Caching strlen was a particularly clever trick that sidestepped a bunch of more complicated solutions.
I played through GTA V, enjoyed it, and tried out the online mode afterward.
I've logged in exactly twice. Load times like that may be worth it to a hardcore gamer, but I have no patience for it. There's no shortage of entertainment available to someone with a PC, a reasonable internet connection, and a modicum of disposable income. Waste my time and I'll go elsewhere for my entertainment.
Wow, many people argue how optimized GTA was and then this. I wonder how much money they lost because of this. I often stopped playing because it just took too long to load.
GTA, at least the core gameplay and the single player mode, is quite well optimised. The game ran well even on the cheaper side of gaming PC hardware.
This... this is GTA online. It's a cash cow designed to suck cash out of your pocket. Ads for things you can spend your money on are shown while "connecting", so if this delay wasn't introduced intentionally, it sure isn't a high priority fix. The code isn't part of the optimised, streamlined, interactive part of the game, it's part of the menu and loader system.
Most of these online games/services have so-called "whales" that contribute most if not all of the income the platform makes. If these whales are willing to spend the wads of cash they throw at the platform, they won't even care for another five minutes of ads. The amounts of cash some of these people spend is obscene; the millions Take Two profit from GTA every year are generally generated by only a tiny (usually a single number percentage) of the total player base.
In the end, I doubt they've lost much money on this. They might've even made some from the extra ads.
> GTA, at least the core gameplay and the single player mode, is quite well optimised. The game ran well even on the cheaper side of gaming PC hardware.
It's easy to forget that GTA5/GTA:O was originally a 360/PS3 game, getting a game of that scope running at all on a system with just 256MB of RAM and VRAM was an incredible achievement.
The A-Team developers who made that happen were probably moved over to Red Dead Redemption 2 though, with GTA5s long tail being handled by the B-Team.
GTA V (the single player game) is quite well optimized and needs a frame rate limiter on most newer systems because it will run at over ~180 fps, at which point the engine starts to barf all over itself.
GTA Online is a huge, enormously buggy and slow mess that will generally struggle to run at 80 fps on a top-of-the-line 2020 system (think 10900K at over 5 GHz with a 3090) and will almost never cross the 120 fps threshold no matter how fast your system is and how low the settings are.
It's a coding strategy intended to optimize around console refresh targets first and then "do whatever" for PC build.
Generally, a AAA console game will target 30hz or 60hz. Therefore the timing loop is built to serve updates at a steady pace of 30 or 60, with limiting if it goes faster. Many game engines also interpolate animations separately from the rest of the gameplay, allowing them to float at different refresh rates. Many game engines further will also decouple AI routine tick rates from physics to spread out the load. Many game engines now also interleave updates and kick off the next frame before the first is complete, using clever buffering and multithreaded job code. All told, timing in games is one of those hazard zones where the dependencies are both numerous and invisible.
When you bring this piece of intricate engineering over to PC and crank up the numbers, you hit edge cases. Things break. It's usually possible to rework the design to get better framerate independence, but doing do would be invasive - you'd be changing assumptions that are now baked into the content. It isn't just fixing one routine.
Because if you run the simulation at a different frame-rate from the rendering it is a huge amount more work. Suddenly you have to start marshaling data through intermediate data structures and apply interpolation to everything.
If you then try to run the simulation in parallel with the rendering (rather than between some frames) it is even more work, since inter-thread communication is hard.
This stuff might seem easy for very good programmers, however on a game you want to hire a wide range of programmer skill and "game-play programmers" tend to be weaker on the pure programming front (their skills lie elsewhere)
Because unfortunately, in order to draw something on the screen, you need to determine what you draw first. So, before each screen refresh, a thing called "game simulation tick" happens.
Yep, sometimes I feel like having a drive around, and then I remember how long it takes to load and play something else instead.
If you end up in a lobby with undesirables and are forced to switch, you've got another long wait.
It always fascinates me how sometimes people defend things just because they're a fan, even if the particular aspect they're defending doesn't make sense!
I've seen this happen with some other games which are not the best optimised for PCs, but the fans will still defend the developers, just because they like the brand
It's part of social validation. You inherently want other people to like the things you like, because it validates your choices. This in turn means you'll defend things you like.
Even "rebels", who supposedly relish having fringe tastes, want other rebels to approve of their tastes.
The more strongly you stake your identity as "fan of X", the more social disapproval of X hurts.
I disagree. There is no contradiction. JSON can be a different beast for C, C++, Java, backend coders. You can implement complex 3D graphics while struggling with JSON.
For example, my backend Java guys struggled heavily with JSON mappers. It took them forever to apply changes safely. My department consumes a lot of data from backend systems and we have to aggregate and transform them. Unfortunately the consumed structure changes often.
While a JSON mapper in our case in JAVA was sort of exceptionally hard to handle, a simple NodeJS layer in JavaScript did the job exceptionally easy and fast. So we used a small NodeJS layer to handle the mapping instead of doing this in Java.
Moral of the story: Sometimes there are better tools outside your view. And this seems to be many times the case for JSON. JSON means JavaScript Object Notation. It is still tough for OO languages to handle.
Not that C strings do not suck, but with pascal strings we could discuss in this thread how implicitly copying a slowly decreasing part of a 10mb string at every tokenizer iteration could miss a developer’s attention. It’s completely unrelated to C strings and is completely related to bytefucking by hand. Once you throw a task like “write a low-level edge-case riddled primitive value shuffling machine” into an usual middle level corporative production pipeline, you’re screwed by default.
I'm with you. I hate null terminated strings. In my first experiences with C, I specifically hated them because of strlen needing to scan them fully. When C++ introduced string_view, my hate grew when I realized that I could have zero-copy slices all the way up until I needed to interface with a C API. At that point you're forced to copy, even if your string_view came from something that was null terminated!
Could this be worked into a compiler/stdlib from the back-end? Could a compiler/stdlib quietly treat all strings as a struct of {length,string} and redefine strlen to just fetch the length field? Perhaps setting a hook to transparently update "length" when "string" is updated is not trivial.
Edit: hah, I'm decades late to the party, here we go:
Most modern libraries replace C strings with a structure containing a 32-bit or larger length value (far more than were ever considered for length-prefixed strings), and often add another pointer, a reference count, and even a NUL to speed up conversion back to a C string. Memory is far larger now, such that if the addition of 3 (or 16, or more) bytes to each string is a real problem the software will have to be dealing with so many small strings that some other storage method will save even more memory (for instance there may be so many duplicates that a hash table will use less memory). Examples include the C++ Standard Template Library std::string...
I don't think you could do it transparently, because it's expected to pass the tail of a character array by doing &s[100] or s + 100, etc. I don't think that would be easy to catch all of those and turn them into a string fragment reference.
From c++ class, std::string was easy enough to use everywhere, and just foo.c_str() when you needed to send it to a C library. But that may drags in a lot of assumptions about memory allocation and what not. Clearly, we don't want to allocate when taking 6 minutes to parse 10 megs of JSON! :)
Maybe even more surprising to me is that sscanf() relies on strlen().
I would have expected libc to take that use case in consideration and use a different algorithm when the string exceeds a certain size. Even if the GTA parser is not optimal, I would blame libc here. The worst part is that some machines may have an optimized libc and others don't, making the problem apparent only in some configuration.
I believe standard libraries should always have a reasonable worst case by default. It doesn't have to be perfectly optimized, but I think it is important to have the best reasonable complexity class, to avoid these kinds of problems. The best implementations usually have several algorithms for different cases. For example, a sort function may do insertion (n^2, good for small n) -> quicksort (avg. nlog(n), worst n^2, good overall) -> heapsort (guaranteed nlog(n), slower than quicksort except in edge cases). This way, you will never hit n^2 but not at the cost of slow algorithm for the most common cases.
The pseudo hash table is all GTA devs fault though.
I don't understand why it would need to use strlen anyway. Why wouldn't it treat the string like a stream when scanf is already coded to operate on an actual stream?
I gave up on playing GTA:O because everything took so long to load, having never spent a dime. I have to imagine there is so much lost revenue because of this bug; I hate to be harsh but it is truly an embarrassment that someone external to R* debugged and fixed this before they did (given they had 6 years!).
Same here, being slow as molasses really spoiled the game for me. I wonder if the same software quality caused other unnecessary in-game loading times. I felt that time spent playing GTA:O was about 60% waiting in lobby/playing, and the remaining 40% actual play time. Same with RDR2 by the way.
These problems come down to which programmers were allowed to remain on the team and how much they were allowed to change. So all similar performance problems will remain.
I can stomach a long load time (but that's not to excuse how godawful this load time is), just as long as it's worth it. I need some kind of resolution/ultimate goal, but in GTA:O there really isn't one; it's just grind-grind-grind so that you can get yet another car, or another house, or another thing that you already have plenty of. Lather, rinse, repeat in perpetuity.
I just felt like every time I logged in I went, "So what's the point here?".
Fixing bugs that reduce engagement is profitable. I am pretty confident this kind of bug has easily costed Rockstar in the order of 10-100 million dollars _per year_.
You could easily model how increased retention/engagement rates would translate to increased microtransaction revenue from drastically decreased load times. You could even bucket test it if you wanted to directly quantify the results, but I'm sure that the cost payback would be a matter of days.
It would make a great addition to the Accidentally Quadratic blog: https://accidentallyquadratic.tumblr.com/ (haven't been updated in ~2 years, but maybe the author is around).
GTA:O shows advertisements for in-game purchases on the loading screen. How many advertisements you see is a function of how long the loading screen takes.
Something tells me this "problem" was discovered long ago at Rockstar HQ, and quietly deemed not to be a problem worth solving.
I was going to say surely this has extremely diminishing or even negative return past 30-60 seconds, but then I remembered lots of people are willing to sit through 10 minutes of commercials to watch 20-30 minutes of TV. So I guess for the right type of customer it works?
I remember realizing I was 10-20 minutes into an infomercial instead of regular commercials late at night when I was young. I was probably distracted at the time but man did I feel stupid when I realized that it wasnt just a really long P90X commercial.
It depends on the channel, really. I remember once I was sitting at an eatery in the Philippines (basically a cafeteria with immediately-available food) and the TV just played commercials from the time I sat down until I left.
Found that they get more purchases BECAUSE of the long loading time, despite bouncing other players (the ad theory and happy coincidence for them to have the ad placement slots from shitty engineering)
The engagement team was told bullshit by the engineering team about how impossible it is to fix that issue
Yep this was me about 4 years ago. I distinctly remember sitting through the loading screen and being absolutely astonished at how long it took. I never fired it up again because I just couldn’t be bothered to wait that long.
I'm one that had pre-order bonus on GTA Online but gave up in part due to the loading times. If I have one hour to play I don't want to spend 10 minutes on loading screens.
It's funny to think that I would probably pay some amount to have GTA Online without these absurd loading times and without modders/hackers :D
But for the ones that don't, it equals a virtual commute, so it might be a good filter and the audience beyond that watershed is more serious about spending money there. ;)
Agree, this is up there in the top tier of amazing stories I've read here on HN. I admire T0ST's technical and writing skills; first rate combination. Massive kudos, would like to shout a cup of coffee.
(I also really like the design and presentation of the article; I'm running out of superlatives here.)
I loved the article. Are you planning to write any guides/tutorials about reverse engineering games? Seems like you have a lot of practical experience. I (and probably many other people) would be really excited if you started writing about how you do all these in detail with practical examples. I would even be glad to pay for such content.
This is really cool - how did you develop the background knowledge to solve this? I'm trying to learn more about low-level stuff and I would have no idea how to approach solving a problem like this
I was a reverse engineer for years and never was able to do anything like quoting docs. I'd be constantly googling or using reference material. The only real attribute I'd suggest is tenacity.
Both when looking at a particular problem, but also in sticking to RE in general for long enough to pick up the skills and tricks that make you quick. There are countless tricks you pick up that cleave off huge amounts of time that would otherwise be wasted.
And with most RE software(especially the mentioned inaccessible interactive disassembling tool) you'll get that by default, waiting for a 70mb binary to analyze isn't fun :)
Parsing 63k items in a 10 MB json string is pretty much a breeze on any modern system, including raspberry pi. I wouldn't even consider json as an anti-pattern with storing that much data if it's going over the wire (compressed with gzip).
Down a little in the article and you'll see one of the real issues:
> But before it’s stored? It checks the entire array, one by one, comparing the hash of the item to see if it’s in the list or not. With ~63k entries that’s (n^2+n)/2 = (63000^2+63000)/2 = 1984531500 checks if my math is right. Most of them useless.
The JSON patch took out more of the elapsed time. Granted, it was a terrible parser. But I still think JSON is a poor choice here. 63k x X checks for colons, balanced quotes/braces and so on just isn't needed.
Time with only duplication check patch: 4m 30s
Time with only JSON parser patch: 2m 50s
It’s an irrelevant one. The json parser from the python stdlib parses a 10Mb json patterned after the sample in a few dozen ms. And it’s hardly a fast parser.
I did a very very ugly quick hack in python. Took the example JSON, made the one list entry a string (lazy hack), repeated it 56,000 times. That resulted in a JSON doc that weighed in at 10M. My initial guess at 60,000 times was a pure fluke!
Dumped it in to a very simple sqlite db:
$ du -hs gta.db
5.2M gta.db
Even 10MB is peanuts for most of their target audience. Stick it in an sqlite db punted across and they'd cut out all of the parsing time too.
There's a "but" though: you might end up getting banned from GTA Online altogether if this DLL injection is detected by some anti-cheat routine. The developers should really fix it on their end.
It's highly unlikely you are going to get banned on GTA even with cheats. The anti-cheat is a joke. The game is filled to the brim with cheaters. If me and my friends play, we play with cheats just to protect ourselves from other cheaters.
Game cheat dev here: Just to provide some context, the GTA Online client is woefully horrible at doing client-side validation on the packets it receives from other peers. (there isn't an authoritative server)
This means that anyone in your session can send you a weirdly-formed packet to crash your game. Most cheats have protections against this by just doing Rockstar's job and adding better validation around packet interpretation routines.
Using "cheats just to protect [your]selves" actually makes a lot of sense.
I am absolutely shocked about this finding. The amount of money on microtransactions Rockstar lost because of this single issue must be gigantic.
The amount of people that got turned off by the loading times over the years is massive.
It's mind boggling.
Probably this was written under a very strict release deadline and it worked ok at the time (less items for microtransactions). The problem lies with the management that never picked up the issue once it became a big problem. Pretty sure that any developer in R* is capable of optimizing this parser.
It's the kind of thing that's very easy to accidentally write, it's not that shameful. What's shameful is not investigating the load times at all, since the problem is so easy to see when any measurement is done.
It's easy to write because it doesn't run noticably slowly on smaller data, and it's easy to accidently introduce quadratic behaviour in some systems. Obviously if someone tested this on 10MB of JSON and saw it took minutes and thought that was reasonable, that's a bit ridiculous, but I doubt at the time the code was written anyone expected it to be fed such a large JSON object.
I imagine there is a senior programmer working for another game company. They are currently kicking themselves about the poorly performing and rushed loading code they wrote while still working at R*. But there is nothing they can do about it now, since they have moved on.
As they say, a lot of classified stuff and closed-source code remains classified and closed not because it contains important secrets, but because those who hold the lock feel too ashamed and embarrassed to show the contents.
The bug is more devious than that. The code looks linear at a glance and the culprit is that sscanf is actually O(N) on the length of the string. How many people would expect that?
My personal pet peeve is Windows Update (and their products installation routine in general). I bet that it’s n^3 somewhere deep and they carefully curb than n for decades.
Good call. I'd love to read a post-mortem on why it was even possible for Windows XP's update check to be as slow as it was. I've definitely waited around 2 hours one time just for the check to complete after finishing an installation.
I loved so much reading it, I was thinking that if someone were to write fictional, sherlock holmes like fantasy story where our sherlock would take some (maybe fictional) widely used software at each episode, investigate it like this, and reveal some (fictional) grand bug in the end- I'd totally read it.
Yeah I know it sounds stupid, but I suspect real Sherlock Holmes was inspired by a true story like this one too, and at least some contemporary detectives started to enjoy reading them.
There’s no need for the examples to be fictional, there are more than enough real world cases to share. Sadly, many of my personal ones end in “I filed a bug with an annotated screenshot of decompiled code indicating where they should fix it but nothing happened”.
Reading things like these is bittersweet. One one hand, I am glad to see that the art of figuring out “why is this thing slow” is still alive and well, even in the face of pushback from multiple fronts. On the other hand, it’s clear that the bar is continually rising for people who are looking to do this as a result of that pushback. Software has always had a bottleneck of the portion of the code written by the person with the least knowledge or worst priorities, but the ability to actually work around this as an end user has gotten harder and harder.
The first hurdle is the technical skill required: there has always been closed source software, but these days the software is so much more complex, often even obfuscated, that the level of knowledge necessary to diagnose an issue and fix it has gone up significantly. It used to be that you could hold and entire program in your head and fix it by patching a couple bytes, but these days things have many, many libraries and you may have to do patching at much stranger boundaries (“function level but when the arguments are these values and the call stack is this”). And that’s not to say anything of increasing codesigning/DRM schemes that raise the bar for this kind of thing anyways.
The other thing I’ve been seeing is that the separation between the perceived skills of software authors and software users has increased, which I think has discouraged people from trying to make sense of the systems they use. Software authors are generally large, well funded teams, and together they can make “formidable” programs, which leads many to forget that code is written by individuals who write bugs like individuals. So even when you put in the work to find the bug there will be people who refuse to believe you know what you are doing on account of “how could you possibly know better than $GIANT_CORPORATION”.
If you’re looking for ways to improve this, as a user you should strive to understand how the things you use work and how you might respond to it not working–in my experience this is a perpetually undervalued skill. As a software author you should look to make your software introspectable, be that providing debug symbols or instructions on how users can diagnose issues. And from both sides, more debugging tools and stories like this one :)
Doesn't surprise me at all. It's an O(n^2) algorithm (strlen called in a loop) in a part of the code where N is likely much smaller in the test environment (in-app purchases).
Overwatch is another an incredibly popular game with obvious bugs (the matchmaking time) front and center. And gamers are quick to excuse it as some sort of incredibly sophisticated matchmaking - just like the gamers mentioned in OP.
It's easy to to say it's something about gamers / gaming / fandom - but I have a locked down laptop issued by my bigcorp which is unbelievably slow. I'd bet a dollar there's a bug in the enterprise management software that spins the CPU. A lot of software just sucks and people use it anyway.
I am not sure Overwatch's matchmaking time is a bug per se. The time estimates are bad for sure. But the matchmaker can really only be sure of one state -- if you queue for a match, a match will be made. The rest is predicting who will show up, along with some time-based scale for making a suboptimal match in the interest of time. Players absolutely hate these suboptimal matches, so the time threshold ends up being pretty high. The rest seems to just be luck; will the right combination of 11 other people be in the right place at the right time?
I think it could be improved, but it doesn't strike me as being buggy.
(Overwatch itself, lots of bugs. Tons of bugs. If they have any automated tests for game mechanics I would be pretty surprised.)
No, it doesn't add up. I can see dozens of groups filling up and queueing at my groups level as I wait for matches. Worse, many of my matches just aren't that evenly balanced. Even if you believe the game is dead now, things were just as bad (10+ minute queues for full 6 stacks) at the peak. They don't do tight geographic bindings - I live on the US west coast and regularly get Brits and Aussies in my games.
I guess what they are probably doing is batching groups of games and then matching for the entire batch, to ensure nobody gets a "bad" match. What they've missed is that well - 5% of matches are bad baseline because somebody is being a jerk or quits or has an internet disconnect or smurfs or whatever other reasons. They could have picked an algorithm that gave fast matches 99% of the time at the cost of having bad matches 1% of the time and nobody would have noticed, because their baseline bad match rate is so high. Optimization from too narrow a perspective.
Honestly, the OW matches I get aren't any more balanced that the COD matches I used to get, and I got those in a minute, not 15.
I think that player variance is the hardest thing for the matchmaker to account for, and a factor that makes a big difference in the quality of the match. Bronze duo'd with Top 500 is always a shit show, even if the other team has the same duo. It don't think it can be made to work.
(My experience shows this is the case; quickplay matches, with no grouping restrictions, are always more of a shitshow than ranked, which has some grouping restrictions.)
Similarly, an individual's performance variance makes a big difference that a mere arithmetic average can't account for. A 3900 player probably plays like a 4100 player when fresh and warmed up, but like a 3500 player when drunk, sleepy, and mad. The actual SR/MMR will converge to a time weighted average of those two states, but if you're in a 4100 game with that player when they're in their 3500 state, it's probably unwinnable. Maybe the matchmaker could attempt to predict this, but it would probably make people mad. Personally, I think that's the nature of a 12 person game composed of 12 randos. There is very little an algorithm can do to tune that (short of changing abilities/hitboxes/cooldowns of each player, which would make people mad).
Then there are people that abuse the matchmaker by exploiting uncertainty in their favor (creating a new account). Every time I have played in a 6 stack in ranked, we've always played against 6 brand new accounts that are clearly better than the matchmaker thinks they are, which sucks. (In quickplay, I generally have very good games in a 6 stack though.)
Overall, I hate to say this, but I think they're doing the best they can. I don't think there's enough data to make a good match 100% of the time, which is why people find 11 players and play "scrims" instead of ranked/quickplay. I don't know where you're able to see other groups and decide that they're a suitable match -- 90% of players have private profiles, and those that don't only play like 10 ranked games a season, so you can't really gauge how good or bad they are from public data.
(As for brits and aussies joining your west coast games... people VPN in to do that on purpose. Legend has it that west coast gamers are better than east coast gamers, so people all over the world VPN to the west coast and make it a self-fulfilling prophecy.)
I think you might be missing my point. Maybe an analogy - the classic problem in distributed database is CAP - consistency, availability, partition tolerance - pick two. Historically, a lot of people picked AP to build their highly available databases. In the context of Spanner, Google basically said this was the wrong tradeoff - availability is more highly impacted by external events like client to server networking incidents than by the design architecture - so you should pick CP instead.
I'm making the same point with regard to matchmaking. Overwatch has tried to optimize for good matches, ignoring all the issues you rightly describe above, and thinking about the tradeoff between time and good matches without regard to external events that impact matches like smurfs, uneven play, DCs, etc. They'd have been better off optimizing for fast matchmaking. It's bad engineering in plain sight, and gamers go out of their way to justify it.
That makes sense. The question is would more players quit the game because of long queue times, or because they got stomped when they were new. (That would be the result of just picking the first 12 people in the queue and throwing them in a match.)
I think there is some value in caring about that case. I started playing Overwatch with no FPS background (at 31!) and I never felt like I was in unwinnable games. All the players were as bad as me. (I still remember my first games when a D.va bomb would reliably get 6 kills.)
It's certainly not unique to game consumers. People in general just blame every fault on "it's physically impossible to solve". One big reason why corporations get away with creating non stop worse and worse products.
not surprising - the game industry is absolutely notorious for cutting corners. didn't know they cut corners this much though.
will r* fix it? maybe, especially since some person literally did half of the work for them. But given R* is a large company, this probably wont be fixed for a long time, and GTAO is probably being maintained by the lowest bid contractor group.
They probably have all of their full time assets working on the next iteration of GTA.
>But given R* is a large company, this probably wont be fixed for a long time, and GTAO is probably being maintained by the lowest bid contractor group.
They've also made just an absolute assload of money from GTA:O in spite of the godawful load times. Why bother spending the money to fix it when people are happy to deal with it and keep giving you their own cash?
Even after cutting loading by 70% it still take a minute? I haven't played any AAA titles for a long time. But even 30s is way too long. Especially I used to play with HDD. Considering modern SSD can be 30x Faster in Seq Read and Random Read up to 200x.
Is 1 min loading time even normal? Why did it take so long? I never played GTA Online so could someone explain?
Thank god. I always suspected that those loading times were cause by some retarded implementation detail. GTA5 is not that complex to justify that kind of loading times. Even the hardware has scaled massively since their launch and it doesn't even matter.
It so often is. This aspect of modern computing annoys me so much - modern computers, networks and cdns are so fast nowadays that most actions should be instant. My OS should boot basically instantly. Applications should launch instantly. Websites should load almost instantly. I’ll give a pass for 3D modelling, video editing, AAA video games and maybe optimized release builds of code. But everything else should happen faster than I can blink.
But most programs are somehow still really slow! And when you look into why, the reason is always something like this. The code was either written by juniors and never optimized because they don’t know how, or written by mids at the limit of their intelligence. And full of enough complex abstractions that nobody on the team can reason holistically about how the whole program works. Then things get slow at a macro level because fixing it feels hard.
Either way it’s all avoidable. The only thing that makes your old computer feel sluggish for everyday computing is that programmers got faster computers, and then got lazy, and then shipped you crappy software.
The most infuriating response to this is “the programs are doing so much these days!” Well, yes, a chat app might do emoji and stuff now. But it’s certainly not doing 1000x the number of things…
Yes! And a lot of those things are pointless, buggy busywork for the project managers involved. I’d rather well designed, minimal, fast, intuitive software with not many features over something like Xcode - packed full of buggy features that lag or crash the whole ide regularly. Polish is unsexy, but usually way more important than we give it credit for.
As they said at Zynga, move slow and fix your shit.
When it was released the game didn't have all the microtransactions so it probably took no time at all to process the JSON even with this issue.
Then over time they slowly add data to the JSON and then this O(n^2) stuff starts to creep up and up, but the farther away from release you are, the less likely that the kind of engineers to who do optimisation and paying any attention.
I had heard about this giant JSON from friends in the GTA V modding community. OP's idea of what it is used for is right. My guess is that this JSON was quite smaller when the game released and has been increasing in size as they add more and more items to sell in-game. Additionally, I speculate that most of the people with the knowledge to do this sort of profiling moved on to work on other Rockstar titles, and the "secondary team(s)" maintaining GTA Online throughout most of its lifespan either didn't notice the problem, since it's something that has become worse slowly over the years, or don't have enough bandwidth to focus on it and fix it.
It's also possible they are very aware of it and are saving up this improvement for the next iteration of GTA Online, running on a newer version of their game engine :)
Still much better than e.g. Ubisoft which repainted safety borders from red to blue and removed shadows in TM2 Canyon few years after release, also breaking a lot of user tracks. (If you’re not sure why, it was before a new iteration of the game)
More importantly how do you release a game that takes 6 minutes to load? This is why mobile gaming has the advantage. In those 6 minutes I could have played a round of a game that’s quite satisfying and put it down already. This just seems sloppy.
I've actually opened a game with long loading times and alt tabbed out, because I knew it would take a while. I booted up another game to play a little bit until the first game finishes loading. 3 hours later I was done playing and realized that I was supposed to play game #1.
It’s quite possible that there were way less items in the store on launch, or during testing. Then it could easily be overlooked. Of course no excuse to not fix it later.
When I used to play world of Warcraft, it never took more than 30 seconds or so to load. It got much faster over the years - when I was playing a few years ago it was more like 5 seconds from character selection to being in the world.
Nothing like the 6 minutes people are talking about for GTA. That’s ridiculous.
I played the game on PS3 and PC. The loading time at launch for PS3 (and later for PC, albeit on SSD) wasn't great, but it also wasn't nearly this terrible.
From a game programming perspective, I'm sure at launch there were very few extras to obtain, so this method ran fast and didn't raise any red flags.
But as time has worn on they've added a ton of extra items and it's become a real problem. What it does show is that probably most of their team are working on creating new things they can sell, vs testing and maintaining the codebase for the last N years.
It's never been "good", I played since 2013, Xbox 360 and later repurchased the game for PS4, the online load times were not just annoying they outright broke the game for me. To be having fun and then have a many minute delay while being pinned to the screen (because after loading from a mission you're never in a safe place).
Looking down on the ground through the clouds at the san Andreas's streets with that wispy air sound while waiting for those large thud noises which could come randomly will forever be etched into my memory as something which completely broke the fun I was having, especially when playing with friends and trying to do Heists later in the products life.
And because of this: getting people to play was really difficult, the combination of huge updates which took hours to download (PS4 has slow access to it's drive even if you upgrade to SSD) and the insanely long loading times once you have the patch culminated in many hours of lost gameplay.
I remember a quote from Steve Jobs which fits here: "Let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"[0]
I think that the author hasn't checked them all. Even this isn't checking them all.
The MUSL C library' sscanf() does not do this, but does call memchr() on limited substrings of the input string as it refills its input buffer, so it's not entirely free of this behaviour.
The sscanf() in Microsoft's C library does this because it all passes through a __stdio_common_vsscanf() function which uses length-counted rather than NUL-terminated strings internally.
A glib summary is that sscanf() in these implementations has to set up state on every call that fscanf() has the luxury of keeping around over multiple calls in the FILE structure. They're setting up special nonce FILE objects for each sscanf() call, and that involves finding out how long the input string is every time.
It is food for thought. How much could life be improved if these implementations exported the way to set up these nonce FILE structures from a string, and callers used fscanf() instead of sscanf()? How many applications are scanning long strings with lots of calls to sscanf()?
Addendum: There are C library implementations that definitely do not work this way. It is possible to implement a C library sscanf() that doesn't call strlen() first thing every time or memchr() over and over on the same block of memory.
Neither P.J. Plauger's nor my Standard C library (which I wrote in the 1990s and used for my 32-bit OS/2 programs) work this way. We both use simple callback functions that use "void*"s that are opaque to the common internals of *scanf() but that are cast to "FILE*" or "const char*" in the various callback functions.
OpenWatcom's C library does the same. Things don't get marshalled into nonce FILE objects on every call. Rather, the callback functions simply look at the next character to see whether it is NUL. They aren't even using memchr() calls to find a NUL in the first position of a string. (-:
> limited substrings of the input string as it refills its input buffer,
As far as I can tell, that copying helper function set to the read member of the FILE* never actually gets called in this path. I see no references to f->read() or anything that would call it. All of the access goes through shgetc and shunget, shlim, and shcnt, which directly reference the buf, with no copying. The called functions __intscan() and __floatscan() do the same. __toread() is called but just ensures it is readable, and possibly resets some pointers.
Even if it did, that pretty much does make it entirely free of this behavior, though not of added overhead. That operations structure stuffed into the file buffer doesn't scan the entire string, only copying an at most fixed amount more than asked for (stopping if the string terminates earlier than that). That leaves it linear, just with some unfortunate overhead.
I do find the exceedingly common choice of funneling all the scanf variants through fscanf to be weird. But I guess if they already have one structure for indirecting input, it's easy to overload that. (And somehow _not_ have a general "string as a FILE" facility, and building on top of that. (Posix 2008 does have fmemopen(), but it's unsuitable, as it is buffer with specified size (which would need to be calculated, as in the MS case), rather than not worried about until a NUL byte is reached.))
You've missed what happens in __uflow() when __toread() does not return EOF. (And yes, that does mean occasional memchr() of single characters and repeated memchr()s of the same memory block.)
> Posix 2008 does have fmemopen(), but it's unsuitable, as it is buffer with specified size (which would need to be calculated, as in the MS case), rather than not worried about until a NUL byte is reached.
With fmemopen(), you only need to calculate the length once at the start, right? And then you can use the stream instead.
It is! Not mentioned anywhere in the manpages either & there’s no a priori reason for sscanf() to need to call strlen() on the input string, so most programmers would never expect it to.
Pretty sure I would have made this error in the same situation, no question.
OpenBSD is also doing the same thing. It seems almost universal, unless the libc author has specifically gone out of their way to do something different!
This is why people should use commonly-available packages instead of rolling their own version of whatever dumb algorithm they think they can write. This happens all the time. Bugs have been fixed by others, but everyone is too smart to use someone else’s code.
Sometimes those commonly used packages end up being whatever dumb algorithm the author came up with, and nobody spends the time to verify if the package is worth it's popularity.
Doubt that. Popularity comes with heaps of PRs, some useful some less.
If anyone used a JSON parser that took 4 minutes to parse a file you bet the author would know by the time the 100th user comes around.
I had a tiny barely-used detection library that didn’t detect it correctly in a new Edge browser update. Someone complained about it within the first month. The library has 20 stars and Edge has 500 users.
Edit: Correction, it was 9 days after the browser release.
This is some first rate debugging, and great writing to boot. I hope Rockstar sees this, fixes the bug and then pays this fella something. Great post, thanks for sharing!
How many bets they'll leave it like this on the current version and fix it for the PS5 version, to show how dramatic of a change it is between consoles?
Red Dead Redemption 2 introduced a bug where the exe would reduce the volumne on start to 19%.
Its now at least 4 month old.
So whats the problem? When you tab+alt out of fullscreen, to change your soundvolumen everytime you start the game, you have to rechange the graphics configuration as well.
I solved it by .net code i found on github which would run fur 5 minutes and would put the volumne up as soon as it ifinds the rdd2 process...
Yeah no. While the C++ library ecosystem is painful to use, it still doesn't justify hand-rolling a JSON parser and there are certainly high quality hash-based container implementations available too, but even the standard one should beat the one used here.
But there is no "standard one"...reaffirming the point that you disagreed with. C++ standard library is blatantly missing key pieces given how long the language been around.
Are you arguing that there are no standard hash-based containers in the standard library of C++? Because there definitely are, even though nowadays there often are better alternatives outside the standard library.
It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.
I do not agree with the sibling comment saying that this problem only looks simple and that we are missing context.
This online gamemode alone made $1 billion in 2017 alone.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
Instead, someone with no source code managed to do this to an obfuscated executable loaded with anti-cheat measures.
The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.
(And yes, I might also still be salty because their parent company unjustly DMCA'd re3 (https://github.com/GTAmodding/re3), the reverse engineered version of GTA III and Vice City. A twenty-year-old game. Which wasn't even playable without purchasing the original game.)
> The fact that this problem is caused by Rockstar's excessive microtransaction policy (the 10MB of JSON causing this bottleneck are all available microtransaction items) is the cherry on top.
For what it's worth, 10MB of JSON is not much. Duplicating the example entry from the article 63000 times (replacing `key` by a uuid4 for unicity) yields 11.5MB JSON.
Deserialising that JSON then inserting each entry in a dict (indexed by key) takes 450ms in Python.
But as Bruce Dawson oft notes, quadratic behaviour is the sweet spot because it's "fast enough to go into production, and slow enough to fall over once it gets there". Here odds are there were only dozens or hundreds of items during dev so nobody noticed it would become slow as balls beyond a few thousand items.
Plus load times are usually the one thing you start ignoring early on, just start the session, go take a coffee or a piss, and by the time you're back it's loaded. Especially after QA has notified of slow load times half a dozen times, the devs (with fast machines and possibly smaller development dataset) go "works fine", and QA just gives up.
> Plus load times are usually the one thing you start ignoring early on, just start the session, go take a coffee or a piss, and by the time you're back it's loaded.
In GTA V, when I tried to enjoy multiplayer with my friends the abysmal load times were what killed it for me.
You actually have to load into the game world - which takes forever - before having a friend invite you to their multiplayer world - which takes forever, again.
So both a coffee, and a piss. Maybe they fixed that now?
Then when you want to actually do an activity like a deathmatch you have to wait for matchmaking and then the loading - takes forever. Once you are finally in a match it's okay but as soon as the match ends you have to wait for the world to load again and then queue again which takes bloody forever. Spend 2hrs playing the game and have only a few matches, more time spent looking at loading screens than actually playing anything.
> more time spent looking at loading screens than actually playing anything.
This could easily compete for the most expensive bug in history, up there with the Pentium Bug. It might have halved the revenue potential of a billion dollar franchise.
Judging from your word choice "deathmatch" and your experience with long loading/matchmaking times I guess you might be a fellow Quake Champions player. Even if you are not, I agree that long loading times are a mood killer when you just want to play a couple of quick matches after work in your limited free time. It is even worse when you know the game's development is abandoned and it will never get fixed, even though you enjoy the game itself.
GTA V has a deathmatch mode and the parent comment sounds like it's talking about that. Especially the "once it's over, you need to load into the primary session, then wait for matchmaking, then wait for loading again" sounds exactly like GTA V.
Back in dialup/DSL days I discovered a texture compression issue in America's Army (the free US Army game) that doubled its download/install size. Typical download times were about a day and the resume mechanism was poor, so this had the potential to save a lot of grief, not to mention hosting costs. I emailed them, they banned me for hacking, and the next version still had the same issue. Shrug.
I was only able to find contact information for one person who I knew was probably technical (from their posts), so I sent it to them.
I never learned the "other side" of this story, but a few years later the same dev team tried to recruit me at a CS contest, to which I politely declined.
More details: I was young, without credit card, and gaming on a mac. AA was free and mac compatible. For a while -- apparently mac ports of unreal engine games were approximately all done by a single very productive contractor and from what I understand the US Army, uhh, stopped paying him at some point. So he stopped releasing the mac ports. From my point of view, this meant that I could only play with other mac users and couldn't use any of the fancy new maps. Logs indicated that the compatibility problems with the new maps were not particularly deep, so I got to parsing the unreal map files and was able to restore compatibility by deleting the offending objects. I implemented texture decoding/encoding mostly for curiosity and because textures were well documented in the reverse engineering "literature." I imagined a workflow where someone would bulk export and re-import textures and aside from the texture names the one piece of metadata I needed was the format: RGBA (uncompressed) or DXT (compressed)? I realized that I could easily identify DXT compression from the image histogram, so I didn't need to store separate metadata. Nifty! But it didn't work. Lots of textures stored in uncompressed RGBA8888 "erroneously" round-tripped to DXT. After poring over my own code, I eventually realized that this was because on many of the textures someone had enabled DXT compression and then disabled it, dropping the texture quality to that of DXT while bloating the texture size to that of RGBA8888 (other textures were still stored as DXT, so compression itself was still working). I wrote a quick tool to add up the wasted space from storing DXT compressed textures in uncompressed RGB format and it came out to about half the total disk space, both before and after the top level installer's lossless compression. They could have re-enabled compression on most of the textures where they had disabled it without loss in quality, and if they had wanted a list of such textures I would have been able to provide it, but it didn't go down that way. When I shared what happened with my father, who had served, his reaction was "Now that's the Army I know!"
Oh man, this brings back memories. That game was a great tactical shooter back in the day. Sadly my PC was unable to keep up with the higher requirements required by it's updates.
Reminds me on loading "G.I. Joe" (from Epyx) on the C64 with a 1541 floppy disk. However, the long loads came after every time you died and meant you also had to swap 4 disks.
I remember as a kid I went to someone's birthday party in the 80s and we wanted to play a karate themed game on something that used a cassette tape. It took so long to load we went and played outside!
To be fair to GTA V, I don't think my installation was on a SSD because it was 50GB or something at the time (now it's 95GB?), but that said when it released SSDs were not as cheap or widespread as they are now so that's their problem. The linked article shows the initial load to be much shorter which did not match my experience.
Not to one up but we didn’t have any storage for our c64 for the first year or so. We would team up to enter a game from Byte or whatever (one reader, one typer, one proofreader) and then protect that thing with our lives all weekend to keep people from unplugging it. The machine code games were the easiest to type but if they didn’t work you were kind of hosed lol.
C64 tape with turbo was actually faster (~500 Bytes/s) than non turbo Floppy (~400 Bytes/s).
Many 8bit Atari owners will have horror memories of aborted Tape loads. After over 20 years someone finally discovered a bug in original ATARI Tape loading ROM routine resulting in randomly corrupted loads, no amount of sitting motionless while the game loads would help in that case :)
Oh gods, now I'm reminded of a cassette based game where there was one enemy where if he got a solid hit on you, you got bumped back to the beginning of the level.
Which meant the game displayed "rewind to mark 500 and then hit play" and you had to do that to restart lolsob.
I read that after patches GTA V is now around 95GB.
Call of Duty: Black Ops Cold War is around 200GB fully installed with all features (HD textures etc).
Some people have insinuated this is intentional to crowd out other games on console hard disks and make moving away from CoD have an opportunity cost. It's probably just laziness.
I haven't looked into it in the past but some prior offenders had a lot of space wasted from uncompressed multi-lingual audio. Instead of letting us choose a language it installs them all so you can switch in game, and uncompressed for saving the CPU for game logic. For CoD the optional HD texture pack is 38GB so that's still a lot unaccounted for.
Titanfall did this. If I recall correctly, the full install size was about 48GB, 35GB of which was just uncompressed audio. And that was back in the days when 120GB (or less) SSDs were common. A total self-own and never fixed or understood.
It's not like decoding audio takes enough time on any modern multi-core processor to disrupt the game loop. It's not even on the radar.
I mean. Jesus christ. Devs used to be proud they could optimize and run their programs on slow machines. What happened? Thats like making a game for casual players but only < 5% of the global population can even afford to play it. How does this make business sense?? I'd understand if it was a game breaking new ground with some hectic VR or something like that but it isnt.
> You actually have to load into the game world - which takes forever - before having a friend invite you to their multiplayer world - which takes forever, again.
Is that... the same problem? Is microtransaction data different in your friend's multiplayer world than it is in the normal online world?
The article mentions story mode loading as well as online loading, but as I mentioned in another comment the story time shown there is much lower than what I experienced, probably because SSDs are now standard and were rarer in 2013 (I could not justify 50GB+ on this one game at the time). So it may be a mixture of factors.
Was new guy at a startup.
Soon noticed that chuck Norris was in our compiled JavaScript.
Turned out Someone had included the entire test suite in production deploy.
Had been like that for nearly a year.
A few minutes of work brought out client js file from 12MB to less then 1mb.
Ha, this is one of the reasons why I also include outlandish and wrong-looking stuff in unit tests. If we see where it doesn't belong, then we know for sure that we are doing something wrong.
Most often I use unicode strings in unexpected alphabets (i.e. from languages that are not supported by our application and that are not used by the mother tongue of any developer from our team). This includes Chinese, Malayalam, Arabic and a few more. There was a time when I wanted to test the "wrong data" cases for some deserialising function, and I was part annoyed and part amusingly surprised to discover that doing Integer.parseInt("٤٣٠٤٦٧٢١") in Java does parse the arabic digits correctly even without specifying any locale.
I've seen "Carmen Electra" myself :-) And other funny (more gross) names in the dark corners of huge databases...
I also had a customer discover a playboy center page at the end of a test we sent to them. One of the dev thought it'd be a nice reward. Things went very bad from him right after the phone call...
You mention quadratic behaviours and there's probably some truth to that, but it seems to me that it's partly a C++ problem. In any other langauge nobody would even consider hacking up JSON parsing using a string function. They'd use the stdlib functional if available or import a library, and this problem wouldn't exist.
A lot of other languages make use of the c standard library functions to parse floats (and to do various trigonometric functions), so they may be more similar than you imagine.
Not super relevant, though. The average standard library from another language is a lot more hardened than the function written by Joe Sixpack in C last night.
But C++ had at least a hash_set/hash_map since forever (or just set/map which are still better than this)
I'm sure there are libraries to parse json in C++ or at least they should have built something internally if it's critical, instead they have someone less experienced build it and not stress test it?
>I'm sure there are libraries to parse json in C++
There certainly are, but adding a library is much more difficult in C++ than pretty much any other language which seems to tempt people into hacky self-built parsing when they really ought to know better.
Yea, given how easy it was for the author of the post to find it, I would guess that literally nobody in the last decade bothered to run a profiler to see where they were spending time.
The only possible explanation is that management never made it a priority.
I could see this happening. A project/product manager thinking "We could spend $unknown hours looking for potential speedups, or we could spend $known hours implementing new features directly tied to revenue"
Which is kind of ironic since this fix would keep players playing for more time, increasing the chances that they spend more money.
I think we need a new vocabulary to cover situations like this one. It's not just that other issues took priority here, it's that this wasn't even entered into the list of things to give a crap about. It's something like digital squalor.
For the online games I worked on (a few of the recent NFS games) the items database was similar to the final set quite early in production and we kept an ongoing discussion about load times.
I really liked this article, but I am a bit surprised that this made it into production. I have seen a few instances of this type of slowdowns live for very long, but they tend to be in compile times or development workflow, not in the product itself.
And because a merchandising push in many games may be another 10-50 items, the first couple times the % increase is high but the magnitude is low (.5s to 1s) and by the time you're up to 1000, the % increase is too small to notice. Oh it took 30 seconds last week and now it's 33.
Boiling the frog, as it were. This class of problems is why I want way more charts on the projects I work on, especially after we hit production. I may not notice an extra 500ms a week, but I'm for damn sure going to notice the slope of a line on a 6 month chart.
I heard that Chrome team had this KPI from very early on - how much time it takes for Chrome to load and it stayed the same to date. i.e. they can't make any changes that will increase this parameter. Very clever if you ask me
Google lately "optimized" Chrome "time for the first page to load" by no longer waiting for extensions to initialize properly. First website you load bypasses all privacy/ad blocking extensions.
Yeah I think that's the kind of odd behaviour that those KPI's end up causing; they 'cheat' the benchmark by avoiding certain behaviour, like loading extensions later.
I mean I can understand it, a lot of extensions don't need to be on the critical path.
But at the same time, I feel like Chrome could do things a lot better with extensions, such as better review policy and compiling them to wasm from the extensions store.
It would be interesting to see what JSON library they used that uses scanf for parsing numbers. Nothing like a painters algorithm type scenario to really slow things down, but also JSON numbers are super simple and don't need all that work. That is hundreds of MB's of unneeded searching for 0s
Unlikely to be a library, either it's libc or it's homegrown.
The only thing most game companies do when it comes to external libraries is to copy the source code of it into their repo and never update it, ever.
OpenSSL is this way, it's a required installation for Playstation but debugging it is seriously hard, and perforce (the games industries version control of choice) can't handle external dependencies. Not to mention visual studio (the game industries IDE of choice..) can't handle debugging external libraries well either.
So, most game studios throw up the hands, say "fuck it" and practice a heavy amount of NIH.
Visual Studio keeps toying with the idea of a "NuGet for C++" and it is amazing that it still hasn't happened yet. It may seem to indicate that it isn't necessarily the IDE that can fix it, but the user's attitude. How much of the NIH and "just copy that dependency into the tree" is still encouraged for "security" [0] and "control"/"proprietary source"/"management" reasons?
[0] Despite it being an obvious anti-pattern that you aren't going to update dependencies that require copy/paste and manual merge reviews, so security problems should be obviously more rampant than in systems where updating a dependency to the latest security patch is a single install command line (or update button in a GUI), there still seems to be so many C++ devs that love to chime in to every HN thread on a package manager vulnerability that they don't have those vulnerabilities. They don't "have" dependencies to manage, no matter how many stale 0-Days they copied and pasted from outside projects, they don't count as "dependencies" because they are hidden who knows where in the source tree.
That certainly is the most recent attempt. They've had projects of one sort or another going back at least as far as 2013 from mentions in public blog posts but so far none of them seem to have got much traction with the community. Here's hoping it works this time?
In another decade, there's going to be a story here about somebody getting their hands on the original source for this game, and the JSON parser will be a 10-line function with "//TODO: optimize later" at the top.
But is quadratic the real issue ? Isn't that a developer answer ?
The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases. And what is small? Medium? Large?
The truth is that there is no one size fits all and assumptions need to be reviewed periodically and adapted accordingly. And they never are... Ask a DBA.
Yes. That is literally the entirety of the issue: online loading takes 5mn because there are two accidentally quadratic loops which spin their wheel.
> The best algorithm for small, medium or a large size are not the same and generally behave poorly in the other cases.
“Behaves poorly” tends to have very different consequences: an algorithm for large sizes tends to have significant set up and thus constant overhead for small sizes. This is easy to notice and remediate.
A naive quadratic algorithm will blow up your production unless you dev with production data, and possibly even then (if production data keeps growing long after the initial development).
quadratic is a fancy way of saying "this code is super fast with no data, super slow once you have a decent amount"
The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list.
Why quadruple? Because you end up scanning a list which is twice as long. You have to scan that list twice as many times. 2x2 = 4. The larger list no longer fits in the fast (cache) memory, among other issues. The cache issue alone can add another 10x (or more!) penalty.
> quadratic is a fancy way of saying "this code is super fast with no data, super slow once you have a decent amount"
Well, that is an abuse of the term, by people that sometimes don't actually know what that really means. Up to a point, quadratic IS faster than linear after all for example. Too many developer love too abuse the word blindly.
If it is badly tested with no data, it is badly tested with no data. Period. Not "quadratic".
> The problem is that when you double the amount of stuff in the JSON document, you quadruple (or more) the scanning penalty in both the string and the list.
My point was precisely it depends on the data and initial assumption are to be routinely revised. I was making a general point.
Maybe the guy was pinky-sworn that the JSON would hardly change and that the items were supposed to be ordered, sequential and no more than 101. For all you know it is even documented and nobody cared/remembered/checked when updating the JSON. But we don't know, obfuscated code don't comes with comments and context ...
Or, it is actually a real rookie mistake. It probably was, but we don't have all the facts.
> Well, that is an abuse of the term, by people that sometimes don't actually know what that really means. Up to a point, quadratic IS faster than linear after all for example. Too many developer love too abuse the word blindly.
There is absolutely no guarantee that a quadratic algorithm has to be faster than a linear algorithm for small N. It can be, in some situations for some algorithms, but the complexity class of the algorithm has nothing to do with that. A quadratic algorithm may well be slower than a linear algorithm for any N.
The only thing the complexity class tells us is that starting from some N the linear algorithm is faster than the quadratic algorithm. That N could be 0, it could be 100, or it could be 1 billion.
In my experience it's usually between 0~1000, but again, that depends. The complexity class makes no such guarantees. The complexity class tells us the general shape of the performance graph, but not exactly where the graphs will intersect: this depends on the constants.
> If it is badly tested with no data, it is badly tested with no data. Period. Not "quadratic".
It is both. The problem is that the algorithm has quadratic complexity. The fact that it was badly tested caused this fact to remain hidden while writing the code, and turn into a real problem later on.
>Well, that is an abuse of the term, by people that sometimes don't actually know what that really means. Up to a point, quadratic IS faster than linear after all for example. Too many developer love too abuse the word blindly.
The problem with this argument is that if the data size and constants are sufficiently small enough people don't care about whether the linear algorithm is slow. In the case of JSON parsing the constants are exactly the same no matter what string length algorithm you use. Thus when n is small you don't care since the overall loading time is short anyway. When n is big you benefit from faster loading times.
I honestly don't understand what goal you are trying to accomplish. By your logic it is more important to keep short loading times short and long loading times long rather than do the conventional engineering wisdom of lowering the average or median loading time which will sometimes decrease the duration of long loading screens at the expensive of increasing the duration of short loading screens.
In the small case here, there is no meaningful difference in speed between parsers. Using a quadratic algorithm has no advantage and is just an incorrect design.
The popular view is that companies who write software know how to prioritise, so if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile.
I disagree. If there are no internal incentives for the people who know how to fix this to fix it, or if there's no path from them thinking fixing it could improve revenues to being assigned the ticket, things like this won't get fixed. I can fully believe the load times will result in fewer users and lower expenditure.
I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.
You have just described why I laugh anytime someone complains that government is inefficient. ANY organization of sufficient size is "inefficient" because what a large organization optimizes for (for reasons I cannot explain) cannot align with what that organization's customers want optimized.
With the added difference that governments also have to be far more procedural by virtue of the way they are set up. Regardless of size they are accountable and responsible to a far higher degree in the eyes of the population they represent so there is a legitimate reason to be "slow".
In games the added reason to be slow is that game code is by definition some of the least mission critical code one could find (competes with 90% of the internet web code). Your Linux or Windows code might run a hospital's infrastructure or a rover on another planet. A game on the other hand can launch with bugs the size of your windshield, and can stay like that forever as long as people still pay. And people will pay because games are not unlike a drug for many people.
As such most game coding teams and coders are "trained" to cut every corner and skimp on every precaution. They're not needed beyond a very low baseline as far as software is concerned.
Look at the amount of bugs or cheats incredibly popular games like GTA or CoD have. These are billion dollar a year franchises that leave all this crap on the table despite all the money they make. They have all the resources needed, it's a conscious call to proceed like this, to hire teams that will never be qualified enough to deliver a high quality product and will be encouraged to cut corners on top of that.
Source: a long time ago I worked for a major game developer in a senior management role (unrelated to the dev activity) and left after feeling like "facepalm" for too long in every single SM meeting.
> but the volume of released games vs other types of software makes any sort of comparison unfair.
No comparison is entirely fair but I think your objection is unfounded. The quantity of games being released is irrelevant when we're talking about the quality of their code. Which is really bad for most big titles.
If anything the comparison was unfair in the other direction. Sure, an OS or browser have a lot of bugs. But if games and their code had a fraction of the scrutiny something like Windows gets you might just find out that 4 lines of code were written "by the book". It's something any honest game dev will confirm, game code is a stack of spaghetti code on top of more spaghetti code. The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on. Then you notice there's no widespread pushback from gamers (because unless it's an absolute bomb there won't be) and move on with the next round of "features", nobody has time for fixing bugs or combing the spaghetti.
One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy. I've done a lot of hiring over my career and of one thing I'm certain. Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack. It was a lesson I learned the hard way.
Issues like the low criticality of game code, the "crunch" work style, the idea that you should just get it out there as quickly as possible, the lack of serious scrutiny into the matter, etc. all compound each other to create a coding (and coordination) style that's hard to shake off.
> No comparison is entirely fair but I think your objection is unfounded.
I should have clarified.
Doing an absolute numerical comparison in terms if "bad game code" to "bad B2C code" is unfair, because now days there is a lot more game dev code than B2C code (if we ignore the web, which is a rather lot of code to ignore I'll admit :) ).
All that said, most of the games on my phone run very well. Most of the games on my PC run very well. Do the massive over budget AAA titles have issues? Of course. Human organizations are really bad at creating large projects, we all know that social interaction overhead scales non-linearly with # of people, large projects get bogged down.
(Now throw in B2B software where the CTO makes the purchasing decisions and only people lower level in the company have to use it. All of a sudden enterprise software starts to suck as much as AAA games!)
> The philosophy is that you can just go ahead with bad code because you can always fix it with a patch later on.
Sure, for AAA games. But plenty of games that I play don't have those sorts of issues.
Meanwhile my Windows start menu only opens every other time I press the start menu button. This happened for years, then it was fixed for a bit, then a few patches later it started happening again.
The start menu on Windows should be the 2nd to last thing that breaks, right behind the mouse driver.
Without huge testing efforts, large software projects will collapse under its own weight. The type of software isn't material to the problem.
> One other problem is that eventually some people coming from the gaming industry will end up switching to other types of software development but will stick to the philosophy.
I've hired plenty of ex-game devs, I agree they have some holes in their skills. But I've also found out that if you sit them down they can write code that traditional software engineers couldn't manage. Spending too long in any one paradigm fixes ones mindset into that paradigm.
I also agree that the quality of code someone puts out is highly influenced by the environment they work in. But I've worked with plenty of ex game devs who take pride in writing correct code the first time around.
> Whenever someone came with most of their career in game development or most of the recent experience I asked for the CV to be put at the bottom of the stack.
That's a shame, because I wouldn't have accomplished some of the things I've done in my career if I hadn't hired a few game developers along the way! I've seen some stupid crazy stuff get pulled off, code that "shouldn't be able to exist", but did exist, and was rock solid reliable, all written by game developers.
People are people, and sure 9 out of 10 the person who bangs on kernels for a living is probably super careful about every line of code they write, but I'm not going to judge an entire field because the most bureaucratic outputs of that field are horrible.
That'd be like saying everyone who writes web apps is an incompetent developer because Jira is slow.
Or that all software developers are bad because, honestly, the majority of users only notice what we make when it breaks.
Well, we only hear about the small % of games that have huge problems, not the larger number of games that work just fine out of the box.
Any sufficiently large institution, over time, will prioritise self-preservation over achieving their core mission. This is sociology 101. Once a company has enough users to make it hard or impossible to measure immediate performance, self-preservation is achieved with internal manoeuvering and selling to execs.
It's not a perfect remedy, but you have to loop in the people affected by decisions as part of the decision making structure. That is, for example, customers and workers have to be part of the management structure.
This doesn't happen because it would reduce the power of top decision makers and potentially impact profits. e.g. a customer might ask for a chronologically ordered timeline on Facebook, but that would harsh engagement metrics, revenue, etc. If stuff like this did happen more often though, you'd get products and services that more often achieve their stated aims.
I understand what you're saying. When I was working as a Product Manager, I had to try really hard to do things for the customer (after user testing etc.)
Normally the response from management was "but will that increase our profits?", to which my answer was "eventually, yes"
Even if you take engineering you see that adding more links on a chain increases latency and demands more auxiliary circuitry. But at least in engineering you can design each part to do what you want it to do close to perfection. And it will scale better because you can build everything to spec and bin, which is why we automate a lot of tasks. With humans that can't happen. Human "binning" is a constantly moving target.
After tangentially working on this for a long time I'd say that the core issue is so deeply ingrained in human psyche that it may not even be a matter of education (starts early), let alone organization (starts happening when everything else is "set in stone"). There's no organizational structure that fits the types of activities we humans tend to do these days and that can deliver low latency, consistent results at scale. We mitigate issues in one area by accentuating them in others.
You can have one large flat structure but the load on the single coordinating circuit (the manager) will compromise quality. You can split the thing in multiple, individually coordinated units but the added layer of coordination and latency will compromise performance.
Maybe some form of general purpose but not quite full AI, something that combines human like intelligence and engineering like consistency, might be able to do what humans are supposed to but without the variability of humans (which is both good and bad).
Introducing AI into work relations is how you turn every org into Uber or Amazon delivery: platforms where the worker has no real agency on his work, the apotheosis of alienation. I have no doubt that someone will try it (already we see it creeping in for hiring), I just think it will be Fundamentally Bad.
We don't really know what it is simply because we haven't introduced any "real" AI anywhere, let alone:
>> some form of general purpose but not quite full AI
Talking about efficiency as you scale up large organizations, it's inevitable that humans will introduce delays and variability in the work which cannot be eliminated because it's human nature, biologically and psychologically. Since humans can't change on a timescale that makes this discussion relevant, the only way for very large organizations (like large companies or governments) to operate just as efficiently as small ones is if they rely (quasi)exclusively on some AI that's as capable as top individuals at delivering results but with fewer of the drawbacks. Not only would it not operate as 100.000 distinct entities but as one or a few, it would also run consistently and predictably.
There was a fantastic discussion some years ago on ways to design an organization to minimize the tendency to drift towards self-preservation instead of remaining customer-focused.
The HN discussion[1] was started by an article that provided numbers that seemed to suggest that Wikipedia's spending was slowly spiraling out of control.
That really is a fantastic quote and a fantastic thread!
What really stuck out is that Buffet said its the most important thing about Business and he didn't learn it in Business school. Same! I did an MBA and it was the biggest elephant in the room.
Fair point. The next poster explains how Amazon identifies and tries to mitigate it:
"if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right"
Many people would say "more accountability" but I've seen that used successfully to deflect lightning strikes to innocent people who were then fired so... I'd like to know as well.
Competition/choice, which means that self-preservation requires that they care about efficiency. It obviously that wasn't enough here, but definitely tames some inefficiencies.
Organizations are like spheres. Only a small part of the sphere has an exposed surface that is in contact with the outside world. As you grow the sphere most of the mass will be inside the sphere, not near the surface.
And I laugh every time people claim that it is only governments that can be inefficient. Most large commercial companies are inefficient and almost not functioning at all.
That the process breaks down in some cases doesn't mean they don't know how to prioritize. They clearly know how to prioritize well enough to make a wildly successful and enjoyable game. That doesn't mean no bad decisions were made over the last decade or so of development.
Like anything else, things will be as bad as the market allows. So I'd expect monopolies to do a worse and worse job of making good decisions and companies in competitive fields to do a better and better job over time. Thus the difference between TakeTwo and Facebook, and the need for lower cost of entry and greater competition in all economic endaevors where efficiency or good decision making is important.
Does it? Just because something is good or successful doesnt mean it was made well. Thats why we are seeing stories of crunch, failures of management compensated by extreme overwork.
Now you are moving the goalposts to a philosophical discussion as to what types of business you like or don't like, and I'm not sure any progress can be made with that approach. Some will share your values, others will find them repugnant, and in any case it doesn't have much bearing on rockstar load times.
I dont think anyone likes excessive loading times or crunch. You might argue about the importance of optimization or the necessity of crunch, but i dont think anyone can argue this isnt a failure of management and communication
> I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month.
The messenger website has been atrocious for me lately. On my high-powered desktop, it often lags a bit, and on my fairly high-end laptop, it's virtually unusable. I thought it must be something I changed in my setup, but it's oddly comforting to hear that I'm not the only one with such issues.
[tinfoil hat] A part of me believes that this was their intention in slowing down their browser site to the extent it has (and there is absolutely zero reason as to why it should load as slow as it does) in order to drive downloads for this application. [/tinfoil hat]
It's funny. That page says "A simple app that lets you text, video chat and stay close to people you care about." If it's so simple, then why does the browser version of the site take forever to load?
Yeah, it used to work well in mobile browser, then only in desktop mode in mobile browser, then not at all. Now it only "kinda sorta" works in desktop browser on ridiculously overpowered PC.
Gotta keep the walls tight in that private garden.
Unfortunately I'm on Linux, and they don't seem to provide a Linux client. I did try out https://github.com/sindresorhus/caprine/, which worked really well, but I'm one of those weirdos who prefers websites to desktop apps.
> if a problem like this isn't fixed, it's because they've done the calculations and decided it's not worthwhile.
If it ever reached the point where it had to be an item in a priority list, it's already a failure. Some developer should have seen the quadratic behavior and fixed it. It's not the type of thing that should ever even be part of a prioritized backlog. It's a showstopper bug and it's visible for every developer.
> I can fully believe the load times will result in fewer users and lower expenditure.
Is GTA online still attracts new users in droves? I doubt.
If the old users live with the loading time for years, they are likely to continue living with it. It would be nice if Rockstar fixes it, but I doubt it would be anything except a PR win.
I rarely play it and the load time is actually the main factor. If I have an hour to play a game waiting for GTA V to load unfortunately feels like a waste of time and a chore, so I play something else.
Before GTA online they entertained themselves in other ways. Eventually, they'll move on. The more friction there is to continue playing GTA online, the easier it is for there to be something to pull them away. Rockstar are now competing to be a use of someone's time, not for them to buy the game.
> I think we'll see this happen with Facebook Messenger. Both the apps and the website have become slow and painful to use and get worse every month. I think we'll start to see engagement numbers dropping because of this.
In fact, I think the iOS app for FB Messenger did get a redesign due to problems and it’s rewritten from scratch? I remember being pleasantly surprised after the big update… It became lightweight, integrates well with iOS and supports platform features.
On the other hand, the desktop app or the website is a shitshow :-(
I imagine the conversation between the programmer(s) and management went exactly like this:
Management: So, what can we do about the loading times?
Programmer(s): That's just how long it takes to load JSON. After all, the algorithm/function couldn't be more straightforward. Most of the complaints are probably coming from older hardware. And with new PC's and next-gen consoles it probably won't be noticeable at all.
Management: OK, guess that's that then. Sucks but nothing we can do.
Management had no idea of knowing whether this is true or not -- they have to trust what their devs tell them. And every time over the years someone asked "hey why is loading so slow?" they get told "yeah they looked into it when it was built, turns out there was no way to speed it up, so not worth looking into again."
And I'm guessing that while Rockstar's best devs are put on the really complex in-game performance stuff... their least experienced ones are put on stuff like... loading a game's JSON config from servers.
I've seen it personally in the past where the supposedly "easy" dev tasks are given to a separate team entirely, accountable to management directly, instead of accountable to the highly capable tech lead in charge of all the rest. I've got to assume that was basically the root cause here.
But I agree, this is incredibly embarrassing and unforgiveable. Whatever chain of accountability allowed this to happen... goddamn there's got to be one hell of an internal postmortem on this one.
I can pretty much guarantee that there was no discussion with management like that. From experience, live ops games are essentially a perpetually broken code base that was rushed into production, then a breakneck release schedule for new features and monetization. I've personally had this conversation a few times:
Programmer: Loading times are really slow, I want to look into it next sprint.
Management: Feature X is higher priority, put it in the backlog and we'll get to it.
At my last job I had that conversation several times :( Our website would regularly take 30s+ for a page to load, and we had an hour of scheduled downtime each week, because that’s how long it took the webapp to restart each time code was pushed. “Scheduled downtime doesn’t count as downtime, we still have the three 9’s that meets our SLA, and there’s nothing in the SLA about page load times. Now get back to building that feature which Sales promised a client was already finished”...
Aside from being generally shameful, the real kicker was that this was a "website performance & reliability" company x__x
Reminds me of working for a company in the 1990's that ran constant television ads convincing everyone their experts could fix everyone's networking problems. OTOH, as an engineer working at the company the file share used for editing code/building/etc, died for an hour or two seemingly every day when the network took its daily vacation.
Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN, or the ones we sent onsite were as incompetent.
> Many of us, after having run out of other crap to do, would sit around and wonder if the "B" grade network engineers were assigned to run the company LAN
Internal IT is almost invariably a cost center, the technicians providing the service you are selling to customers are working in a profit center. So, yeah, probably that plus be managed in a way which focussed on minimizing cost not maximizing internal customer satisfaction or other quality metrics.
Former gamedev here. I can vouch for this conversation. The idea that management would ask about slow loading times doesn't seem to be too realistic in my experience.
I once did get to look an optimsing a bad load time... only because at that point it was crashing due to running out of memory. (32-bit process)
In this case it was a JSON, but using .NET so Newtonsoft is at lease efficient. The issues were many cases of:
* Converting string to lowercase to compare them as the keys were case insensitive. (I replaced with a StringComparison.OrderinalCaseInsensitve)
* Reduntant Dictionary's
* Using Dictionary<Key, Key> as a Set. replaced with HashSet.
The data wasn't meant to be that big, but when a big client was migrated it ended up with 200MB of JSON. (If there data was organised differently it would've been split accross many json blobs instead)
It would also be nice to handle it alls as UTF8 like System.Text.Json does. That would half all the strings saving a fair bit. (I mean the JSON blob it starts with is converted to UTF-16 because .NET)
Ughh Every time I want to fix anything I have to sneak it in with something else I'm working on, or wait so long to get approval that I forgot all the details.
I stopped doing this after realizing that all the extra work was just taking time away from my family. The last time I did anything out of work hours was when I told my manager that I was doing some optimization in my free time and he responded "I don't want you working on that in your free time, you could be working on this instead!". I don't plan on doing anything out of work hours again. He left, and now we're hiring more people, instead, since I can't get everything done. Win for everyone.
Oh i never work outside of work hours. I meant that I'll just include fixes to things that are tangentially related to my current user story, or even sometimes not related at all. It helps that the team implicitly trusts my code, so anytime I say "oh and I also fixed this", it's fine. The problem is that if I say "oh I want to fix this" it gets put in the system, and ranked, and just kills my momentum.
Normally I'd agree with you but this particularly problem is SO visible, and experienced by everyone, that I have to think management must have looked into it. I mean, it's the loading screen. They couldn't not encounter it themselves, every time.
But I could be wrong. I've only worked with sites and apps not gaming.
I don't think you choose to become a manager at one of the world's top video game companies if you don't love video games.
Hell, I don't think one of the world's top video game companies would hire you as a manager if you didn't convince them, in interviews, that you understand and love their products by using them yourself.
Using your own company's products whenever possible is generally a pretty important part of being a manager.
I think someone in QA likely might have the scenario you describe, but I think management might only be playing the game 2-3 times a day at most. They're likely trying to keep the studio organized enough to have the game ship decently.
I experienced this to incredible extremes. It was taking our clients two hours to load their data into an external tool that they use, daily, to view that data. This was caused by our use of a data store that was only half supported by that tool. I showed that if we took two weeks to switch to the other data store, their load times would be 30 seconds. It took two years, and a new manager, to get that implemented. Unfortunately, most of the clients are still using the old data store. They haven't had time to switch to the new version...
I had that exact conversation at my old job. They only started listening to me when some requests started timing out because of Heroku's 30 second request duration limit.
Same thing happened with memory consumption. The problem was ignored until we regularly had background jobs using > 4 GB of memory, causing cascading failures of our EC2 instances and a whole bunch of strange behaviour as the OOM killer sniped random processes.
The old maxim of "Premature optimization is the root of all evil" has over time evolved to "If you care one iota about performance, you are not a good programmer".
That belief is getting a bit outdated now that computing efficiency is hitting walls. Even when compute is cheaper than development, you're still making a morally suspect choice to pollute the environment over doing useful work if you spend $100k/yr on servers instead of $120k/yr on coding. When time and energy saved are insignificant compared to development expense is of course when you shouldn't be fussing with performance.
I don't think the anti-code-optimization dogma will go away, but good devs already know optimality is multi-dimensional and problem specific, and performance implications are always worth considering. Picking your battles is important, never fighting them nor knowing how is not the trick.
I agree 100% - the whole cheery lack of care around optimization to the point of it becoming 'wisdom' could only have happened in the artifice of the huge gains in computing power year on year.
Still, people applying optimizations that sacrifice maintainability for very little gain or increase bugs are still doing a disservice. People who understand data flow and design systems from the get-go that are optimal are where it's at.
Also, it’s not “never optimise”. It’s “only optimise once you’ve identified a bottleneck”. I guess in a profit-making business you only care about bottlenecks that are costing money. Perhaps this one isn’t costing money.
This one isn’t measured to be costing money. Trying to figure out how much money you’re losing as a result of performance problems is extremely difficult to do.
Not all performance problems have obvious "bottlenecks", some are entirely second-order effects.
For instance, everyone tells you not to optimize CPU time outside of hotspots, but with memory usage even a brief peak of memory usage in cold code is really bad for system performance because it'll kick everything else out of memory.
Precisely. Hasn't GTA Online done over a billion in revenue?
Given how incredibly successful it's been, it's conceivable the suits decided the opportunity cost of investing man-hours to fix the issue was too high, and that effort would be better spent elsewhere.
They're making lots of money but the ridiculous load times absolutely cost them money. It's not worth an unlimited amount of dev time to fix, but they definitely should have assigned a dev to spend one day estimating how hard fixes would be.
Arguably, it could actually make them money, since it provides a window of free advertising. I have no data either way, but I wouldn’t assume long load screens are necessarily bad for business.
Where does the belief that corporations are somehow perfect at doing cost-benefit analysis come from? Has anyone worked in a place where that seemed to be the case?
We're talking about an issue that has been loudly complained about for 7 years (and I am evidence that it makes people play the game much less often than they would like to) that some person without access to the source code was able to identify and fix relatively quickly, it would be surprising if it took an FTE 1 week to find this with access to the source code.
This is one of the most profitable games ever, they could hire an entire team to track this down for half a year and it wouldn't even be noticeable on their balance sheet. And I would bet money that it would have increased their active player base (which they care about because of the micro-transactions) in a noticeable way, as long as players were aware of the improvement so they would try it again.
Then again, using the correct data structure is not really optimisation. I usually think of premature optimisation as unnecessary effort, but using a hash map isn't it.
My belief is that your first goal should be cognitive optimization. I.e. make it simple and clear. That includes using hash maps when that is the proper data structure since that’s is what is called for at a base design level.
The next step is to optimize away from the cognitively optimal, but only when necessary. So yeah it’s really crazy this was ever a problem at all.
OTOH, at some point what is good from a cognitive viewpoint depends on what idioms you use. So it can be helpful to actively chose which idioms you use and make certain to not use those that tend to blow up performance wise.
Given the current code example, I don't even thing idioms come into it. Here a hash map data structure was called for regardless of whatever idioms come into other parts of the design. This is just fundamental and has nothing to do with functional programming, OOP, etc.
This is the key point: premature optimization would be e.g. denormalising a database because you think you might have a performance problem at some point, and breaking the data model for no good reason.
Here the wrong data structure has been used in the first place.
If it doesn't change semantics, it's optimization.
It's just a zero cost one, so if anybody appears complaining that you are caring to choose the correct data structure and that's premature (yeah, it once happened to me too), that person is just stupid. But not calling it an optimization will just add confusion and distrust.
Exactly. And not only being able to pick the correct data structure for the problem, (and possibly the correct implementation), but being able to know what is going to need attention even before a single line of code is written.
Most of my optimization work is still done with pencil, paper and pocket calculator. Well, actually, most of the time you won't even need a calculator.
That doesn't really apply here. I don't even play GTA V but the #1 complain I've always heard for the past 6 years is that the load times are the worst thing about the game. Once something is known to be the biggest bottleneck in the enjoyment of your game, it's no longer "premature optimization". The whole point of that saying is that you should first make things, then optimize things that bring the bring the most value. The load time is one of the highest value things you can cut down on. And the fact that these two low hanging fruit made such a big difference tells me they never gave it a single try in the past 6 years.
Sure it does apply. These complaints come out after the game has been released. They should have optimized this before they released, while they even designed the system. However that's considered premature optimization, when in fact it's just bad design.
See, that's exactly why you're wrong. This wasn't a bad "design". If the fix required to rebuild the whole thing from scratch, then you would have a point and thinking about it "prematurely" would've been a good idea. In this case, both the fixes were bugs that could've been fixed after the game was finished without having to undo much of the work done.
The whole point of the saying is that you don't know what's gonna be a bottleneck until after. Yes by optimizing prematurely, you would've caught those two bugs early, but you would've also spent time analyzing a bunch of other things that didn't need to be analyzed. Whereas if you analyze it at the end once people complain about it being slow, you're only spending the minimum amount of time necessary on fixing the things that matter.
You do understand that "my entertainment takes 6 minutes to load" is a very different problem from "my essential transportation kills people"? And therefore call for different approaches?
Putting aside the bad analogy that the other comment covers, the overall point I was making also covers doing the optimization before game release, but after the game is more or less complete. Rockstar optimizing during the final few months of playtest wouldn't be premature optimization.
Premature optimization is about trying to optimize the micro (a given function), while you don't yet have the macro (the game itself). If the function accounts for 0.1% of the performance of the final game, it doesn't matter if you make it 5x faster since that will only make the game 0.08% faster. You could've spent that time optimizing a different function that accounts for 10% of the performance, and even optimizing that function by 5% would make the game 0.5% faster, which is more impactful.
We used to start up Quake while we waited then we'd forget about GTAO. Later we'd discover GTA had kicked us out for being idle too long. Then we'd just close it.
That should be embarrassing for Rockstar but I don't think they would even notice.
The problem here isn't a lack of optimization, it's a lack of profiling. Premature optimization is a problem because you will waste time and create more complex code optimizing in places that don't actually need it, since it's not always intuitive what your biggest contributors to inefficiency are. Instead of optimizing right away, you should profile your code and figure out where you need to optimize. The problem is that they didn't do that.
I'd like to add, while GTA is running I'm really impressed by its performance, even / especially when it was on the PS3; you could drive or fly at high speed through the whole level, see for miles and never see a loading screen. It is a really optimized game, and that same level is continued in RDR2.
Which makes the persisting loading issue all the weirder.
I think this part of the Knuth's quote is central:
> Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered.
And he is explicitly advocating for optimizing the critical part:
> Yet we should not pass up our opportunities in that critical 3%.
And somehow, people have latched onto the catchphrase about "early optimization" that was taken out of context.
I went back and had a look at that maxim a few years ago and found that it actually doesn't say what many people claims it says. And definitively not as the blanket excuse for slow code that it has always to some degree been used as.
The reason for the misunderstanding is that the kinds of practices it actually talks about are uncommon today. People would often take stuff written in higher level languages and reimplement them in assembler or machine code. Which makes them more time-consuming to change/evolve.
It also isn't like it is hard to figure out which part of a piece of software that is taking up your runtime these days. All worthwhile languages have profilers, these are mostly free, so there is zero excuse for not knowing what to optimize. Heck, it isn't all that uncommon for people to run profiling in production.
Also, it isn't like you can't know ahead of time which bits need to be fast. Usually you have some idea so you will know what to benchmark. Long startup times probably won't kill you, but when they are so long that it becomes an UX issue, it wouldn't have killed them to have a look.
Back in the day when people talked about premature optimization it was about trivial things like people asking on stackoverflow whether a++, ++a or a += 1 is faster. It obviously is a net loss since ultimately it literally doesn't matter. If matters to you, you are already an expert in the subject and should just benchmark your code.
It's not just believable, but it's normal. I have spent quite a bit of my career maintaining software, and I don't recall one employers where low hanging fruit like this wasn't available everywhere.
The problem is not that developers can't optimize things: you will find some developers capable of figuring this problem out anywhere. What makes this low hanging fruit so popular is the fact that we aren't measuring enough, and even when we do, we aren't necessarily prioritizing looking into things that are suspiciously slow.
In the case of this example, the issue is also client-side, so it's not as if it's costing CPU time to Rockstar, so it's unlikely you'll have someone who can claim their job description includes wondering if the load times are worth optimizing. When problems like this one get solved is because someone who is very annoyed by the problem and either convinces a developer to even look into the problem. Most of the time, the people that suffer, the people that get to decide how to allocate the time, and the people that are in a position to evaluate the root cause of the problem never even get to talk to each other. It's the price we often pay for specialization and organizations with poor communication.
Organizations where the people deciding what has to be done next, and where the company culture dictates that the way forward is to either complete more tickets faster, or find ways to be liked by your manager, are not going to be fostering the kind of thinking that solves a problem like this one, but that's a lot of what you find in many places. A developer with a full plate that is just working on the next feature isn't going to spend their time wondering about load times.
But instead we end up blaming the developers themselves, instead of the culture that they swim in.
Hear hear, we should make a punch a dummy manager/BA/Code standards 'lead' day...
This code looks like someone with almost no experience hacked it together but because they were an intern and likely Rockstar is a toxic place to work, it never gets prioritized to be fixed.
I think if managers prioritized cycle time, metri s more, they'd find that they are encouraging a lot of practices which lead to horrible efficiencies - "measure twice cut once" is a positive mantra which leads to more solid designs with less bugs.
Agile sort of addressed this problem but unfortunately only at small size scales. Iteration and story capacity got overprioritized over quality, customer engagement, and self-leading teams.
Plus things such as scaled agile suffer from the oxymoron of planning fast iteration - if you have a master plan you lose the ability to respond to change and iterate, or you allow iteration and you must accept if any team iterates the whole team must discard the plan...which at some point means you either accept high cycle times or you figure out a way to decouple functionality to the extent the planning becomes the standard fallacy of waterfall - wasting meeting time to go over a plan that isn't based on anything.
I suspect that the core engine programmers moved onto other projects long ago, leaving GTA:O running with mostly artists and scenario designers to produce more DLC.
This bug wouldn't present in the first couple years with the limited amount of DLC, so by the time it got ridiculous there wasn't anyone left with the confidence to profile the game and optimize it. A junior dev could fix this, but would probably assume that slow loads are a deep complex engine problem that they won't be able to fix.
Alternatively, management would declare that there's too much risk doing more technical engine work, and not sign off on any proposed "minor" optimizations because it's too risky.
> This bug wouldn't present in the first couple years with the limited amount of DLC
GTA Online loading times have been infamous for a very long time. They were already unjustifiably bad when they released the game for PC, and at that point engine programmers would surely be involved.
This is very much the likely scenario. The money is in further DLC. The existing GTAO engine is "done" from their perspective.
I'd guess also that the next version of the base engine is in RDR2 or later and doesn't have these issues. But at the same time they likely wouldn't backport the changes for fear of cost overruns.
Something I've noticed in highly successful companies is that problems never get fixed because the sound of the money printer from the core business is deafening.
Our customer portal loads 2 versions of React, Angular, Knockout and jQuery on the same page? Doesn't matter, it's printing billions of dollars.
Rockstar's money printer is so loud that they don't care about problems.
Same thing for Valve, their money printer is so loud that they barely bother to make games anymore and let the Steam client languish for years (how did they let Discord/Twitch happen?).
>Same thing for Valve, their money printer is so loud that they barely bother to make games anymore and let the Steam client languish for years (how did they let Discord/Twitch happen?).
Not sure that's a fair criticism.
Alyx was widely praised. Artifact... Wasn't. I don't know about Dota Overlords. And that's just the last couple of years.
They've also developed hardware like SteamLink, Steam Controller, some high-end VR gear...
They develop a LOT. They just don't release a while lot.
I agree there should be a lot more work and effort in the client. And they constantly fuck up handling their esports.
But I don't think "barely bother to make games anymore" isn't one of them.
It's fair. They are not a games developer anymore. Alyx was good but a boutique game made by a behemoth games marketplace company and before it came out, almost a decade had passed since Portal 2...
Valve released Dota 2, and then ported it to a brand new engine, they've been consistently adding updates to that game for 10 years, they've been working on CS:GO, with similar effect, I think you lot forget that the majority of Valves developers were working primarily on team-fortress and counter-strike as mods before they hired them, they're primarily a multiplayer house.
Valve unlike most companies maintains their games for more than 10 years.
Just because they haven't released anything obvious to the casual observer like new single player titles that is easily marketed is only showing your ignorance of Valves entire catalog of games, and attitude to development.
I suspect sharepoint is the platform version of excel & VBA. You & I might hate it, but it gets powerful capabilities into the hands of regular people.
sharepoint is probably many non-engineer's very first exposure to actual version control with checkouts, checkins, version history, and merge.
This comment is absolutely hilarious to me, because the main selling point for SharePoint (for the longest time) was that you could version control Microsoft Office documents. Then intranets happened, someone wanted to build a Turing machine with it, and now it's the monstrosity we know now.
The JS API is total crap. One has to use it to develop anything that gets a bit more complex, but MS chose to ignore things that exists in the ES standard (like normal iterators, wtf) for quite some time to preserve backwards-compatibility with IE.
This make neither debugging nor any further exposure to the API any fun.
SP, like any standard platform (think i.e. SAP) invites the systems integrators to customize the crap out of it, which tends to impact the bad performance even more.
I see. We have some internal facing stuff that our IT has done using sharepoint. Its not pretty/modern by any stretch, but it works just fine. Was curious what the world looks like from the other side.
Isn't the search issue more of a indexing engine problem though? Can you plug in other engines?
I still have no idea what Sharepoint even is. The way I’ve always seen it used is a way to host sites with file hosting tied to them. It feels like an over engineered CMS.
In addition to other comments, Steam Chat had significant in-roads with the gaming audience that would eventually form the foundation of Discord. It is quite plausible, had Steam improved chat earlier, that Discord might have never gotten the traction it got.
>(with way better quality voice chat, give it a try some time if you haven't yet).
Well, I'm using VoIPs basically for >decade everyday and voice quality was never something I cared about (I mean that all soft that I've used was somewhat decent in that matter)
the most important thing is - how to get all people on the same program? and I'm finding it not so realistic to get all friends to Steam
is it really unbelievable? companies this big tend to prioritize hiring a shit ton of middlemen (VPs, project managers, developer managers, offshore managers) in order to avoid paying out for talent to build and constantly maintain the project. I guess paying a shit ton of money to 1 person to manage 10+ poorly paid contractors works out for them, accounting wise.
If one really examined the accounting for GTAO, I would bet that most of the billions of dollars that were earned in micro transactions went to marketing, product research, and to middle management in the form of bonuses.
Even if you view this as a business decision rather than a technical one, any smart project manager would realise a 6 minute loading time literally costs the company millions per year in lost revenue. (How many times have you felt like firing up GTA Online only to reconsider due to the agonising load time). I would guess this was simply a case of business folk failing to understand that such a technical issue could be so easily solved plus developers never being allowed the opportunity to understand and fix the issue in their spare time.
The insane loading times are literally the exact reason I haven’t played in years. Every time I played I just ended up frustrated and got distracted doing something else while waiting, so I just quit playing altogether. I don’t know how people stand the loading times.
Maybe the load times inadvertently work like the intentional spelling mistakes in a Nigerian scammer's email.
It's bait for the non-discerning customer who is more likely to empty their wallet for microtransactions because they have less experience with games so don't know what is normal :)
It's kind of hard to believe. GTA5's online mode is their cash cow, and 6 minute load times are common?! It's kind of amazing people even play it with those load times. It's such a simple problem that one dev could have found and fixed it within a day.
It's not at all hard to believe if you've been playing video games for a while.
Everything is getting slower and slower, and nobody cares.
When I played the Atari 2600, I had to wait for the TV to warm up, but otherwise there were no games with anything approaching load times (with 128 bytes of RAM in the console, who would know). The NES didn't have much in the way of load times either, but you did have to fiddle with the cartridge slot. SNES and Genesis didn't usually load (Air Buster being a noticeable exception). CD based systems sure did like to load, but that's somewhat understandable. In the mean time, more and longer boot screens. The Switch uses a cartridge (or system flash/SD cards), but it likes to load forever too.
PC Gaming has had loading for longer, but it's been getting longer and longer.
Some arcade games have lengthy loading sequences, but only when you turn them on while they read their entire storage into RAM so they can be fast for the rest of the time they're on (which in arcades is usually all day).
> Everything is getting slower and slower, and nobody cares.
It really depends. The latest crop of games I’ve played (Doom Eternal, Cyberpunk) loads way faster than games from a few years back (aforementioned GTA-V, Shadow Warrior 2…).
This is also on the same machine, so it’s not the hardware that makes it faster.
The more important thing about DirectStorage is probably that it will encourage games to use multithreaded async IO rather than serializing all their IO requests even when the underlying storage device requires dozens of simultaneous requests to deliver its full throughput.
I’m not entirely convinced that DirectStorage can do DMA directly from the device to the GPU. I suspect that even current NVMe devices aren’t quite fast enough for this to be a huge deal yet.
I think, but I’m not entirely sure, that Linux can do the peer to peer DMA trick. One nasty bit on any OS is that, if a portion of the data being read is cached, then some bookkeeping is needed to maintain coherence, and this adds overhead to IO. I wouldn’t be surprised if consoles had a specific hack to avoid this overhead for read-only game asset volumes.
Does it need "multithreaded async IO" or just "async IO"? It's usually async _or_ multithreaded; the native multi-request I/O APIs are single threaded, but if you have multithreaded I/O using simpler APIs, the system is batching them into one request at the cost of a little latency.
Kernel-mediated async disk IO is still a mess on all major platforms except for newer Linux kernels with io_uring. There's no way to call the APIs in a way that won't block sometimes, and to even have a snowball's chance in hell of not blocking requires giving up on the kernel's disk cache.
Also you're probably going to want to do multithreaded decompression anyway, and it'll be more efficient if you have the threads completing the reads do the decompression themselves. So in any case you probably want multiple threads handling the completion events.
NVMe is natively a multi-queue storage protocol, so there's no reason for the application or OS to collect IO requests into a single thread before issuing them to the lower layers. The normal configuration is for each CPU core to be allocated its own IO queue for the drive. But multithreaded synchronous (blocking) IO often isn't enough to keep a high-end NVMe SSD properly busy; you run out of cores and get bogged down in context switching overhead at a few hundred thousand IOPS even with a storage benchmark program that doesn't need any CPU time left over for productive work.
With a sufficiently low overhead async API (ie. io_uring) you can saturate a very fast SSD with a single thread, but I'm not sure it would actually make sense for a game engine to do this when it could just as easily have multiple threads independently performing IO with most of it requiring no synchronization between cores/threads.
"Hey we have this great new tech that makes things even faaster!!"
2 years later
"GTA 6 found to have double online load times, denies claims that game performs worse than GTA 5, tells people to upgrade their systems"
4 years later:
"Tech blogger reverses code, realizes someone managed to loop access between hard drive and gpu despite extremely common modern tech, gets 10x boost after spending a day fixing junk product"
Better technology just hasn't met its match from dumber management and more bureaucratic dev shops...
>It's not at all hard to believe if you've been playing video games for a while.
>Everything is getting slower and slower, and nobody cares.
Yes, modern games are so inefficient and laggy nowadays. Once your game world reaches a certain size it becomes unplayable and that's just in single player. Once you add 10 players you start to hit performance limits very quickly.
There were a few systems for the 2600 that used a cassette tape to load larger games than would fit on a ROM cassette.
I can’t recall the name, but I had the hack and slash adventure game variant. The connector on the custom cartridge was fiddly and required a stout rubber band to reliably work.
I am always amused by comments like this. You have no idea what development practices they follow (neither do I) but it's hilarious to read your tone.
GTA has achieved tremendous success both as an entertaining game and as a business. It's enjoyed by millions of people and generates billions in revenue. As per this article, it has startup problems (which don't seem to actually really hurt the overall product but I agree sound annoying) but the bigger picture is: it's a huge success.
So - Rockstar has nailed it. What exactly is your platform for analyzing/criticizing their processes or even having a shot of understanding what they are? What have you build that anyone uses? (not saying you haven't, but.. have you been involved with anything remotely close to that scale?)
You can be right in many places and still wrong in some, and enjoy enormous success as a result of all you have done well. That does not mean nobody can criticize you for something that you have clearly done wrong.
Successful people and businesses can be wrong. You are not making a case for why those development practices are okay, but are simply appealing to authority.
I and most other customers would argue that 6 minute loading times are atrocious, and if there is an easy fix like this, it makes me lose a lot of respect for the developer who doesn’t fix it. It maybe would even make me avoid them in the future.
A reputation is built over years, but can be lost pretty much instantly. Companies have to continue serving their customers to enjoy ongoing success.
They didn’t make an appeal to authority but an appeal to commercial success, and they’re right on.
The fact that GTAO is so popular should make most HNers rethink what they know about the commercial necessity of optimization vs building a compelling product.
The loading times were not initially that long and then the slow CPU makes a big difference.
This really only goes to show how much you can get away with if you have an outstandingly popular product that has no direct competition. Chances are that your product is not that compelling, if it performs poorly, that will hurt adoption. It will never become outstandingly popular in the first place.
I don’t need to be successful to have a platform to be outraged. It doesn’t matter that it’s Rockstar, if anything, the fact that they’re so successful and couldn’t be bothered to save so many people literal hours of their lives in loading time makes it worse.
GTA is fine ... but the storytelling is meh. Missions keep repeating, and there's little to draw you in. You drive somewhere, somebody gets whacked, you drive back. Rinse and repeat. The makers try compensate with shocking and crass violence and humor, but at some point it just feels kind of juvenile.
Maybe it got better in recent releases, I kind of stopped following after GTA4.
Tell that to Valve. The Source engine and all its games (Half Life 1, 2, Portal, Alyx) have horrible load times. They might not be as bad as the GTA example but they're long and extremely frustrating.
And yet, no one cares, Those games (and GTA5) all sold millions of copies.
The only way this stuff gets fixed is if (a) some programmer takes pride in load times or (b) customers stop buying games with slow load times.
(b) never happens. If the game itself is good then people put up with the load times. If the game is bad and it has bad load times they'll point to the load times as a reason it's bad but the truth is it's the game itself that's bad because plenty of popular games have bad load times
Also, any game programmer loading and parsing text at runtime by definition, doesn't care about load times. If you want fast load times you setup your data so you can load it directly into memory, fix a few pointers and then use it where it is. If you have to parse text or even parse binary and move things around then you've already failed.
I think there may sort of be another thing going on: Basically, that the length of load time is an indicator, "This is a really serious program." I've sort of noticed the same thing with test machines: the more expensive the machine, the longer it takes to actually get to the grub prompt.
Six minutes is probably excessive, but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game.
> but having GTA take 1-2 minutes to load almost certainly makes people feel better about the money they spent on the game than if it loaded up in 5 seconds like some low-production 2D adventure game.
Given that it has been the most common criticism of the game since it launched, I don't think anyone views it as a sign of quality.
It was probably fast 10years ago when the store had couple of items, the dev back then never thought that it would grow to 60k items. Classic programming right there.
As for profiling, Windows Performance Toolkit is the best available no?
Meh. It's ok to assume low number of items and code accordingly. What is not ok is for the company to ignore such a problem for years, instead if detecting and fixing it.
I've worked a number of places where the engineering culture discourages any sort of fishing expeditions at all, and if I weren't so stubborn everything would take 2-4x as long to run as it does. I say engineering culture, because at 2 of these places everyone was frustrated with engineering because the customers wanted something better, but the engineers would point at flat flame charts, shrug, and say there's nothing that can be done.
Bull. Shit.
There's plenty that can be done because there are parts of a process that don't deserve 15% of the overall budget. The fact that they are taking 1/6 of the time like 5 other things is a failure, not a hallmark of success. Finding 30% worth of improvements with this perspective is easy. 50% often just takes work, but post-discovery much of it is straightforward, if tedious.
My peers are lying with charts to get out of doing "grunt work" when there's a new feature they could be implementing. But performance is a feature.
>It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.
Having played the game, it's not surprising to me in the least.
I have never yet encountered another such 'wild-west' online experience.
It's the only game that is so un-modereated that i've ever played where the common reaction to meeting a hacker that is interested in griefing you is to call your hacker friend white-knight and ask him to boot the griefer-hacker from the lobby.
Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..)
On top of all this, Rockstar attempts to have an in-game economy , even selling money outright to players in the form of 'Shark Cards' for real-life currency , while 'modders' (hackers) easily dupe gold for anyone that may ask in a public lobby.
This isn't just all coincidence; the game lacks any kind of realistic checks/balances with the server for the sake of latency and interoperability -- but this results in every 13 year old passing around the Cheat Engine structs on game-cheating forums and acting like virtual gods while tossing legitimate players around lobbies like ragdolls -- meanwhile Rockstar continues releasing GTA Online content while ignoring playerbase pleas for supervision.
It's truly unique though -- an online battlefield where one can literally watch battles between the metaphorical white hat and black hat hackers; but it's a definite indicator of a poorly ran business when vigilante customers need to replace customer service.
Also, an aside, most 'mod-menus' -- the small applets put together using publicly available memory structs for game exploit -- most all have a 'quick connect' feature that allows hackers to join lobbies much faster than the GTA V client usually allows for. This feature has existed for years and years, and I believe it performs tricks similar to those listed in the article.
>Reports do next to nothing -- and 'modders' have some very real power in-game, with most fights between 'modders' ending in one of them being booted to desktop by the other exploiting a CTD bug (which are usually chat text-parser based..)
Interesting, back in the day the coolest CTD I did was to simply crank up the combo multiplier in S4 League to crash the game client on every player in the room except mine since that game was peer to peer and thus any form of hacking (teleportation, infinite melee range, instant kill, immortality, etc) was possible. The combo multiplier was set to 256 and thus every single particle was duplicated 256 times and this caused the game to crash.
> This online gamemode alone made $1 billion in 2017 alone.
There's the answer right there. They figure it's making $1B/yr, leave it alone. Maintenance? That cuts into the billion. Everyone moved onto the next project.
I would not at all be surprised if the long load time made for a sunk cost that kept people playing for longer sessions rather than picking it up for less than half an hour at a time.
I enjoyed GTA online but haven't touched it in well over a year, and the insane loading times are definitely a big reason why. For those who haven't played the game it's important to emphasize that it's
usually in the 5minute range, and even then you'll regularly end up with
connectivity issues or other problems that will kick you out of the lobby for
yet an other ~5 minute load.
When I played it wasn't uncommon to spend 30 minutes mostly looking at the
loading screen while you were trying to set up a play session with a couple of
friends.
If you're an adult with limited playtime it's just a complete dealbreaker. You
can't just decide to have a quick 20minute play session if you know that you'll
have to spend at least half of it looking at loading screens.
This is actually very possible - if they allowed people to dip in and dip out, they will. I remember when CP2077 came out on GeForce NOW and the wait times were 30+mins. I'd play a 6 hour session until I was booted off simply to make it worth the wait.
I stopped playing GTAV online a few years back because of the crazy load times, not only that but you have to go through 6+ minute load screens multiple times in many sessions.
This oversight has cost them millions of dollars easy.
> the reverse engineered version of GTA III and Vice City
Ohhh. Thank you for telling me about this. I just found a mirror and successfully built it for macOS. Runs so much better than the wine version. But I guess I'll never finish that RC helicopter mission anyway lol
That seems like a misapplication of the fallacy. If we assume 'worth their salt' is a synonym for 'good', then saying any good developer can operate a profiler is entirely reasonable.
I used to play this game a lot on PS4. I actually dropped it due to the ridiculous loading times... I still assumed it was doing something useful though. I can't believe they wasted so much of my time and electricity because of this. Even cheaply-made mobile games don't have bugs like this.
> their parent company unjustly DMCA'd re3
Wow, this is EA games level scumbaggery... I don't think I'm gonna buy games from them again.
Yes but the code was clearly derived directly from a decompiled binary; not ‘clean room’ reverse engineering. Hence, illegal, regardless of whether a dmca takedown notice is filed.
However, games like PUBG and Fortnite are free-to-play (PUBG is only free on mobile?) so in terms of actual sales, you could say GTA is more successful. Still not sure I'd class it as an "online shooter", though.
GTA is many things for many different people. For some people, it's a racing game, for others it is about socializing and modding cars, for some it's a dogfighting game, some spend thousands of hours grinding the same set of PvE missions (for some reason), and for many it's a third and first person shooter / warzone simulator.
I totally agree, the same way many games are. But in the context of saying "x is by far most successful y on the planet" I think we have to stop somewhere. Otherwise an open game like GTA V becomes the most successful game (in terms of copies sold) in pretty much every category you can't reasonably stick Minecraft in. I don't think there's much value in that.
Its the second best selling game of all time[1], and because Minecraft doesn't have guns, I suppose that would qualify it as the most successful online shooter (even if I think another genre would be more applicable).
Well but that's a bit like saying that since there are technically races in GTA it's also the most successful racing game. It will also be the most successful flying game and the most successful sailing simulator....etc etc.
I don't know, even though there is shooting in GTA I don't think I'd call it a shooter.
I agree that there is probably some other genre that would describe it better, but I think it wouldn't be unfair by any means to describe GTA as a third-person shooter.
That list is incomplete. CSGO is the most played game on Steam and is now free to play. I would not be surprised in the slightest if more people have CSGO.
interesting. Steamspy no longer works, but its last report in 2016 said it had 25 million sales (at the time it was roughly on par with minecraft sales). Since then, the concurrent player count has more than doubled, but its very difficult to get information about sales.
Wikipedia has it with 46 million owners on Steam, with the following footnote:
> Not official; the numbers were estimated following a Steam API data leak in July 2018, which gave player ownership estimates for games on the platform that have achievements.
Not very surprising. Twitter doesn't work properly on my desktop, google freezes when showing the cookiewall, github freezes my phone. These are all important projects of billion dollar companies.
Maybe long load times are advantageous? Because it creates longer user sessions on average? If you devote 10 minutes to loading the game you will probably want to play for at least 30 minutes.
I am not saying it is the case, nor understand details of solution in depth to comment on it, but in analogy, this reads to me, like an yelling at person who figure out how to solve rubix cube puzzle steps, because once steps are known solution is simple.
No, other people have pointed this out, this should have been very easy to recognize as inefficient in the source code. More likely the code was written hastily and only tested against very small inputs, and then nobody ever actually tried to improve the famously long load times.
> This online gamemode alone made $1 billion in 2017 alone.
which of course goes to show that at least from a business side, this issue is completely inconsequential and all resources should be used to push for more monetization (and thus adding to the problem by adding more items to the JSON file) rather than fixing this issue, because, clearly, people don't seem to mind 6 minutes loading time.
I'm being snarky here, yes, but honestly: once you make $1 billion per year with that issue present, do you really think this issue matters at all in reality? Do you think they could make $1+n billion a year with this fixed?
The bigger the scale the bigger a few percentage point improvement would be worth. I would generally think if you're at 1bn in revenue you should devote 1%+ percentage points of your workforce towards finding low hanging fruit like this. If 1% of employees deployed to find issues that, when fixed, yield 2% improvement in revenue thats likely a winning scenario
This is losing them money. If they fixed the issue they absolutely would get $1+n billion instead of just $1 billion and that n alone is big enough to pay multiple years worth of 6 digit salaries just to fix this single bug.
I work in a large multi billion company and we have people staring at a slow problem for a decade before a noob comes with a profiler and find they browse every key of a Map instead of calling get and such. Or do 1 million db queries on a GUI startup...
Not surprised they didn't bother for 6 minutes when it takes us 10 years to fix a 30minutes locked startup.
> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years
Agree. I found the slow behavior of sscanf while writing one of my first C programs during an internship^^ You literally just have to google "scanf slow" and find lots of information.
It could be that at the time GTA online first publishes, the list as hashmap isn't too much of an issue, due to limited catalog, but get progressively worse as the inventory grows.
Ofc this is just a hypothesis, but I see the hesitation to change legacy code if it ain't broken as a wide spread mentality.
What is complicated about it is that an online modern 3d game is huge, and there are 95,000 places where a dumb mistake could hurt performance a lot for some customers. You catch 94,999 of them and then "unforgiveable"
If it was that way for a few months and then fixed... still pretty shoddy but sure. However, it has been that way for YEARS, and is one of the most common complaints among players. I wonder how much of the remaining loading time could actually be shaved off if someone with the source code took a crack at it.
> It is absolutely unbelievable (and unforgivable) that a cash cow such as GTA V has a problem like this present for over 6 years and it turns out to be something so absolutely simple.
It is both believable and - by virtue of the fact that, as you said, the series continues to be a cash cow - is apparently forgivable.
Here's the thing: the company has zero reasons to fix this, or other ostensibly egregious affronts like DRM, because gamers keep buying the product. There is literally no economic incentive to 'fix' it.
> How many players have they lost to excessive load times?
Judging by the number of successful sequels the franchise has spawned, the answer is 'an insignificant number'.
The fact of the matter, and the point my comment was trying to make, is that the overwhelming majority of players do not sufficiently care about load times or other complaints to deter them from paying for the game. That is the reality of the situation.
That doesn't make sense, saying that the game still managed to be profitable doesn't show that the loss was insignificant. Those loading times are really annoying and detract from the game, and although I can't quantify the loss either I'd be very surprised if it was insignificant.
I'd say especially since the ones who are most likely to be affected by these issues are working adults with limited playtime who won't want to sit in front of their monitor for 5 minutes waiting for the game to load and also happen to be people with disposable income to pour into a game.
I've lost interest in a lot of online games because the clients always do their mandatory updates on launch, which is exactly the time you want to play the game.
(Same thing for websites - they show you all the annoying popup signup sheets/survey questions the instant you load the page.)
For sure. I fit my gaming into about 2 hours a week. If it takes 6 minutes to load there's no way I'm going to play it. I know I'm not the target market but I'm still a missed opportunity.
Attitudes like yours are why gamedevs keep to themselves.
"Unbelievable" and "unforgivable" eh? It's a greedy attitude. Instead of viewing GTA5 as a success that's brought a lot of people happiness, you view it as a money cow designed to extract every last bit of profit – and time, since this bug caused 70% longer loading times.
Perhaps it's both. But you, sitting here behind a keyboard with (correct me if I'm wrong) no gamedev experience, have no idea what it's like on a triple-A gamedev team with various priorities. The fact that the game works at all is a minor miracle, given the sheer complexity of the entire codebase.
The fact that someone was able to optimize the obfuscated executable is a wonderful thing. But they weren't a part of the team that shipped GTA 5. If they were, they certainly wouldn't have been able to spend their time on this.
This kind of excuse making is one of the reasons I got out of software development. It’s not just gamedev. Priorities are way out of wack when you have time to put in binary obfuscation, but no time to fix such a huge performance bottleneck. The idea that “it’s a miracle software works at all” demonstrates the chronic prioritization and project management competence problem in the industry.
It’s ok to recognize a thing as a business success but a technical failure. In fact many software projects are business successes despite awful and unforgivable quality compromises. You don’t get to whitewash it just because the thing prints money.
Agree that this kind of incompetence exists in all sectors and I think we don't talk about it, it becomes acceptable. We're not trying to blame a single developer, that'd be inappropriate. But, the management and QA culture in a AAA game studio that rakes billions ought to be better.
The complexity is in reverse engineering the binary. The developer has access to the full source code and the profiling tools I presume.
Another one is in Microsoft Flight Simulator, instead of downloading multiple archives, it downloads one, unzips it using a single CPU core and then downloads another one. MSFS 2020 takes a few hours to install and that's not just because of the internet connection, but this shitty installation code.
Building the engine alone takes 30 minutes after each change. You're not going to get anything done in 30 minutes. And the more you work on this, the less you're working on shippable features that make money.
In the mobile space, an app opening faster is so valuable you can make a career out of just that. I haven't worked desktop/console games, but having load times be 6 minutes longer than it needs to be, when you're trying to make money on ongoing microtransactions, has got to be losing you so much more money than the time spent fixing.
That's a fine argument, and I'm sure that if management knew they could get a 70% decrease in load times in exchange for focusing on this one area, they would have done so. But nobody knew. And discovering that would have been expensive.
I'll meet you halfway though: they should have had profiling sessions that pointed to the JSON parsing code as the issue. I imagine that all of their profiling efforts were focused on the runtime performance, not the load time. Simply put, no one did that profiling, and I don't fault them for focusing on runtime performance (which is where the real money is, as Cyberpunk 2077 demonstrated by not having it, and subsequently having their PS4 orders yanked and refunded).
No, Sony pulled Cyberpunk 2077 after CD Project promised refunds and then forced Sony to do it on their end. Sony didnt like that.
Once again - Control by Remedy ran on base PS4 at 10 frames per second, TEN frames. Critically acclaimed, reviewers loved it and didnt tend to mention TEN frames per second on base consoles, not pulled from the store.
This feature would bring in millions of dollars on its own. When your customer base is earning you $1 billion per year a 1% improvement is $10 million per year. You could hire an entire team of 100 engineers for that salary just to fix this single bug.
> Which features would you cut out of the game in order to have fast loading times?
If this is a serious question, I'd say cut any of the new vehicles introduced in the last 2 years. None of them are nearly as impactful as this optimization. In fact, I am having issues imagine any individual feature at all that's as important as this fix.
I’d have to have been there, seen the list of features with eng estimates and trade offs, but yes I would have happily made the call to chuck one of them if it meant a measurably higher quality product, like this massive load time improvement. Hell, that zoom-out-zoom-in animation when you switch characters probably took as much time to code as it would have to fix this bug. I think anyone with good product sense and the balls to make a priority call that might get someone upset could make the right call here.
But after it's already made record-breaking profits and is a huge cash cow with recurring revenue. You could just say "hey, I'll hire one single contract developer to do these kinds of quality of life things" and make a fraction less profit.
Unfortunately this part often kills quality initiatives. Why fix bugs for your existing customers when you can deploy the engineering resources on a DLC or a sequel which will milk those customers for more? There is no more craftsmanship or pride in good work left in software.
"When you're a carpenter making a beautiful chest of drawers, you're not going to use a piece of plywood on the back, even though it faces the wall and nobody will see it. You'll know it's there, so you're going to use a beautiful piece of wood on the back. For you to sleep well at night, the aesthetic, the quality, has to be carried all the way through." -Steve Jobs
GTA-5 broke even within 48 hours of it's release. Nearly a decade later, it still costs $60 for a digital copy with (practically) zero distribution costs. It has made over $6Bn in revenue, and is said to be the most profitable entertainment product of all time.
How much would it have cost to fix this issue?
Is anyone saying that it is a game developers fault? I mean, what is that you think would prevent a game developer from fixing this?
Because I think, anyone even vaguely familiar with the software industry in general is going to come up with answers like:
1. It would not cost very much
2. No it isn't a developers fault, because it's clear that even an intern could fix this
3. Management isn't interested, or is too disorganized, or focussed on cynical extraction of every last bit of profit.
And from that perspective, it certainly does make it seem like a cynical cash cow.
I don't know many game developers, but I do know people in other parts of the software industry and professionals in general. And I think that they keep to themselves because they have first hand experience of how the industry works and understand it better than anyone. The probably sympathise with the right of the public to feel ripped off.
That said, I still paid for the game, I think it's fun. Apparently there is "no alternative" to this state of affairs.
> GTA-5 broke even within 48 hours of it's release. Nearly a decade later, it still costs $60 for a digital copy with (practically) zero distribution costs
Well, that's the nominal full retail price against which the various discounts are measured, sure, but I doubt that's what most people buying it these days pay except if they are getting something else with it. I'm pretty sure it's in the stack of things I've gotten free this year on Epic that's in my “I might check it out sometime” queue, it's $29.98 right now from Humble Bundle, etc.
I don’t think the OP was specifically calling out any game devs. Any engineer who has worked on any software projects knows that you usually can only do as well as your deadlines and managements priorities allow for.. Unless the stars line up and you have the resources and autonomy to fix the issue yourself, and love working for free on your own time.
Tweaking two functions to go from a load time of 6 minutes to less than two minutes is something any developer worth their salt should be able to do in a codebase like this equipped with a good profiler.
But, I fully agree with your assessment, for what it's worth.
I read that to mean "If a manager would say to one of his developers, who are probably all worth their salt, take a profiler and go figure out why our load times suck, then this would be fixed."
But a manager shouldn't even have to do that, because in a well functioning team, if the dev leads come back with such a fix, they won't get punished for going off the reservation and would probably be doing this of their own initiative.
But if things like that get met with "why have you wasted time on this? we gave you the list of priorities and it does not include load times" then dev leads will make sure all developers time is filled with things which are prioritized.
I did mean to reply to you (72 days ago plus in another thread) I just can't make a Twitter, like I've tried and doesn't like my IP or email addresses or something, not sure
Hmm, that's unfortunate. Well, I hope you find a way to get on Twitter. Your thoughts are always welcome, and there are a lot of interesting people in the ML scene there.
Parsing JSON?! I thought it was some network game logic finding session magic. If this is true that's the biggest WTF I saw in the last few years and we've just finished 2020.
Stunning work just having binary at hand. But how could R* not do this? GTAV is so full of great engineering. But if it was a CPU bottleneck then who works there that wouldn't just be irked to try to nail it? I mean it seems like a natural thing to try to understand what's going on inside when time is much higher than expected even in the case where performance is not crucial. It was crucial here. Almost directly translates to profits. Unbelievable.