The Test Case That Bricked a Wii Test Kit
- Anecdotes, Quality Assurance, Video Games
Back when I was working at Eidos Montréal, part of my responsibilities included ensuring that the games we were producing for the Wii followed Nintendo's Wii Programming Guidelines. This is the story of how I bricked a Wii RVT-R Reader test kit by doing my job.
The issue
The Wii Programming Guidelines were a set of requirements and recommendations to be followed in order to get Nintendo's license for publication, for instance what should happen when ejecting the game's disc or when losing connection to the Internet. Among them were two seemingly redundant statements, which went something like this (paraphrasing, definitively not exact quotes):
Before saving, check if there's enough user space available. If not, show error message X instead.
and
Before saving, check if there's enough system memory available. If not, show error message Y instead.
To clarify, the user space is a portion of system memory in the Wii console reserved to the user for storing their save data and downloadable content. The rest is reserved for the operating system.
Now, you'd think that the physical system memory is completely useless. Given that the user space is a fraction of system memory, only the user space check is required, right? Well, the problem is that the Wii's architecture does not have a physical barrier between the two memory zones as they share the same storage partition, so they can overflow on each other at the file system level. A bug on Nintendo's part, a memory corruption issue or a hacked console could cause such an overflow to occur.
So in order to perform the physical space test on a Wii game, I had to somehow simulate this set of conditions to trigger on a Wii test kit, using Nintendo's tools to do so. However, the only tools given by Nintendo to do so were the Wii test kits themselves, and a program to fill up user space with junk data and calculate the remaining physical and user spaces at the file system level.
The solution
Wii test kits have a really cool feature where you could install or uninstall any official Wii Menu ever released. When no Wii Menu is installed, a debug menu would appear. This debug menu allowed copying unencrypted save data between the Wii console and an SD card.
I figured that maybe if I created save data for a fake game, by giving it a completely made up directory name that shouldn't match any possible Wii game ID, that the debug menu would accept it. I had no idea how the system would react to this however, so I first took some precautions:
ME: Hey boss, I would like to perform a compliance test that requires me to fill up the Wii's system memory. I have no idea if it's gonna work and there's a risk it could cause my test kit to break. Can I get the OK from you before proceeding?
BOSS: Yeah sure, go ahead!
ME: You're sure? I'm serious here...
BOSS: Yeah sure!
ME: OK thanks!
Well that was easier than expected!
I went back to my desk and tried. To my surprise, the debug menu happily accepted my fake game data, and Nintendo's junk data utility reported an increase in physical space only. Success!
So, I filled the entire memory with junk data through the debug menu until the entire Wii system memory was filled, and made sure there was user space available. I then reset the console, booted up the game I was testing (I forgot which one) and tried to save the game.
The game saved successfully.
Wait what?
Going back to the junk data utility, I could see that there was free space available that appeared out of nowhere after my reset. This usually does not happen. I figured the reset probably caused some cache to get cleaned due to the restricted available space, so I deleted the game's save, filled up the system memory again, and tried again.
The game saved successfully. Again.
I deleted the game's save, filled up the system memory again, and reset. Still more free space, so I filled up the system memory again and reset again. I repeated this process about a dozen times, each time with less and less free space appearing, making me believe I was close to succeeding. That is, until after the 14th reset or something, the following message appeared on the screen:
FATAL BOOT ERROR
"Uh oh... let's try resetting again", I thought.
Black screen.
I tried pressing buttons on the controllers to see what would happen, but there was absolutely nothing. No video, no audio, no wireless or Bluetooth connections, nothing. Not even a full power cycle worked. The only obvious indication that the console was actually running was the LED on the front of the console that turned green when pressing the Power button.
So I went back to see my boss with my newly-bricked console:
ME: Hey boss, remember when I said that my test could break my Wii test kit?
BOSS: Yeah?
ME: Well it happened, so...
I gave the test kit to my boss.
ME: ...there ya go. Have fun!
I turned around, returned to my desk, and resumed my work using a spare test kit. Meanwhile, my boss sent the defective unit back to Nintendo for repairs.
I thought it would be the end of the story.
Nintendo's reaction
About two weeks later, we received a phone call from Nintendo's engineers. They were completely stumped, as every single diagnostic check they could run at the hardware level passed, and yet the console refused to boot. They were desperate for answers and wanted more information as to what happened exactly. I'm not sure exactly what my boss said or didn't say to them at the time, but I found it rather funny that Nintendo's own engineers didn't realize it was a memory corruption issue.
In any case, I documented the exact steps I performed, and my boss forwarded the information to them. Apparently they were satisfied with my report and would investigate.
Again, I thought it would be the end of the story.
To my surprise, less than 48 hours after sending my report, I noticed on Nintendo's developer portal (fun fact: said portal was called Wario World) that a new version of the Wii Programming Guidelines had been published. The changes? The removal of the system memory check guideline, and a few fixed typos. That's it.
This means that Nintendo was able to verify my claims, report their findings, take an executive decision on the Wii Programming Guidelines to abolish the guideline that lead me to brick their hardware, create a new revision of the Wii Programming Guidelines and all related documents, make them available in both Japanese and English, and do all of this in less than 48 hours.
I have to say, I've never seen Nintendo react so quickly about anything before or since, except for copyright issues.
The cause
Nintendo's solution was to remove the faulty guideline, which they probably deemed unnecessary themselves, and it was probably the best solution in this particular case. Ideally they should have patched their debug menu too, but I'm not sure if Nintendo had the ability to release such a patch, or if it would have been worth it to do so anyway.
What I find interesting however is that I was apparently the first person to ever try this, even though the Wii had been released for many years before my discovery. I believe the reason for this is because while Nintendo had plenty of test cases documented for their Wii Programming Guidelines, this particular guideline had none, and since QA resources with technical skills are rare, it was not trivial to figure out the test case I came up with.
Also, what I find even more interesting is that this whole situation arose because Nintendo didn't come up with a test case for this in the first place. If they did, they might have prepared proper tools for such a test and designed their console to resist against the scenario which bricks it, or determine that the effort is not worth the risk and save time for everyone.
Given this, I think this is a very interesting example of what can happen when testability is not taken into account during design.
Related articles I wrote
My Personal Video Game Completion List
- Anecdotes, Video Games
I thought it would be fun to track the long list of video game that I have beaten and/or completed for reference, so I've done just that! There may be a few mistakes here and there due to secret features unknown to me, or due to misremembering details of my past gaming experiences, but I believe the…
Current Generative AIs Have Critical Quality Issues
- Business, Quality Assurance, Security
The hype for generative AI is real. It is now possible for anybody to dynamically generate various types of media that are good enough to be mistaken as real, at least at first glance, either for free or at a low cost. In addition, the seemingly-creative solutions they come up with, and the…
After 8 Years, Double Fine's Hack 'n' Slash Secret Room Has Finally Been Cracked
- Video Games, Security
In the history of obscure video game secrets, not many has been quite infamous as the SecretRoom.lua puzzle in 2014's computer hacking game Hack 'n' Slash by Double Fine. Since the game's release, a mysterious encrypted file was found in the game files, yet despite the very nature of the game being…
Upgrading Your Cybersecurity from Cowboys to Sheriffs
- Security, Business, Anecdotes
Roaming throughout the countryside, dangerous desperados are awaiting in their hideout for the perfect opportunity to rob their victims in silence. Powerless, the authorities have posted wanted posters on public boards with cash bounties for any information that could lead to their arrest or death…
Validating and Viewing OpenAPI Definitions with Docker
- Quality Assurance, Programming
Here are a few commands I crafted to validate and easily read API definitions in the OpenAPI format, using Docker and open source tools provided by Swagger. I have yet to convert them into proper shell scripts, but I hope these will be helpful nonetheless. The commands are designed to be run in a…