Keeping Up

How Chrome stays fresh without getting in your face, and why its on-disk structure is a little unconventional

Google’s philosophy on software updates is simple: nobody should ever be using an outdated version of an application. On the web, this is easy to accomplish. When a new version of the software that powers a web site is ready, it replaces the old. The original Google Search page prototype is recognizable as an ancestor of today’s equivalent, but it looks different and acts differently. There isn’t a single person in the world today using that 1998 version with its measly 25 million pages, many of which are probably long-gone by now. At any point in time, everyone searching Google is as up-to-date as possible. As the web changes, as features are added, and as the design is tweaked, the world will always be using the current version of Google Search.

What’s Chrome got to do with it?

The same principles that apply to keeping Google Search users current can apply to users of client software. In this context, “client software” simply refers to applications running on your own computer. As with Google Search, the thought is that as client software like Chrome changes, features are added, and its design is tweaked, the world should always be using the current version. Everyone using Chrome should be as up-to-date as possible.

It’s easy to keep Google Search users updated, because all of the search software Google writes runs on Google’s own computers, which are under Google’s control. Keeping Chrome updated is a different story, because the software Google writes runs on your computer, which is under your control. Whenever a piece of software’s author is different from its user, a conflict arises. In this case, the author wants the users to be on the latest approved version, but the user might not want to dedicate time to installing an update, or might not even be aware that a new version is available.

The desire to keep users current is more significant than just keeping everyone on the “latest and greatest” version. As Chrome evolves, it receives fixes for bugs, its speed improves, and it sees new features added. Some bugs might affect stability by causing Chrome to crash or behave erratically. Others might even impact your computer’s security or your own privacy. It’s an absolute priority to deploy fixes for these bug classes. It’s far less onerous to provide support for recent versions than for an arsenal of obsolete versions.

What’s bad for the goose is bad for the gander

The traditional approach to keeping client software updated is to offer new versions to users as they become available. Thanks to the Internet’s ubiquity, this is now handled almost exclusively online. At a specified interval, the software might check to see whether it’s up to date. If it’s not, then it’ll offer its user the option to perform an update. If the user agrees, then the update process typically involves downloading the new version of the software, after which it will be installed, replacing the now-outdated version. Generally, the installation step can’t proceed while the outdated version is running, so the user is usually prompted to quit the program while the update is applied. Whether they actually need to or not, some updates even want the entire computer to restart when they’re done installing.

From the standpoint of keeping all users up to date all the time, the big problem with this traditional approach is the word “if.” What happens if the user doesn’t agree to perform the update? Well, nothing. No update is downloaded, no update is installed, and the user continues using the outdated version. I can’t even blame people for not wanting to update. Most users probably just want to get on with their lives, and having to take time out of their lives to quit a program they were in the middle of using and maybe even restart their computer isn’t a very attractive proposition. As a result, they keep on going with their old versions, and they’re periodically irritated when some box pops up to ask them the same question about updating that they’ve already said “no” to countless times.

This traditional approach is bad for me as the author and it’s bad for you as the user. There really shouldn’t be so much tension between us. We figured out a way to improve upon the status quo.

Silence!

The traditional approach can be simplified by removing the “if” from the equation. By never asking the user any questions like “wanna update now?” and instead always assuming that it’s a good idea to update, the software can stay out of the user’s face, and ensure that the latest version is always present. In a sense, this approach is even easier than asking the questions, because nobody even needs to write the code responsible for asking the user.

This is the first thing that I did when implementing Mac Chrome’s updater. It turned out to be a huge mistake.

Recall that in order to apply an update, the new version needs to be downloaded and then installed, replacing the old version. What happens if the old version is already running? The new version sort of collides with it in unexpected and interesting ways. Interesting forensically, that is. It’s never interesting in a way that I’d want a user to experience.

In Mac Chrome’s case, quietly installing an update in the background while the program was running interacted very poorly with Chrome’s multi-process architecture. Chrome’s IPC ping-pong game relies on the browser, renderer, and other processes being able to communicate with one another. Since they’re all part of the same application, they expect to be able to communicate in a common language, and this language is specific to each version of the application. But if you’re running browser version 4, and behind your back, Chrome is updated to version 5, the next time the browser tries to start a renderer, it’ll start a version 5 renderer. Renderer 5 can’t figure out what Browser 4 wants from it, and Browser 4 has no way to start up Renderer 4, so you’re stuck running version of Chrome that really can’t do anything at all. The only workaround is to quit Chrome and restart it, by which I mean quit Browser 4 and launch Browser 5, which does know how to talk to Renderer 5.

Since the update was performed silently, in the background, and without any way to prevent it, this system just interrupted you without any warning, and didn’t even provide a clear indication that you could recover by restarting the application. This is certainly worse than the problem that we were trying to solve.

Traditional update systems get around this problem by asking the user to quit the program in order to apply the update, but I’m imposing a design constraint: I don’t ever want to ask the user anything. Some update systems download the update and then wait to install it until the user quits the program or next tries to start it, but this approach means that the user winds up having to wait for the update to be installed, and I don’t ever want anyone to have to wait for me.

Peace and Quiet

Ultimately, the solution to the update problem was simple, if not unconventional. Instead of replacing the old version of the application with the new one, the new version is installed alongside the old. If you happened to be running the old version when the new one was installed, it would be able to continue running without experiencing any interruptions. If it was Browser 4, then when it needed a renderer, it would still get Renderer 4, and the two would be able to conduct intelligent discourse. A subsequent launch of Chrome would get you version 5.

I was able to make this work by leveraging the fact that Mac applications are really bundles. A application bundle is a directory that can contain everything it needs to function, and it’s represented to users as a single self-contained icon. All I had to do was take the framework, which is where all of the interesting parts of the program live anyway, as I described in a previous article, and put it into what I call the “versioned directory.” If you poke around inside the innards of the Chrome application bundle on the Mac, you’ll find this “versioned directory” inside Contents/Versions. Each new version of Chrome gets its own versioned directory.

When you start Chrome, dyld, the Mac OS X loader, sees that it also needs to load the framework. The main executable program specifies the framework to the loader by its precise location within the versioned directory. In the this example, dyld will look for the framework in Contents/Versions/12.0.712.0:

mark@rj bash$ otool -L 'Google Chrome.app/Contents/MacOS/Google Chrome'
Google Chrome.app/Contents/MacOS/Google Chrome:
        @executable_path/../Versions/12.0.712.0/Google Chrome Framework.framework/Google Chrome Framework (compatibility version 712.0.0, current version 712.0.0)
        /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.4.0)
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4)

Evidently, I’ve got version 12.0.712.0 installed, and a quick check of the About box confirms that this is the version I’m actually running. Now take a look at the versioned directories:

mark@rj bash$ ls -l 'Google Chrome.app/Contents/Versions'
total 0
drwxr-xr-x  4 root  wheel  136 Mar 18 23:02 11.0.696.16
drwxr-xr-x  4 root  wheel  136 Mar 23 06:05 12.0.712.0

I’ve got versions 12.0.712.0 and 11.0.696.16 installed. I’m running 12.0.712.0 and am not using the older version at all any more, but when the update to 12.0.712.0 occurred, I had been actively using 11.0.696.16, so the older versioned directory couldn’t have been removed at that time.

Cleaning Up

The update procedure itself is responsible for removing old versions, but it’s very careful not to remove any versioned directories that look like they might be in use when an update is applied. As an extra measure of safety, the updater will always save the versioned directory containing the version being updated from. This means that once Mac Chrome has updated itself, it’ll always contain at least two copies of the entire program. This keeps the old version usable for as long as you continue to run it, but as soon as you quit the old version, there’s no longer any way to start it, because the browser application will then load the updated framework. The old version will stick around until it can be cleaned up during a subsequent update.

This system works beautifully. Users always have the latest approved version of Chrome available to them, but aren’t ever bothered by questions that interfere with their work, or play, or both. The only drawback is that the old versioned directory sticks around after it ceases to be useful, but you’d probably only notice this if you compared the size of a freshly-installed copy of Chrome to one that had been updated. Given the massive amount of storage available on contemporary computers, carrying a little extra heft inside of the Chrome application is a small price to pay in exchange for everything working so smoothly.

It might seem smart to let Chrome clean up after itself and remove old versioned directories when it launches, or at some other point, rather than having to wait for another update to run. In reality, this wouldn’t be a robust or reliable solution. Chrome, or the user running it, might not actually have permission to make changes within the application bundle. In contrast, during a successful update, the updater has already proven that it has the requisite level of access, and is in the best position to attempt removal of the old versioned directory.

Stragglers

In the end, I was able to get nearly everything stashed away neatly within the versioned directory, but there are a few files that Mac OS X insists live at specific locations within an application’s bundle. Fortunately, none of these files have a significant impact on Chrome once it’s running.

Poking around in Chrome’s Contents/Resources directory will reveal these stragglers. As of the current version of Chrome, it includes a couple of icons (.icns files), files supporting scripting and managed preferences, and copyright messages translated into 52 languages (all of those .lprojs). One level up, in the Contents directory, there’s an Info.plist containing general information about the program, and the application’s code signature.

Sign me up

Astute readers might wonder how Chrome’s update scheme works with code signing. Every copy of Chrome that ever leaves Google is “signed” before making its way to a user’s computer. The signature assures that every file contained within the application is present and in the same condition that it was in when the program was built, and that nothing’s been added, removed, or changed. It’s a way to assure that nobody’s tampered with the application, and that you’re running the real deal.

Whenever we release a new version of Chrome, we sign it in such a way that only the contents of that version’s versioned directory are considered for the signature. That way, even if one or more older versioned directories are present on your computer, the signature can still check out as valid, as long as nothing else has been touched.

Symbolic Links

While I was designing this scheme, I had initially assumed that I’d be able to use symbolic links to handle those files that, for one reason or another, couldn’t live within the versioned directory. For example, if I were to replace fr.lproj with a symbolic link referring to something within the versioned directory, I’d be able to keep the French copyright message, «Tous droits réservés.», inside the versioned directory, too. As it happens, a bug in Apple’s code signing system causes it to completely ignore symbolic links. In this scenario, someone would be able to point the fr.lproj symbolic link elsewhere, perhaps changing the message to «Voulez-vous coucher avec moi ce soir?», and the modification would be undetectable to the code signing system. An awkward invitation like this one is a tame example, but things could certainly get much worse from there.

As a result, I didn’t pursue this design any further, and I created a new policy: no more symbolic links in the application.

Ironically, the code signing procedure itself creates some symbolic links, although they don’t negatively impact its own operation. It’s amusing that the code signing process creates these symbolic links when it has such a hard time coping with those established by others. Go figure.

Reminder!

Chrome’s silent auto-update provides a way to get new versions onto your hard drive, but it doesn’t do anything to make sure that you stop using an obsolete version and switch to a newer one. Without quitting Chrome, a user might obliviously go on using an obsolete copy even after one or more updates are installed. Chrome doesn’t crash nearly frequently enough to force users to restart it (or to find a more stable web browser). As a form of gentle encouragement, Chrome’s got an upgrade detector that puts a little badge on the wrench menu’s icon a while after an update is installed. If you notice the badge and open the wrench menu, you’ll be encouraged to “update Google Chrome.” Because the update has already been installed by the time the badge shows up, this menu item just quits and restarts Chrome, so what’s perceived as an “update” is actually incredibly rapid. Thanks to the session-restore feature, the new version of Chrome will start up and open all of the same tabs you had open in the old version.