Oxymoronical

This is a screenshot of some of my latest (and most exciting) work on my project to rewrite the extension manager. I’ve just implemented support for a special kind of extension that can install (and uninstall, and enable, disable, upgrade and anything else you can think of) without the user needing to restart Firefox. This is of course to allow add-ons developed on the Jetpack platform to install without restarts but the feature is going to be available to any extension author, there are just some restrictions to how these extensions work.

The precise spec for this isn’t final yet so I’m not going to show you precise examples but the basics are that unlike normal extensions the platform won’t load anything from the restart-less extensions except for one file. This simple JavaScript file will be called when the extension is loaded or unloaded allowing the extension to either set itself up or tear every trace of it out of the app. This may seem limited but there is actually very few things that an extension can’t do from this (and hopefully we’ll get those fixed), it just involves more manual work. Maybe we’ll simplify some of that in the future, but for now it’ll be pretty raw and ready for use. Of course the Jetpack platform basically does all the loading and unloading work for you…

The big project that I have been working on for quite some time now is a complete change to the architecture of the add-ons manager backend. It’s a big scary prospect since (IMHO) the code is pretty crucial to the success of Firefox and many other Mozilla based applications. Without extensions I don’t think we’d be where we are today, in fact it was because of extensions that I got involved in the project in the first place.

The current incarnation of the extension manager has served us well over the past years but for some time it has been under strain. There are always more features we want to add but certain aspects of its design make it hard to do that. Things like changing from an RDF based persistent storage to anything else are hard because the concepts are so ingrained in the code. It’s now at the point where changing anything (particularly in the startup code) is very difficult since unexpected side effects are almost sure to happen. I tried various approaches to slowly iterating the current code to something better but ultimately changing anything required changing everything so I finally bit the bullet and concluded that a rewrite was the way forward (I know, normally they aren’t the right choice but I’m, pretty confident that it is in this case).

It is that rewrite that I’ve been working on on and off for something like half a year, if not longer. Some of the nice benefits that we should see:

• Switching from RDF to a SQLite storage model (and making it easier to switch in the future)
• Startup and other performance improvements (probably not immediately though)
• Less bothersome dialogs for the user
• Vastly simpler APIs for application/extension developers including:
  • Proper separation of the backend and UI code
  • Support for pluggable add-on types

I didn’t actually mean for this post to be this long actually (after all it is about simplification), it was just this last point that I felt like demonstrating. I just wrote the basic part of the code that implements the automatic daily background update checks. It uses the new API that will be available and as well as checking for updates it also downloads and installs any it finds silently (currently, I’ll be adding the notification and controls etc. soon). It struck me just how simple it was to write this:

AddonManager.getAddonsByType(null, function(addons) {
  addons.forEach(function(addon) {
    if (addon.permissions & AddonManager.PERM_CANUPGRADE) {
      addon.findUpdates({
        onUpdateAvailable: function(addon, install) {
          install.install();
        }
      }, AddonManager.UPDATE_WHEN_PERIODIC_UPDATE);
    }
  });
});

That is some 11 lines of code to perform an update check to the add-on’s update server, retrieve information about the newest available update and start downloading and installing it.

If you are an extension developer and include executable files in your XPI package (binary or shell scripts) then you may be seeing problems in Firefox 3.6.

Back between Firefox 3.6 beta and Firefox 3.6 RC we took a small fix to the extension manager that changed how we extract the files from the XPI package. The fix involved adjusting how we accessed files to avoid hitting problems with certain anti-virus tools that would occasionally lock files in the middle of extraction making us fail to install the add-on. A side effect to this fix leaves us setting file permissions on the extracted files in a slightly different way to previously. This side effect means that the executable permission is getting stripped from all extracted files. If you try to execute these files with nsIProcess it will likely fail.

There is a bug on file and I have a patch almost complete so hopefully this should be fixed in Firefox 3.6.1 but until then you can workaround this in your extension by setting the executable permissions on the file yourself. Assuming you have a file variable that is an nsIFile pointed at the executable, just do:

file.permissions = 0755;

In case you were wondering why we enforce file permissions at all, it is because it turns out there are quite a lot of different zip tools that developers use to build add-ons. Some of them are unfortunately broken and embed bogus permissions into the generated XPI. After extraction it leaves us with files that are unreadable/unwritable which makes the add-on fail to work correctly. This most commonly affects developers on windows (where permissions are a little laxer in general) who get odd bug reports from users on Linux which is respectfully refusing to use the files. The easy fix is to enforce read/write permissions on the extracted file in the first place.

Ever since Firefox 2 we have vaguely supported a form of extension dependency. That is marking an extension as requiring particular versions of another extension.

The support is currently very limited and when a user tries to use an add-on that depends on something they don’t have they are pretty much left in the cold. While we tell them it requires something we don’t tell that what it requires or give them any easy way to download and install it.

Given this poor implementation it is perhaps no surprise that very few extensions make use of the feature (something like 50 listed on AMO, many of which are outdated). This raises questions over the cost of continuing to support this feature particularly now when we are undergoing large scale changes to the add-ons manager architecture to give better support to personas and jetpacks.

Before I made any final decisions I wanted to gather some feedback on what people thought about completely dropping the support for dependencies for now. We will always have the option to add it back again (even in a more functional state) at a later time should we decide it is worthwhile. Unfortunately it’s certainly not on the cards right now to complete the implementation as it stands and even leaving it as it is is costing significant work during the refactoring project.

Extensions can always handle dependencies themselves at runtime by testing for the presence of whatever they depend on. They can even then give the user a way to get what they need which makes for a far better user experience than the platform currently offers so I’m not sure I see a convincing reason to keep the feature as it is around for now.

Back in the mists of time I wrote some code to make nightly testers’ lives easier by giving them a simple preference to flip if they wanted to be able to install and use incompatible extensions. It’s been more than three years since then and the use of this preference has grown beyond its original use. It is now something recommended to regular users everywhere from forums to comments in news articles as a way to use their extensions in the new major Firefox releases.

Don’t get me wrong, letting users upgrade sooner than they otherwise might is a good thing, but the preference is a dangerous beast. It is pretty simple for a user to set the preference and then forget about it leaving them able to install incompatible extensions that break their Firefox without realising it. This costs Mozilla time as well since we get quite a number of bug and crash reports to look at that turn out to be caused by extensions that are dutifully marked incompatible with the user’s Firefox.

We’ve been mulling over ways to change this for a while but now we’ve actually gone and done something about it. We still want nightly testers and early adopters to be able to use incompatible extensions if they need to but we also want to make the preference not be a one shot deal that lasts till the end of time. The plan we’ve come up with is to change the preference’s name with the Firefox version, so for Firefox 3.6 (and all security releases) the preference will be extensions.checkCompatibility.3.6. When switching to a future 3.7 testers and users will have to set a new pref extensions.checkCompatibility.3.7 to say they still accept the risks of running with incompatible stuff.

Nightly users will have to make the changes slightly more often since the preference will also track whether the version is one of the development alphas or betas, so for the 3.6 betas the preference would be extensions.checkCompatibility.3.6b, for the current trunk extensions.checkCompatibility.3.7a. These are just normal preferences of course, if you frequently switch between different Firefox versions you can just set both necessary preferences. The change should land on the trunk in the next day or so and then the 3.6 builds a day or so after that.

There is just one oddity, if you’re testing nightlies and you update to a build with the change then you likely won’t notice any of your extensions disabling themselves, that won’t happen till either you toggle the pref or you switch to a build with a different Firefox version number in it.

Update: For more in-depth information this change was made in bug 521905.

As part of the ongoing work to bring basic support for lightweight themes (based on the ideas from the Personas extension) into Firefox 3.6 I’ve today landed the main UI parts that allow users to see and select between lightweight themes they have used recently. Dão landed most of the backend last week  but we’re still waiting on the support for installing new lightweight themes before this feature will be truly usable in the development builds. For the time being here is a shot of what the UI looks like in the add-ons manager after you have used some lightweight themes.

The idea is that users will install lightweight themes from websites. The add-ons manager will include the most recent lightweight themes used to allow the user to use them again. For this release they are offered as a straight alternative instead of other installed themes. We’re going to replace the default icon there with something more themey.

I have been remiss in not posting a status update about this in two weeks, but that is mostly because we have unfortunately had to slow down work on this feature. The problem is that a string freeze became necessary for all toolkit code (the code shared with the Firefox mobile browser and where this feature would have lived). Unfortunately this all came up over a small period when I was travelling long distances and having to take time out to satisfy immigration authorities that I wasn’t a terrorist intent on bringing down the U.S. government.

In the end there just wasn’t enough time to finalise the UI and be confident that it was correct before the freeze. This means that this feature isn’t going to make Firefox 3.6, however the intention is to continue working on it to get it into Firefox 3.7. It will just be on hold for a short time since I need to devote myself to Firefox 3.6 blockers.

This week has been the Firefox work week where almost all of the team, including me, made our way to Mountain View. This pretty much means that you spend the entire week in meetings since when you’re remote it can be hard to keep in sync on everything. Of course this means that the amount of coding is pretty low for the week so there isn’t a great deal of progress to report here.

Status

This week, Alex and I met to come up with a rough design for what this is going to look like now that we know that the doorhanger notifications we were hoping for aren’t going to make Firefox 3.6.

Whiteboard mockup

Next steps

• Implement the planned design

This is a weekly status update on the feature to notify users about add-ons that third-party installers have added to Firefox. You can read more about it in last weeks blog post or on the project wiki.

Status

There has been little progress this week mostly due to waiting to see whether the doorhanger notifications UI was going to arrive in time for Firefox 3.6. It looks like it isn’t so instead we are discussing using an in-content page to describe the add-ons that have been installed and allowing users to disable them.

Regular readers of my blog will probably spot that this is very similar to an idea I had a couple of months ago. What we’ll probably try to do is integrate both new third-party add-ons and new add-ons installed by the user into the same in-content page. It’s possible that we won’t be able to go the full stretch and allow add-ons to add their own content into this page in the first pass, that will depend on time constraints. I think that even without that this is still a very worthwhile feature.

Next steps

• Develop more detailed UI mockups.
• Implement the in-content page to read the already available data on new add-ons.

I’ve just landed code that allows extensions to include their icon in the add-ons manager view even when disabled. Currently extensions provide their icon by giving a chrome iconURL for us to load. This can only work when the extension is enabled since we don’t register the chrome otherwise. Themes on the other hand provide their icon as a simple file called “icon.png” alongside the install.rdf. Well extensions can do this too now and it will be used in preference to the iconURL but more importantly works at all times (well not before installation yet, but that is at least feasible this way).

We’ll continue to support iconURL for a time to come I imagine, but if you want to keep your icon when users disable you then just put your icon.png alongside the install.rdf (keep the iconURL if you want to work in Firefox 3.5 and lower of course).

This is only in trunk builds right now (today’s Linux nightly has it, tomorrow’s OSX and Windows nightlies will have it). It will be making its way to the Firefox 3.6 nightlies soonish.