Multiple breaking changes are coming for components in extensions

Are you an extension or application developer? Have you written any XPCOM components, JS, binary or otherwise? If not you can probably ignore the rest of this post, unless you are interested anyway.

If you do then you might be interested to hear that your components are probably going to break in an upcoming Firefox nightly, maybe as early as next week. I’m going to blog specific examples on the changes you need to make once we have better documentation up and builds to test against, for now it is just important for you to know that the changes are coming.

Changing XPCOM component registration

One of the things we are doing is changing how XPCOM components are registered. Previously during registration we would load all binary and JS component files and call them asking them to register themselves. If you used XPCOMUtils.jsm then some of that ugliness will have been hidden from you. The future though will be registration using manifest files, similar to how chrome is registered (in the same files in fact). Every single JS or binary component will need fixes to support this. The amount of work to do is fairly trivial per component and it is possible to support both old and new styles of registration in the same component but until that work is done your components will not work.

Starting extension components later

Currently there are a bunch of different notifications that components can listen for to perform actions during startup. Commonly xpcom-startup, app-startup and profile-after-change have been the most common. The latter is generally the best to use since this is one of the earliest notifications that occurs after the profile folder (and hence preferences and other services) is available.

The news is that once we make our changes components from extensions will no longer be able to register for and receive the xpcom-startup or app-startup notifications at all. profile-after-change will be the first notification that components can be loaded by.

Most of the reasons for loading on app-startup were either because developers didn’t know about profile-after-change or at one point it used to be necessary to load for app-startup just to be able to register for profile-after-change in the observer service. That is no longer the case as you can register for it in the category manager exactly as you can for app-startup.

Why?

You might well ask why we are making all these changes.

Currently whenever we detect a potential change in the set of usable components (meaning either the application version has changed or an extension has been added/removed/enabled/disabled) we must throw away existing component registrations and then perform what has become known as the EM restart where the application is restarted during its startup (in theory invisibly to the user) to make sure that any component that weren’t meant to be loaded are unloaded and to re-register everything and load any new components that need it.

Registration is itself a costly process since every component must be loaded and executed when really many if not most of them do not need to be loaded during startup. In the E10S world there are even more problems. Content processes must either register all components during every startup or somehow share a component cache with the chrome process.

The new registration model allows us to do away with the EM restart. On startup rather than relying on a potentially stale component cache we will read the registrations for the main application out of a manifest file. This will give us enough of XPCOM to then be able to load the extension manager and do any necessary work to install/uninstall/update any extensions after which we can simply load the component registrations for those out of the manifest file. Startup then just proceeds as normal, no restart is necessary since nothing from extensions can have been loaded when it shouldn’t.

E10S content processes can just read the component registrations during startup, since they are in simple manifest files this will go a lot faster

Documenting the new Add-ons Manager

I’ve spent some time this week transferring all the API documentation for the new add-ons manager from the Mozilla wiki to the Mozilla Developer Network. This should now be the place to go for the definitive info.

Right now it is pretty dry, for the most part just pure API info with no examples. Before I started working more on that side of things I wanted to ask what kind of examples people might like to see documented?

Support for dropping XPI files into the extension install locations might be going away

For some time now Firefox has supported a way of installing extensions that involves simply copying the extension’s XPI file into one of the extension install locations. The next time Firefox runs it would pop up the install dialog for the extension and allow the user to choose whether to install it or not.

I don’t know how many people use this feature and while the code to do it (at least for the profile folder) isn’t terribly complex, it is additional code that may not be necessary. Right now the new add-ons manager doesn’t support it and I’ve heard only a couple of people comment on its absence but nightly testers are by no means representational so I’m asking a little more widely whether people have a real need for keeping this working in Firefox 4?

To be clear we aren’t talking about the method of installing where you extract your extension into a directory in the install locations, nor are we talking about the method where you create a text file in the install location containing the path to your extension.

Where is the updated Nightly Tester Tools?

Many of you nightly testers may have noticed that Nightly Tester Tools’ compatibility override feature doesn’t work with the new add-ons manager and may be wondering when I’m planning to issue an update to fix that. The more astute of you may have noticed that there hasn’t actually been a real code update to Nightly Tester Tools in 2 years, barring a couple of simple app compatibility fixes. Those with a sharp memory will remember that I said just under 2 years ago that I was ceasing work on my extensions in my spare time. I suggested that Nightly Tester Tools might still receive the odd update but obviously that hasn’t happened and the truth is that I can’t see it happening anytime soon. I’m too busy with that whole real life thing to even be able to work on projects I do enjoy, let alone maintain old stuff that no longer really interests me.

This unfortunately leaves a sizable number of users losing a feature that they liked and still potentially have a need for. I can only really see a couple of possible roads to follow from here:

  1. Do nothing. Users will be annoyed for a time but eventually find ways around what they needed NTT for.
  2. Find someone else to pick up and maintain NTT. I’ve had numerous requests for the source code for many of my extensions over the past two years, none have ever apparently tried to do anything though. Perhaps someone out there will pick up the torch this time?
  3. Point all the users to something else, like the Add-on Compatibility Reporter (once that is updated to work on trunk). While nothing else I know of works quite like NTT at least something is better than nothing, and ACR has the benefit of being Mozilla supported, provides Mozilla with valuable information about add-on compatibility and may be rolled into Firefox at some point.

Option 3 is the only one available that involves any work on my part but probably the choice that leads to less user annoyance, unless someone reading this wants to take up the challenge or has a better idea?

Update: Part of the Mozilla QA team are going to take over development and maintenance of Nightly Tester Tools, let them know what you want to see!

The new add-ons manager is here

Finally, after far too much time, the new add-ons manager is about to land in trunk nightlies. I am putting together the final patches to land now. The bit most people will see is the new UI so I guess I’ll steal Boriss’ image for you to look at here with the same caveats. What you see on trunk over the next few days is just the initial steps to switching to a redesigned UI and (more importantly from my point of view) a totally new extension manager backend that will make it easier for us to improve and build upon in the future. The changes are so large that it is important to get more people testing it now while it still looks fairly unpolished so we can pick up problems that we’ve missed.

New Add-ons Manager UI

Tomorrow (Friday) the QA team are holding a test day on the new add-ons manager, if you want to help test for issues you can join us in #testday on irc.mozilla.org

We are filing bugs with [rewrite] in the status whiteboard and blocking either the backend tracking bug or the UI tracking bug. So you can check the dependency tree to see what issues we already know about and plan to take care of. Please file bugs for anything else you see, especially when it is something that is wrong compared to Firefox 3.6.

There will be some add-ons broken in the new builds. This is fairly normal for any API update. As a general rule if some add-ons work but others don’t then contacting the authors of those broken add-ons is the best idea.

Developers of other applications may find things broken, ultimately this is unavoidable when changing APIs on this scale, please contact me if you need some help getting things working again.

How do restartless add-ons work?

I blogged a short time ago about how we’re adding support for a new form of add-on to Firefox that can install and uninstall without needing to restart the application. Since then I’ve been finalizing a specification for how the platform will load these add-ons, trying to keep it simple but still give developers everything they commonly need. The planned specification is now available and if developers have comments then I’d like to hear them. Currently there isn’t a version of Firefox that implements it but that should change in the next day or so when I make the changes to the add-ons manager project branch and very soon when it all lands on trunk.

My hope is that once on trunk this spec won’t change but obviously this is quite new so we may see changes for a short time if add-on developers come across problems.

How we’re breaking some extensions in the near future

You may have read some reports about how we’re re-implementing the bulk of the extension manager in Firefox. It’s been a long running project (something like a year since I first really started planning how to do it). Things are finally started to come together and all being well we are likely to look at landing the first pieces of this on the trunk nightlies in as little as a weeks time. I’ll be up front, this isn’t going to be a perfect landing. There may be some thing that are missing and other bits where the user experience isn’t as perfect as it will be finally. Of course there may also be bugs we have to rush to fix. Despite all this we feel that we’re about at the point where exposing it to the hands of thousands of nightly testers is the best way forward. Your eyes spot things that we miss, even things that may seem obvious to you and you’re vital to us getting these sorts of features polished and really just how they should be before they get released to the world at large in a Firefox release.

One thing I wanted to mention is how this will affect the extensions themselves. In many cases as you might hope there will be no change at all, the extensions will continue to work as they did on previous trunk builds. Those that do anything to the add-ons manager UI will see obvious problems since that UI has changed considerably. There is also a set of extensions that are likely to see issues because of API changes. This is pretty much normal for any feature but I thought I’d mention two common uses that extension developers are going to have to change in their code.

Accessing file and version information for extensions

The extension manager interface as it used to exist is gone with the new code. Some extensions would have used this to access their extension’s version and packaged files. There is a replacement so it mostly just means changing code to something similar to this:

Components.utils.import("resource://gre/modules/AddonManager.jsm");

AddonManager.getAddonByID("my-addon@foo.com", function(addon) {
  alert("My extension's version is " + addon.version);
  alert("Did I remember to include that file.txt file in my XPI? " +
        addon.hasResource("file.txt") ? "YES!" : "No :(");
  alert("Let's pretend I did, it's available from the URL " + addon.getResourceURL("file.txt"));
});

Hopefully to developers who used the old APIs this will look a lot nicer than it used to, I certainly prefer it. There are a couple of things to note. The old API would give you an nsIFile directly to play with. The new API gives you a URL. It is currently possible to get an nsIFile from this (as it is a file:// url), however in the future we may be no longer extracting the files out of an XPI for an extension so it is worth trying to just use the URL where you can. We might add a simple method for getting say an input stream to the files as well.

The second thing to note is that this API is asynchronous. As performance becomes more and more of an issue for the platform we have to start moving APIs to be asynchronous to allow file accesses to happen on background threads leaving the UI responsive while we wait for data. In most cases this probably won’t cause developers much of a problem, unfortunately it is difficult to make this appear to operate synchronously safely. People may know the trick of spinning an event loop while you wait for data, that is kind of unsafe since it can allow unexpected reeentrancy into code that isn’t anticipating it.

This change to an asynchronous API brings up the other main change that I am sad to have to make but cannot see a way around.

FUEL has to change

FUEL was designed to be a simple stable wrapper around the nitty gritty world of parts of XPCOM and XUL. It has I believed served fairly well in that purpose though in retrospect there are some things it might have been nice to have different about it. The problem right now though is that FUEL has wrappers around the old extension manager and they are synchronous. They simply cannot stay and work with the new extension manage, which means we have to change a part of the FUEL API slightly. This is unfortunate but I’ve tried to make the change as low impact as possible, there is in fact just a single change. Previously to get an add-ons version you might have done:

alert(Application.extensions.get("my-addon@foo.com").version);

Now you will have to do:

Application.getExtensions(function(extensions) {
  alert(extensions.get("my-addon@foo.com").version);
});

Again it switches from synchronous to asynchronous, but that is really the only change here. The extensions object passed into the callback function is basically the same as the current Application.extensions object, only a slight difference in that it is a snapshot of the list of extensions at the time of retrieval. That doesn’t make a great deal of difference for normal extensions as they cannot install or uninstall without a restart but it may have an effect if you are looking at the new restartless add-ons, in which case you can just get a new copy of the extensions object.

Look Ma, no restarts!

An extension installed without restarting Firefox

This is a screenshot of some of my latest (and most exciting) work on my project to rewrite the extension manager. I’ve just implemented support for a special kind of extension that can install (and uninstall, and enable, disable, upgrade and anything else you can think of) without the user needing to restart Firefox. This is of course to allow add-ons developed on the Jetpack platform to install without restarts but the feature is going to be available to any extension author, there are just some restrictions to how these extensions work.

The precise spec for this isn’t final yet so I’m not going to show you precise examples but the basics are that unlike normal extensions the platform won’t load anything from the restart-less extensions except for one file. This simple JavaScript file will be called when the extension is loaded or unloaded allowing the extension to either set itself up or tear every trace of it out of the app. This may seem limited but there is actually very few things that an extension can’t do from this (and hopefully we’ll get those fixed), it just involves more manual work. Maybe we’ll simplify some of that in the future, but for now it’ll be pretty raw and ready for use. Of course the Jetpack platform basically does all the loading and unloading work for you…

Simplifying

The big project that I have been working on for quite some time now is a complete change to the architecture of the add-ons manager backend. It’s a big scary prospect since (IMHO) the code is pretty crucial to the success of Firefox and many other Mozilla based applications. Without extensions I don’t think we’d be where we are today, in fact it was because of extensions that I got involved in the project in the first place.

The current incarnation of the extension manager has served us well over the past years but for some time it has been under strain. There are always more features we want to add but certain aspects of its design make it hard to do that. Things like changing from an RDF based persistent storage to anything else are hard because the concepts are so ingrained in the code. It’s now at the point where changing anything (particularly in the startup code) is very difficult since unexpected side effects are almost sure to happen. I tried various approaches to slowly iterating the current code to something better but ultimately changing anything required changing everything so I finally bit the bullet and concluded that a rewrite was the way forward (I know, normally they aren’t the right choice but I’m, pretty confident that it is in this case).

It is that rewrite that I’ve been working on on and off for something like half a year, if not longer. Some of the nice benefits that we should see:

  • Switching from RDF to a SQLite storage model (and making it easier to switch in the future)
  • Startup and other performance improvements (probably not immediately though)
  • Less bothersome dialogs for the user
  • Vastly simpler APIs for application/extension developers including:
    • Proper separation of the backend and UI code
    • Support for pluggable add-on types

I didn’t actually mean for this post to be this long actually (after all it is about simplification), it was just this last point that I felt like demonstrating. I just wrote the basic part of the code that implements the automatic daily background update checks. It uses the new API that will be available and as well as checking for updates it also downloads and installs any it finds silently (currently, I’ll be adding the notification and controls etc. soon). It struck me just how simple it was to write this:

AddonManager.getAddonsByType(null, function(addons) {
  addons.forEach(function(addon) {
    if (addon.permissions & AddonManager.PERM_CANUPGRADE) {
      addon.findUpdates({
        onUpdateAvailable: function(addon, install) {
          install.install();
        }
      }, AddonManager.UPDATE_WHEN_PERIODIC_UPDATE);
    }
  });
});

That is some 11 lines of code to perform an update check to the add-on’s update server, retrieve information about the newest available update and start downloading and installing it.