addons – Page 3 – Oxymoronical

How to extend the new Add-ons Manager (or how I built a simple greasemonkey clone in an evening)

One of the goals of the new add-ons manager API was to create something that was itself extensible. A couple of times in the past we’ve had to add new types of add-ons to the UI like Plugins and Personas. In both cases squeezing them into the UI was something of a kludge involving a bunch of custom code for each case. We already have a number of new types of add-ons that we want to add, things like search plugins which are currently managed by their own custom UI.

To help simplify this the API is largely type-agnostic. Internally it uses a number of so-called add-on “providers”, each of which may make information about add-ons available. Each provider is basically a JavaScript object with functions defined that the API can call to ask for information about add-ons that the provider knows about. The provider then just has to pass back JavaScript objects to represent each add-on. Each of these must have at minimum a set of required properties and functions and may also include a set of optional properties. The full set is defined in the API documentation.

With this design the user interface doesn’t need to care about implementation details of any of the providers, how they store their data or what exactly their add-ons are and do. Because each gives objects that obeys the same interface it can just display and manipulate them.

To try to show this all off I recently put together a small demo extension for the Mozilla Summit that registers a new type of add-on to be displayed in the main add-ons manager. This is a short overview of some of the highlights and I’ll make the code available for people to look at and take examples from. The add-on was a basic implementation of Greasemonkey allowing user scripts to be installed, managed through the add-ons manager and do it all as a restartless add-on.

Making a restartless add-on

Add-ons don’t have to be developed with Jetpack to make them restartless, although the Jetpack SDK certainly makes things easier on you, at the expense of less access to the internals of the platform.

The first thing to learn about making a restartless add-on is that you can forget about using XUL overlays or registering XPCOM components to be called at startup. Neither are supported at the moment, and maybe never will. Instead you have to provide a bootstrap script. This is a simple “bootstrap.js” file in the root of the extension that should include a “startup” and “shutdown” function. These are called whenever Firefox wants to start or stop your add-on either because the application is starting up or shutting down or the add-on is being enabled or disabled. You can also provide “install” and “uninstall” methods to be notified of those cases but that is probably unnecessary in most cases.

At startup the demo extension does some basic things. It registers for some observer notifications, registers a new add-on provider (I’ll talk more about that below) and does a little work to include itself in the add-ons manager UI (again, see below).

The rule is this. Anything your add-on does after being started must be undone by the shutdown function. The shutdown function often ends up being the inverse of startup, here it removes observer notification registrations, unregisters the add-on provider and removes itself from the UI. It also shuts down a database if it was opened.

Implementing a new provider

This extension implements probably the simplest possible provider. As far as the API goes all it supports is requesting a list of add-ons by type or a single add-on by ID. These functions pass add-on objects to the callbacks. For this add-on these objects are held in a database so that code does some fairy uninteresting (and horribly synchronous) sql queries and generates objects that the API expects.

Adding new types to the UI

Perhaps the hardest part of this extension is getting the new type of add-on to display in the UI. Unfortunately one thing that we haven’t implemented so far is any kind of auto-discovery of add-on types. Instead the UI works from a mostly hardcoded list. This is something that we think it would be nice to change but at the moment it seems unlikely that we wiull get time to before Firefox 4, unless someone wants to volunteer to do some of the work.

The demo extension works around this restriction by inserting some elements into the add-ons manager window whenever it detects it opening. In particular it adds an item to the category list with a value attribute “addons://list/user-script”. The add-ons manager UI uses this kind of custom URL to decide what to display when a category is selected. In this case it means displaying the normal list view (that plugins and extensions currently use) and to ask the API for add-ons of the type “user-script”. There is also some code there that overrides the normal string bundle that the manager uses to localize the text in the UI to allow adding in some additional strings. The code I am showing is of course badly written in that it is hardcoded and so could not be localized, please forgive me for cutting corners with the demo.

That is basically all that is needed to have the UI work to display the new add-ons from the registered provider however the demo also throws in some style rules to pretty things up with a custom icon

Notifying the UI of changes

When you implement your own provider you have to be sure to send out appropriate notifications whenever changes to the add-ons you manager happen so that any UI can update accordingly. I won’t go into too much detail here, hopefully the AddonListener and InstallListener API covers the events you need to know about enough. You can see the script database send out some of these notifications.

Get the full code

This has been a very short overview of the highlights of this demo, hopefully enough for the interested to pick up the code and make use of it themselves. The full source of the extension is available from the mercurial repository. Right now I wouldn’t really release this as an extension. As I’ve mentioned it uses synchronous sql queries (on every page load no less!) and cannot be localized. These things can be fixed but this was just made as a demo in basically one evening to show off the sorts of things that are possible with the new add-ons manager.

Introducing the new Add-ons Manager

Add-ons have really been an integral part of Firefox ever since before its first release. In fact Firefox has had an add-ons manager of some form since version 0.2 (which was at that time called Phoenix). Firefox 4 will include a completely redesigned add-ons manager and while many nightly testers will have already seen our work, now the beta release is out I wanted to talk about some of the new features it includes. If you’re interested I’m also writing a companion piece that talks about the history of the add-ons manager from its first appearance through to what the future may bring.

Add-ons Manager Redesign — The new Add-ons Manager

Changing the home of add-ons

The most obvious change that you’ll see is that the add-ons manager in Firefox is no longer a separate window. Instead it appears as a tab within the main Firefox window, just like webpages. One of the big things that I think revolutionized web browsing was the introduction of tabs to allow you to keep many webpages open in the same window. This is because in general it is pretty difficult to keep track of multiple windows. Opening one tends to block the other making it difficult to quickly switch around them unless you are very careful about where you place them all. This really applies to parts of the user interface as well as webpages themselves. With the new design you’ll never go back to your webpage to read something and have a hard time finding the add-ons manager again.

The Firefox user experience team has been talking about putting parts of Firefox’s user interface into tabs for some time now and other web browsers have done this sort of thing already. When we were redesigning the add-ons manager this was one of the first choices that we made.

Learn more about add-ons

One of the first big features that I worked on when I started at Mozilla was adding the “Get Add-ons” pane to the old add-ons manager. It was an area that allowed users investigating what this part of Firefox was for to see a list of a few recommended add-ons as well as do simple searching for add-ons listed at addons.mozilla.org and install them right there without having to open a webpage. It turns out that many use this as their main way to get add-ons, somewhere around one in five of every add-on downloaded from addons.mozilla.org comes through this pane. What we wanted to do with this redesign was to extend the sorts of information and recommendations that we can provide directly in the manager.

While still only showing a placeholder in the current beta the revamped Get Add-ons section will eventually include recommended and features add-ons, lists of the most popular add-ons as well as a short overview of what add-ons are for those who have never used them before.

Make changes without restarting

The new add-ons manager supports a new type of add-on, one that doesn’t need you to restart Firefox in order to use it. It is possible for quite a lot of extension developers to change their add-on to use this feature but perhaps the easiest way to support it is to use the new Jetpack SDK to build your add-ons. An example of an extension built on the SDK is MailPing, a simple tool to let you know wen you have new emails at your Google or Yahoo mail account.

Find the add-ons you need

Once you have a large number of add-ons installed it can be hard to find the one you are looking for if you want to make changes to it. In fact if you can’t remember whether the add-on you were looking for is a Plugin, Extension or Theme then the old manager made you look through each list till you found it. With the new manager we’ve added a few tools to help you. You can quickly sort the add-ons in a list in a few different ways. We’re also building search right in. Typing something in the search box will look through all your add-ons for you trying to guess which you were looking for. In the future this will also return results from the extensive catalogue of add-ons available from addons.mozilla.org.

See more about your add-ons

You’ll probably see that in the new manager there is room for more information about your add-ons that was previously unavailable unless you visited addons.mozilla.org. Although in the initial beta lots of this information is not real, by the time we release Firefox 4 we expect you to be able to see information about ratings and reviews (particularly important when looking for new add-ons to download) and have the ability to easily contribute to add-ons that you could not do without.

Still to come

This is all just the early betas so there is still a lot of new things to come. In particular the way the manager looks right now is the same on every platform and based on early designs. The user experience team are now finalizing how the manager should look on all platforms. Future betas will also bring better feedback for the update and install processes.

Hopefully you’ll find the new add-ons manager easier to use but either way we are always looking for feedback so why not let us know what you think?

History of the Add-ons Manager

With all of the work that has gone into the new add-ons manager for Firefox 4 I thought it would be interesting to take a quick look back at the history of this part of Firefox and a quick look at what the future may hold.

Phoenix 0.2

Even in the earliest versions of Firefox, extensions were supported using the old XPInstall style packages. These had some pretty fundamental problems though in that there was no built in support for uninstalling extensions nor any way to disable them. There wasn’t even an extension manager window to see what you had installed at first. The very first time that a list of extensions and themes appeared in Firefox was way back in version 0.2, back when the product was called Phoenix. It was a very basic user interface and appeared inside the preferences window.

Firebird 0.6

After the product got a rename to Firebird the next incarnation of the manager split the themes and extensions into separate parts of the preferences window.

Firefox 0.9

Everything changed when Firefox 0.9 came along with its standalone extension manager window and support for the new install.rdf packages which are essentially unchanged from the extension packages that are used today.

Themes and extensions were displayed in different windows and a basic update service was in place, to be improved in the backend code for Firefox 1.0. This first version was mainly written by Ben Goodger

Firefox 1.5

Firefox 1.5 saw very few differences on the surface of the extension manager, some slight changes to the visual styling along with the rest of Firefox.

Behind the scenes, Darin Fisher made large changes. He allowed the manager to support loading extensions from different locations on the system including the Windows registry. He also created the ability to just extract extensions into the profile folder, which would be detected automatically the next time Firefox ran. Rob Strong then took over ownership of the manager to get it stable for release. This also saw my first patch to the extension manager, the rather insignificant bug 307925.

Firefox 2.0

Firefox 2.0 finally combined the separate extensions and theme windows into the unified add-ons manager.

Underneath more changes were going on including the first blocklist service to allow us to remotely disable extensions that were found to be harmful to users. Since its creation we have very rarely used this feature but when it has been used it has helped prevent security exploits and crashes.

Firefox 3.0

With Firefox 3 we started including Plugins in the add-ons manager window and for the first time you could download and install add-ons from addons.mozilla.org directly within the add-ons manager. This was the first big feature that I worked on in Firefox and I’m always pretty happy when I hear from the add-ons team just how much it is used.

Internally the blocklist got upgraded to support blocking plugins and new install locations were added for platforms other than Windows to support integration with other applications on the system.

Firefox 3.5 and 3.6

No real visual changes happened in Firefox 3.5 and 3.6 however Firefox 3.5 improved the blocklist service yet again, allowing for a couple of different levels of severity while Firefox 3.6 added support for warning users about outdated plugins on their system and brought the fledgling Personas into the manager allowing quick switching between simple backgrounds for the main Firefox window.

Firefox 4.0 beta

Firefox 4 is seeing a complete redesign of the add-ons manager adding support for extensions that don’t require restarts, an automatic update system with less interruptions to the user and a more useful way to discover new add-ons.

Add-ons Manager in Firefox 4 beta (early 2010)

Behind the scenes the new add-ons manager is now capable of managing new types of add-ons more easily than it was possible before.

Firefox 4.0

The user experience team are hard at work making the final designs for how the add-ons manager will look in the final Firefox 4 release. While only preliminary this is a quick idea of the sort of thing that they are going for:

Potential design for Firefox 4 final (maybe late 2010)

In the future we have plans to bring more types of add-ons into the main manager. Things like search plugins which currently are managed by their own custom window can fit in here. We also want to simplify customizing your extensions. Although it is unlikely we would stop allowing extension developers to create their own preferences windows we are looking into adding support for changing simple settings directly in the add-ons manager rather than needing to open new window. We’re hopeful that we can set up more automated ways of updating your installed plugins which are often the cause of security and stability problems and we want to significantly improve theme selection to make it easier to see what themes are available and switch between them.

It’s possible that some of these things may even happen in time for Firefox 4.0 but time is running short to get new things in before the final release.

Multiple breaking changes are coming for components in extensions

Are you an extension or application developer? Have you written any XPCOM components, JS, binary or otherwise? If not you can probably ignore the rest of this post, unless you are interested anyway.

If you do then you might be interested to hear that your components are probably going to break in an upcoming Firefox nightly, maybe as early as next week. I’m going to blog specific examples on the changes you need to make once we have better documentation up and builds to test against, for now it is just important for you to know that the changes are coming.

Changing XPCOM component registration

One of the things we are doing is changing how XPCOM components are registered. Previously during registration we would load all binary and JS component files and call them asking them to register themselves. If you used XPCOMUtils.jsm then some of that ugliness will have been hidden from you. The future though will be registration using manifest files, similar to how chrome is registered (in the same files in fact). Every single JS or binary component will need fixes to support this. The amount of work to do is fairly trivial per component and it is possible to support both old and new styles of registration in the same component but until that work is done your components will not work.

Starting extension components later

Currently there are a bunch of different notifications that components can listen for to perform actions during startup. Commonly xpcom-startup, app-startup and profile-after-change have been the most common. The latter is generally the best to use since this is one of the earliest notifications that occurs after the profile folder (and hence preferences and other services) is available.

The news is that once we make our changes components from extensions will no longer be able to register for and receive the xpcom-startup or app-startup notifications at all. profile-after-change will be the first notification that components can be loaded by.

Most of the reasons for loading on app-startup were either because developers didn’t know about profile-after-change or at one point it used to be necessary to load for app-startup just to be able to register for profile-after-change in the observer service. That is no longer the case as you can register for it in the category manager exactly as you can for app-startup.

Why?

You might well ask why we are making all these changes.

Currently whenever we detect a potential change in the set of usable components (meaning either the application version has changed or an extension has been added/removed/enabled/disabled) we must throw away existing component registrations and then perform what has become known as the EM restart where the application is restarted during its startup (in theory invisibly to the user) to make sure that any component that weren’t meant to be loaded are unloaded and to re-register everything and load any new components that need it.

Registration is itself a costly process since every component must be loaded and executed when really many if not most of them do not need to be loaded during startup. In the E10S world there are even more problems. Content processes must either register all components during every startup or somehow share a component cache with the chrome process.

The new registration model allows us to do away with the EM restart. On startup rather than relying on a potentially stale component cache we will read the registrations for the main application out of a manifest file. This will give us enough of XPCOM to then be able to load the extension manager and do any necessary work to install/uninstall/update any extensions after which we can simply load the component registrations for those out of the manifest file. Startup then just proceeds as normal, no restart is necessary since nothing from extensions can have been loaded when it shouldn’t.

E10S content processes can just read the component registrations during startup, since they are in simple manifest files this will go a lot faster

Documenting the new Add-ons Manager

I’ve spent some time this week transferring all the API documentation for the new add-ons manager from the Mozilla wiki to the Mozilla Developer Network. This should now be the place to go for the definitive info.

Right now it is pretty dry, for the most part just pure API info with no examples. Before I started working more on that side of things I wanted to ask what kind of examples people might like to see documented?

Support for dropping XPI files into the extension install locations might be going away

For some time now Firefox has supported a way of installing extensions that involves simply copying the extension’s XPI file into one of the extension install locations. The next time Firefox runs it would pop up the install dialog for the extension and allow the user to choose whether to install it or not.

I don’t know how many people use this feature and while the code to do it (at least for the profile folder) isn’t terribly complex, it is additional code that may not be necessary. Right now the new add-ons manager doesn’t support it and I’ve heard only a couple of people comment on its absence but nightly testers are by no means representational so I’m asking a little more widely whether people have a real need for keeping this working in Firefox 4?

To be clear we aren’t talking about the method of installing where you extract your extension into a directory in the install locations, nor are we talking about the method where you create a text file in the install location containing the path to your extension.

Add-ons manager re-landed

A little sort of coincidental performance regression forced us to back out the new add-ons manager last week. It has now been re-landed with added bug fixes and should be in tomorrow’s nightly.

The new add-ons manager is here

Finally, after far too much time, the new add-ons manager is about to land in trunk nightlies. I am putting together the final patches to land now. The bit most people will see is the new UI so I guess I’ll steal Boriss’ image for you to look at here with the same caveats. What you see on trunk over the next few days is just the initial steps to switching to a redesigned UI and (more importantly from my point of view) a totally new extension manager backend that will make it easier for us to improve and build upon in the future. The changes are so large that it is important to get more people testing it now while it still looks fairly unpolished so we can pick up problems that we’ve missed.

Tomorrow (Friday) the QA team are holding a test day on the new add-ons manager, if you want to help test for issues you can join us in #testday on irc.mozilla.org

We are filing bugs with [rewrite] in the status whiteboard and blocking either the backend tracking bug or the UI tracking bug. So you can check the dependency tree to see what issues we already know about and plan to take care of. Please file bugs for anything else you see, especially when it is something that is wrong compared to Firefox 3.6.

There will be some add-ons broken in the new builds. This is fairly normal for any API update. As a general rule if some add-ons work but others don’t then contacting the authors of those broken add-ons is the best idea.

Developers of other applications may find things broken, ultimately this is unavoidable when changing APIs on this scale, please contact me if you need some help getting things working again.

How do restartless add-ons work?

I blogged a short time ago about how we’re adding support for a new form of add-on to Firefox that can install and uninstall without needing to restart the application. Since then I’ve been finalizing a specification for how the platform will load these add-ons, trying to keep it simple but still give developers everything they commonly need. The planned specification is now available and if developers have comments then I’d like to hear them. Currently there isn’t a version of Firefox that implements it but that should change in the next day or so when I make the changes to the add-ons manager project branch and very soon when it all lands on trunk.

My hope is that once on trunk this spec won’t change but obviously this is quite new so we may see changes for a short time if add-on developers come across problems.

How we’re breaking some extensions in the near future

You may have read some reports about how we’re re-implementing the bulk of the extension manager in Firefox. It’s been a long running project (something like a year since I first really started planning how to do it). Things are finally started to come together and all being well we are likely to look at landing the first pieces of this on the trunk nightlies in as little as a weeks time. I’ll be up front, this isn’t going to be a perfect landing. There may be some thing that are missing and other bits where the user experience isn’t as perfect as it will be finally. Of course there may also be bugs we have to rush to fix. Despite all this we feel that we’re about at the point where exposing it to the hands of thousands of nightly testers is the best way forward. Your eyes spot things that we miss, even things that may seem obvious to you and you’re vital to us getting these sorts of features polished and really just how they should be before they get released to the world at large in a Firefox release.

One thing I wanted to mention is how this will affect the extensions themselves. In many cases as you might hope there will be no change at all, the extensions will continue to work as they did on previous trunk builds. Those that do anything to the add-ons manager UI will see obvious problems since that UI has changed considerably. There is also a set of extensions that are likely to see issues because of API changes. This is pretty much normal for any feature but I thought I’d mention two common uses that extension developers are going to have to change in their code.

Accessing file and version information for extensions

The extension manager interface as it used to exist is gone with the new code. Some extensions would have used this to access their extension’s version and packaged files. There is a replacement so it mostly just means changing code to something similar to this:

Components.utils.import("resource://gre/modules/AddonManager.jsm");

AddonManager.getAddonByID("my-addon@foo.com", function(addon) {
  alert("My extension's version is " + addon.version);
  alert("Did I remember to include that file.txt file in my XPI? " +
        addon.hasResource("file.txt") ? "YES!" : "No :(");
  alert("Let's pretend I did, it's available from the URL " + addon.getResourceURL("file.txt"));
});

Hopefully to developers who used the old APIs this will look a lot nicer than it used to, I certainly prefer it. There are a couple of things to note. The old API would give you an nsIFile directly to play with. The new API gives you a URL. It is currently possible to get an nsIFile from this (as it is a file:// url), however in the future we may be no longer extracting the files out of an XPI for an extension so it is worth trying to just use the URL where you can. We might add a simple method for getting say an input stream to the files as well.

The second thing to note is that this API is asynchronous. As performance becomes more and more of an issue for the platform we have to start moving APIs to be asynchronous to allow file accesses to happen on background threads leaving the UI responsive while we wait for data. In most cases this probably won’t cause developers much of a problem, unfortunately it is difficult to make this appear to operate synchronously safely. People may know the trick of spinning an event loop while you wait for data, that is kind of unsafe since it can allow unexpected reeentrancy into code that isn’t anticipating it.

This change to an asynchronous API brings up the other main change that I am sad to have to make but cannot see a way around.

FUEL has to change

FUEL was designed to be a simple stable wrapper around the nitty gritty world of parts of XPCOM and XUL. It has I believed served fairly well in that purpose though in retrospect there are some things it might have been nice to have different about it. The problem right now though is that FUEL has wrappers around the old extension manager and they are synchronous. They simply cannot stay and work with the new extension manage, which means we have to change a part of the FUEL API slightly. This is unfortunate but I’ve tried to make the change as low impact as possible, there is in fact just a single change. Previously to get an add-ons version you might have done:

alert(Application.extensions.get("my-addon@foo.com").version);

Now you will have to do:

Application.getExtensions(function(extensions) {
  alert(extensions.get("my-addon@foo.com").version);
});

Again it switches from synchronous to asynchronous, but that is really the only change here. The extensions object passed into the callback function is basically the same as the current Application.extensions object, only a slight difference in that it is a snapshot of the list of extensions at the time of retrieval. That doesn’t make a great deal of difference for normal extensions as they cannot install or uninstall without a restart but it may have an effect if you are looking at the new restartless add-ons, in which case you can just get a new copy of the extensions object.