I’ve been acting as the owner for the add-ons manager for the past little while and while I have always cared a lot about the add-ons space it is time to formerly pass over the torch. So I was pleased that Rob Helmer was willing to take it over from me.
Rob has been doing some exceptional work on making system add-ons (used as part of the go faster project) more robust and easier for Mozilla to use. He’s also been thinking lot about improvements we can make to the add-ons manager code to make it more friendly to approach.
As my last act I’m updating the suggested reviewers in bugzilla to be him, Andrew Swan (who in his own right has been doing exceptional work on the add-ons manager) and me as a last resort. Please congratulate them and direct any questions you may have about the add-ons manager towards Rob.
The add-ons manager has a dirty secret. It uses an awful lot of synchronous file I/O. This is the kind of I/O that blocks the main thread and can cause Firefox to be janky. I’m told that that is a technical term. Asynchronous file I/O is much nicer, it means you can let the rest of the app continue to function while you wait for the I/O operation to complete. I rewrote much of the current code from scratch for Firefox 4.0 and even back then we were trying to switch to asynchronous file I/O wherever possible. But still I used mostly synchronous file I/O.
Here is the problem. For many moons we have allowed other applications to install add-ons into Firefox by dropping them into the filesystem or registry somewhere. We also have to do things like updating and installing non-restartless add-ons during startup when their files aren’t in use. And we have to know the full set of non-restartless add-ons that we are going to activate quite early in startup so the startup function for the add-ons manager has to do all those installs and a scan of the extension folders before returning back to the code startup up the browser, and that means being synchronous.
The other problem is that for the things that we could conceivably use async I/O, like installs and updates of restartless add-ons during runtime we need to use the same code for loading and parsing manifests, extracting zip files and others that we need to be synchronous during startup. So we can either write a second version that is asynchronous so we can have nice performance at runtime or use the synchronous version so we only have one version to test and maintain. Keeping things synchronous was where things fell in the end.
That’s always bugged me though. Runtime is the most important time to use asynchronous I/O. We shouldn’t be janking the browser when installing a large add-on particularly on mobile and so we have taken some steps since Firefox 4 to make parts of the code asynchronous. But there is still a bunch there.
Performances is pretty important for the add-ons manager startup code, the longer we spend in startup the more it hurts us. Would this switch slow things down? I assumed that there would be some losses due to other things happening during an event loop tick that otherwise wouldn’t have but that the file I/O operations should take around the same time. And here is the clever bit. Because it is asynchronous I could fire off operations to run in parallel. Why check the modification time of every file in a directory one file at a time when you can just request the times for every file and wait until they all complete?
There are really a tonne of things that could affect whether this would be faster or slower and no amount of theorising was convincing me either way and last night this had finally been bugging me for long enough that I grabbed a bottle of wine, fired up the music and threw together a prototype.
It took me a few hours to switch most of the main methods to use Task.jsm, switch much of the likely hot code to use OS.File and to run in parallel where possible and generally cover all the main parts that run on every startup and when add-ons have changed.
The challenge was testing. Default talos runs don’t include any add-ons (or maybe one or two) and I needed a few different profiles to see how things behaved in different situations. It was possible that startups with no add-ons would be affected quite differently to startups with many add-ons. So I had to figure out how to add extensions to the default talos profiles for my try runs and fired off try runs for the cases where there were no add-ons, 200 unpacked add-ons with a bunch of files and 200 packed add-ons. I then ran all those a second time with deleting extensions.json between each run to force the database to be loaded and rebuilt. So six different talos runs for the code without my changes and then another six with my changes and I triggered ten runs per test and went to bed.
The first thing I did this morning was check the performance results. The first ready was with 200 packed add-ons in the profile, should be a good check of the file scanning. How did it do? Amazing! Incredible! A greater than 50% performance improvement across the board! That’s astonishing! No really that’s pretty astonishing. It would have to mean the add-ons manager takes up at least 50% of the browser startup time and I’m pretty sure it doesn’t. Oh right I’m accidentally comparing to the test run with 200 packed add-ons and a database reset with my async code. Well I’d expect that to be slower.
Ok, let’s get it right. How did it really do? Abysmally! Like incredibly badly. Across the board in every test run startup is significantly slower with the asynchronous I/O than without. With no add-ons in the profile the new code incurs a 20% performance hit. In the case with 200 unpacked add-ons? An almost 1000% hit!
Ok so that wasn’t the best result but at least it will stop bugging me now. I figure there are two things going on here. The first is that OS.File might look like you can fire off I/O operations in parallel but in fact you can’t. Every call you make goes into a queue and the background worker thread doesn’t start on one operation until the previous has completed. So while the I/O operations themselves might take about the same time you have the added overhead of passing messages between the background thread and promises. I probably should have checked that before I started! Oh, and promises. Task.jsm and OS.File make heavy use of promises and I have to say I’m sold on using them for async code. But. Everytime you wait for a promise you have to wait at least one tick of the event loop longer than you would with a simple callback. That’s great if you want responsive UI but during startup every event loop tick costs time since other code might be running that you don’t care about.
I still wonder if we could get more threads for OS.File whether it would speed things up but that’s beyond where I want to play with things for now so I guess this is where this fun experiment ends. Although now I have a bunch of code converted I wonder if I can create some replacements for OS.File and Task.jsm that behave synchronously during startup and asynchronously at runtime, then we get the best of both worlds … where did that bottle of wine go?
For some time now Firefox for mobile has had this nice feature where add-ons could embed their preferences right into the list of add-ons, no need to open a whole a new window like add-ons for desktop have to. During the development of Firefox 4 we were a little jealous of what the mobile team had done and so we drew up some ideas for how the same functionality would look on desktop. We didn’t get time to implement them then but I’m excited that someone from the community stepped up and implemented it for us. Not just that but he made the code shared between mobile and desktop, added some new option types and made it work fine for restartless add-ons which are unable to register their own chrome.
The basic idea is simple. Create a XUL file containing a list of <setting> elements. Different types of settings are possible, checkboxes, input boxes, menulists, buttons, etc. Each one shows up as a row in the details view for an add-on in the add-ons manager. The XUL file can either be just added to your XPI (call it options.xul) or referenced by the optionsURL option in your install.rdf.
As a bonus Geoff also implemented support for in-tab preferences. This makes Firefox load an add-ons options UI in a new tab instead of a new window. Setting the optionsType property to 3 enables this.
One of the nice features that we added to the add-ons manager in Firefox 4 was support for custom add-on types that could be treated the same way as the built-in types, even showing up in the same UI if you did a little work. I blogged a basic example of how to do this and I know since then Greasemonkey and Stylish have been using the support.
A few days ago (and now in Aurora builds) I landed some improvements that simplify the hardest part of this, making the UI appear.
Registering a custom type
Previously in order to add a new type of add-on and have it appear in the add-ons manager you would have to do two things, first tell the add-ons manager module that you supported add-ons and secondly manually create some elements in the UI to allow the user to view them. These two things have now been combined:
This registers the provider as before but also tells the module that your provider supports an AddonType with the ID “scripts” and some other information. The documentation explains a little more about the properties. You’ll note it passes an array to allow for providers registering multiple types.
Currently you only need to do this if you want your add-on types to show up in the UI, however we may start requiring providers to register their types to help speed up some of the API calls. If you want to register your type and not have the UI automatically show it just don’t give it a viewType.
Some of the changes we had to make may have broken some add-ons that have been doing this the hard way in the past. Here are the once I’ve identified and some ways to work around the problems:
Types must be registered
The UI now requires that if you’re displaying your add-ons in the regular list view then it must have been registered. If you only care about supporting Firefox 6 and higher then you can just remove your code for adding your type to the category list, if you want to support both then you should either only do that on versions before Firefox 6, or you could always manually add your type to the UI but also register your type with no viewType.
Some IDs changed in the UI
We made some changes to the IDs of elements in the add-ons manager UI, we dropped the “s” on the end of the type selectors. category-extensions became category-extension as an example.
Category items aren’t in the UI immediately
Because we now build the list of types to display in the UI (even the built-in types use the same type registration I’ve talked about here), the elements for those types aren’t in the UI until after the main initialize function for the window has completed. If you try to use overlays that refer to those elements or if you run script that tries to find these elements before the initialize function completes (it is a load event listener) then they won’t exist and you’ll probably find things broken.
Firefox 4 is just around the corner and it’s great to look back over just how far the Add-ons Manager has come since Firefox 3.6. In fact if you want to see the full history look at my earlier post that shows its evolution since Phoenix 0.2. We set out with some pretty lofty goals for Firefox 4 and I’m pretty excited at just how many of them we achieved. I hope everyone appreciates the hard work that Blair, Boriss, Justin, Henrik, Ben, myself and all the others put in to get us to where we are today.
Of course, we aren’t finished. There were a lot of things that we wanted to get done and didn’t get time for and I’m sure we’ll be getting lots of feedback from users of the final release to work on. Since the pressure has been off I’ve been skimming back over bug reports and notes to try to come up with an idea of what we want to work on over the next few releases of Firefox. As you may know the plan is to move to quarterly releases and not really block a release on any particular projects but we will still be getting an idea of what product drivers will prioritise work on and guess at what other things we can fit in.
In the closing stages of Firefox 4 there were a lot of small polish issues identified. We knocked down a lot of them but there are still plenty of tweaks that we’re going to be bringing to the UI to make sure it is as usable as we can make it. It isn’t really worth applying a fixed schedule to these as they are small one-off fixes that can just come as and when ready. We already have some that are basically ready to land.
Incremental API fixes
There are lots of things that I’d like to see done for the platform and APIs that developers use. Again there is no fixed schedule for these but the main things on my mind are changes that make developing restartless add-ons or extending the add-ons manager easier. I’ve already started work on a couple of these and expect to get them landed after the tree reopens.
There are of course also larger projects that will require more resources. I’ve started writing up short project pages for each of these. Each project should start with the people working on it coming to agreement on what the actual goals should be and an implementation plan so until the projects are actually picked up the project pages are mostly made up. I’ve put down a rough idea of when a project might make it into a Firefox release. These are just guesses, if the project turns out to be larger than expected they’ll come later, if people volunteer to work on projects or if the product drivers decide they are a priority then they could come sooner.
You can see that assuming a quarterly release schedule we have enough projects here to last till the end of 2011 at least, quite likely longer. Over that sort of time I guess it’s almost guaranteed that priorities and plans will change in the meantime but hopefully this gives an idea of what our thinking is at the moment. Expect to see progress on some of the things at the top of the list very soon!
As I mentioned before I was part of a presentation at Add-on-Con this year. Myself, Boriss and Justin talked about the new UI changes in Firefox 4 and about the main changes to the add-ons manager. If you’re particularly interested the slides are available here though I guess slides are often just tiny snippets of info from the actual session so if anything catches your eye you’ll need to get in touch and ask us about it.
You might have heard of this web-browser. It’s called Firefox. You may have also heard that a new version is due out soon. As my part in its development I have helped completely reshape the way the add-ons manager looks. The good news is that the large bits of the changes are pretty much done, pretty much all that is left is a bunch of UI tweaks and some small behaviour changes.
This is where you come in. Yes you, the one reading this post. If you have an urge to help out you can look through this list of things we want to get done on the add-ons manager before release, pick one out and go nuts. These are all things that shouldn’t require very much experience working on the add-ons manager or even Firefox in general, though of course some are easier than others. We have some great information about how to get started and also a great community who can help out if you get stuck. So if you’ve ever had a hankering to get involved this would be a great time to dive in.
Once again that list is here: http://bit.ly/duYWdP, give me or Blair a shout if you have questions or don’t know where to get started.
One of the goals of the new add-ons manager API was to create something that was itself extensible. A couple of times in the past we’ve had to add new types of add-ons to the UI like Plugins and Personas. In both cases squeezing them into the UI was something of a kludge involving a bunch of custom code for each case. We already have a number of new types of add-ons that we want to add, things like search plugins which are currently managed by their own custom UI.
With this design the user interface doesn’t need to care about implementation details of any of the providers, how they store their data or what exactly their add-ons are and do. Because each gives objects that obeys the same interface it can just display and manipulate them.
To try to show this all off I recently put together a small demo extension for the Mozilla Summit that registers a new type of add-on to be displayed in the main add-ons manager. This is a short overview of some of the highlights and I’ll make the code available for people to look at and take examples from. The add-on was a basic implementation of Greasemonkey allowing user scripts to be installed, managed through the add-ons manager and do it all as a restartless add-on.
Making a restartless add-on
Add-ons don’t have to be developed with Jetpack to make them restartless, although the Jetpack SDK certainly makes things easier on you, at the expense of less access to the internals of the platform.
The first thing to learn about making a restartless add-on is that you can forget about using XUL overlays or registering XPCOM components to be called at startup. Neither are supported at the moment, and maybe never will. Instead you have to provide a bootstrap script. This is a simple “bootstrap.js” file in the root of the extension that should include a “startup” and “shutdown” function. These are called whenever Firefox wants to start or stop your add-on either because the application is starting up or shutting down or the add-on is being enabled or disabled. You can also provide “install” and “uninstall” methods to be notified of those cases but that is probably unnecessary in most cases.
At startup the demo extension does some basic things. It registers for some observer notifications, registers a new add-on provider (I’ll talk more about that below) and does a little work to include itself in the add-ons manager UI (again, see below).
The rule is this. Anything your add-on does after being started must be undone by the shutdown function. The shutdown function often ends up being the inverse of startup, here it removes observer notification registrations, unregisters the add-on provider and removes itself from the UI. It also shuts down a database if it was opened.
Implementing a new provider
This extension implements probably the simplest possible provider. As far as the API goes all it supports is requesting a list of add-ons by type or a single add-on by ID. These functions pass add-on objects to the callbacks. For this add-on these objects are held in a database so that code does some fairy uninteresting (and horribly synchronous) sql queries and generates objects that the API expects.
Adding new types to the UI
Perhaps the hardest part of this extension is getting the new type of add-on to display in the UI. Unfortunately one thing that we haven’t implemented so far is any kind of auto-discovery of add-on types. Instead the UI works from a mostly hardcoded list. This is something that we think it would be nice to change but at the moment it seems unlikely that we wiull get time to before Firefox 4, unless someone wants to volunteer to do some of the work.
The demo extension works around this restriction by inserting some elements into the add-ons manager window whenever it detects it opening. In particular it adds an item to the category list with a value attribute “addons://list/user-script”. The add-ons manager UI uses this kind of custom URL to decide what to display when a category is selected. In this case it means displaying the normal list view (that plugins and extensions currently use) and to ask the API for add-ons of the type “user-script”. There is also some code there that overrides the normal string bundle that the manager uses to localize the text in the UI to allow adding in some additional strings. The code I am showing is of course badly written in that it is hardcoded and so could not be localized, please forgive me for cutting corners with the demo.
That is basically all that is needed to have the UI work to display the new add-ons from the registered provider however the demo also throws in some style rules to pretty things up with a custom icon
Notifying the UI of changes
When you implement your own provider you have to be sure to send out appropriate notifications whenever changes to the add-ons you manager happen so that any UI can update accordingly. I won’t go into too much detail here, hopefully the AddonListener and InstallListener API covers the events you need to know about enough. You can see the script database send out someofthese notifications.
Get the full code
This has been a very short overview of the highlights of this demo, hopefully enough for the interested to pick up the code and make use of it themselves. The full source of the extension is available from the mercurial repository. Right now I wouldn’t really release this as an extension. As I’ve mentioned it uses synchronous sql queries (on every page load no less!) and cannot be localized. These things can be fixed but this was just made as a demo in basically one evening to show off the sorts of things that are possible with the new add-ons manager.
Add-ons have really been an integral part of Firefox ever since before its first release. In fact Firefox has had an add-ons manager of some form since version 0.2 (which was at that time called Phoenix). Firefox 4 will include a completely redesigned add-ons manager and while many nightly testers will have already seen our work, now the beta release is out I wanted to talk about some of the new features it includes. If you’re interested I’m also writing a companion piece that talks about the history of the add-ons manager from its first appearance through to what the future may bring.
Changing the home of add-ons
The most obvious change that you’ll see is that the add-ons manager in Firefox is no longer a separate window. Instead it appears as a tab within the main Firefox window, just like webpages. One of the big things that I think revolutionized web browsing was the introduction of tabs to allow you to keep many webpages open in the same window. This is because in general it is pretty difficult to keep track of multiple windows. Opening one tends to block the other making it difficult to quickly switch around them unless you are very careful about where you place them all. This really applies to parts of the user interface as well as webpages themselves. With the new design you’ll never go back to your webpage to read something and have a hard time finding the add-ons manager again.
The Firefox user experience team has been talking about putting parts of Firefox’s user interface into tabs for some time now and other web browsers have done this sort of thing already. When we were redesigning the add-ons manager this was one of the first choices that we made.
Learn more about add-ons
One of the first big features that I worked on when I started at Mozilla was adding the “Get Add-ons” pane to the old add-ons manager. It was an area that allowed users investigating what this part of Firefox was for to see a list of a few recommended add-ons as well as do simple searching for add-ons listed at addons.mozilla.org and install them right there without having to open a webpage. It turns out that many use this as their main way to get add-ons, somewhere around one in five of every add-on downloaded from addons.mozilla.org comes through this pane. What we wanted to do with this redesign was to extend the sorts of information and recommendations that we can provide directly in the manager.
While still only showing a placeholder in the current beta the revamped Get Add-ons section will eventually include recommended and features add-ons, lists of the most popular add-ons as well as a short overview of what add-ons are for those who have never used them before.
Make changes without restarting
The new add-ons manager supports a new type of add-on, one that doesn’t need you to restart Firefox in order to use it. It is possible for quite a lot of extension developers to change their add-on to use this feature but perhaps the easiest way to support it is to use the new Jetpack SDK to build your add-ons. An example of an extension built on the SDK is MailPing, a simple tool to let you know wen you have new emails at your Google or Yahoo mail account.
Find the add-ons you need
Once you have a large number of add-ons installed it can be hard to find the one you are looking for if you want to make changes to it. In fact if you can’t remember whether the add-on you were looking for is a Plugin, Extension or Theme then the old manager made you look through each list till you found it. With the new manager we’ve added a few tools to help you. You can quickly sort the add-ons in a list in a few different ways. We’re also building search right in. Typing something in the search box will look through all your add-ons for you trying to guess which you were looking for. In the future this will also return results from the extensive catalogue of add-ons available from addons.mozilla.org.
See more about your add-ons
You’ll probably see that in the new manager there is room for more information about your add-ons that was previously unavailable unless you visited addons.mozilla.org. Although in the initial beta lots of this information is not real, by the time we release Firefox 4 we expect you to be able to see information about ratings and reviews (particularly important when looking for new add-ons to download) and have the ability to easily contribute to add-ons that you could not do without.
Still to come
This is all just the early betas so there is still a lot of new things to come. In particular the way the manager looks right now is the same on every platform and based on early designs. The user experience team are now finalizing how the manager should look on all platforms. Future betas will also bring better feedback for the update and install processes.
Hopefully you’ll find the new add-ons manager easier to use but either way we are always looking for feedback so why not let us know what you think?
With all of the work that has gone into the new add-ons manager for Firefox 4 I thought it would be interesting to take a quick look back at the history of this part of Firefox and a quick look at what the future may hold.
Even in the earliest versions of Firefox, extensions were supported using the old XPInstall style packages. These had some pretty fundamental problems though in that there was no built in support for uninstalling extensions nor any way to disable them. There wasn’t even an extension manager window to see what you had installed at first. The very first time that a list of extensions and themes appeared in Firefox was way back in version 0.2, back when the product was called Phoenix. It was a very basic user interface and appeared inside the preferences window.
After the product got a rename to Firebird the next incarnation of the manager split the themes and extensions into separate parts of the preferences window.
Everything changed when Firefox 0.9 came along with its standalone extension manager window and support for the new install.rdf packages which are essentially unchanged from the extension packages that are used today.
Themes and extensions were displayed in different windows and a basic update service was in place, to be improved in the backend code for Firefox 1.0. This first version was mainly written by Ben Goodger
Firefox 1.5 saw very few differences on the surface of the extension manager, some slight changes to the visual styling along with the rest of Firefox.
Behind the scenes, Darin Fisher made large changes. He allowed the manager to support loading extensions from different locations on the system including the Windows registry. He also created the ability to just extract extensions into the profile folder, which would be detected automatically the next time Firefox ran. Rob Strong then took over ownership of the manager to get it stable for release. This also saw my first patch to the extension manager, the rather insignificant bug 307925.
Firefox 2.0 finally combined the separate extensions and theme windows into the unified add-ons manager.
Underneath more changes were going on including the first blocklist service to allow us to remotely disable extensions that were found to be harmful to users. Since its creation we have very rarely used this feature but when it has been used it has helped prevent security exploits and crashes.
With Firefox 3 we started including Plugins in the add-ons manager window and for the first time you could download and install add-ons from addons.mozilla.org directly within the add-ons manager. This was the first big feature that I worked on in Firefox and I’m always pretty happy when I hear from the add-ons team just how much it is used.
Internally the blocklist got upgraded to support blocking plugins and new install locations were added for platforms other than Windows to support integration with other applications on the system.
Firefox 3.5 and 3.6
No real visual changes happened in Firefox 3.5 and 3.6 however Firefox 3.5 improved the blocklist service yet again, allowing for a couple of different levels of severity while Firefox 3.6 added support for warning users about outdated plugins on their system and brought the fledgling Personas into the manager allowing quick switching between simple backgrounds for the main Firefox window.
Firefox 4.0 beta
Firefox 4 is seeing a complete redesign of the add-ons manager adding support for extensions that don’t require restarts, an automatic update system with less interruptions to the user and a more useful way to discover new add-ons.
Behind the scenes the new add-ons manager is now capable of managing new types of add-ons more easily than it was possible before.
The user experience team are hard at work making the final designs for how the add-ons manager will look in the final Firefox 4 release. While only preliminary this is a quick idea of the sort of thing that they are going for:
In the future we have plans to bring more types of add-ons into the main manager. Things like search plugins which currently are managed by their own custom window can fit in here. We also want to simplify customizing your extensions. Although it is unlikely we would stop allowing extension developers to create their own preferences windows we are looking into adding support for changing simple settings directly in the add-ons manager rather than needing to open new window. We’re hopeful that we can set up more automated ways of updating your installed plugins which are often the cause of security and stability problems and we want to significantly improve theme selection to make it easier to see what themes are available and switch between them.
It’s possible that some of these things may even happen in time for Firefox 4.0 but time is running short to get new things in before the final release.