Oxymoronical

For some time now I’ve been throwing around ideas for new features that we may want for the add-ons manager. After lots of thought and trying to take on board comments from anyone that would pipe up I decided it was high time to put together an actual plan for what features I have decided should be pursued and what order to start tackling them.

The roadmap itself should be pretty self explanatory, just remember that nothing is ever set in stone. I’ll try to keep it updated as things change, features seem less important or new items are added. But for now this is how I see the add-ons manager evolving over the course of about the next year or so, which may be the next two versions after Firefox 3.1.

Obviously the ordering of the goals may not be quite to everyone’s taste, but I’ve tried to strike a balance between getting new features done and getting groundwork laid to make things better in the future. I’m happy to take any comments people might have into consideration for altering the plan, but chances are I won’t be making any big changes unless you have some pretty damning evidence that a change is necessary

Faaborg has started a discussion on the newsgroups on the relative importance of polish and blocker bugs (I’d provide a link bug Google Groups seems to be refusing to acknowledge existence of the post). I have been for quite some time now very focused on the in-depth blocker issues, partly because that is where all the interesting work is for me, but also because I think it makes more sense for me or one of the other guys with good understandings of how the extension manager works to be making these changes rather than throwing others in at the deep end.

This has of course left little time for me to fix the trivial issues that Faaborg talks about. The things that users probably notice more and make the product look complete or half-done. This is why I’m really pleased that when I posted a list of simple bugs that anyone should be able to tackle there was a fairly good response. Of the 12 bugs on the original list, 9 have been fixed. I wanted to extend my thanks to those guys that took the time to work on those, some of them probably ended up being a little harder work than anticipated.

Of course the polish list is almost never ending. I’ve been adding more to the list of good first bugs, if anyone is interested in helping out then don’t be afraid to just email me or find me on IRC as Mossop and ask how much work something really involves.

I was very excited to learn at the Firefox Summit that Rob Strong was handing over ownership of the Extension Manager module to me. He did great work making the extension manager what it is today but has lately had to be more focused on Installer and windows integration issues.

Last week I spent a large part of my time trying to get myself up to speed on all the old filed bugs, clearing out things that clearly aren’t going to happen and trying to consolidate others. In particular I made the effort to go over every single unconfirmed bug and either resolve as appropriate or request any additional information from the reporter. It is rather sad that many of these were issues that noone commented in after the initial report or even worse the reporter responded with additional information but people (including me) dropped the ball and nothing further happened.

This big triage has been quite useful. Firstly the number of unconfirmed bugs for the add-ons manager has dropped from the mid 70s to under 30 (I would give you exact figures and pretty graphs but bugzilla’s charting is broken). I expect it to drop even further next week when I resolve the issues where the reporter is no longer responding to bugmail. Secondly it has allowed me to spot a pattern amongst 6 or so filed bugs which strongly suggests a new issue that can break add-on installations. While I have yet to be able to reproduce it in a test environment I certainly have more information to go on than previously.

Finally I have been marking all the trivial little issues that I think we could fix, but are unlikely to be priorities. These are the kinds of things that if you were interested in helping out and getting started in developing for Mozilla you could start out on. Many will end up being a few lines of patch, just need to figure out where to put it and test it.

So if you are interested in tackling any of these please give it a go. You can find me on IRC (for reasons best left alone I appear as Mossop) or by email if you need any guidance on getting started. I’m going to try to keep this list updated as I find new bugs that fit there, I still have yet to go through all of the 263 confirmed bugs, maybe a job for next week.

My head is slightly less fuzzy now and I’m almost back on UK time so I thought it would be a good opportunity to give a quick review of how the Add-ons Manager session I ran at the summit went. I won’t go into too much detail, just a look at the main points of discussion and how I felt the room reacted to the ideas. You can also see the slides I prepared or the notes that Basil kindly took for me without me needing to remember to ask. Most of the session was about things we might want to implement and what problems/benefits we might encounter.

Add-on locale packs

The idea of developing the UI for locale packs specific to extensions received a less enthusiastic response than I had expected. This may have been because there were more add-on developers and less localisers in the room, but the general feeling seemed to be that there was not a large benefit and the potential complications, particularly around updating, could cause some large problems.

Dependency resolution

The current add-on dependency implementation is a very rough start. Being able to automatically find and install dependant add-ons for the user, with clear UI to show what is happening, seemed to be something that didn’t have many downsides. There were also suggestions that AMO could handle dependencies and deliver sets of add-ons as multi-xpi packages, though this could cause issues in upgrades.

Add-on conflicts

The moment I mentioned the suggestion of a system to mark two add-ons as incompatible with each other half the room said they wanted it. It was pointed out that some add-ons were already disabling things they don’t like on install (if you hear of that happening without appropriate warning please let us know). It was also quickly realised that under the author’s control it could just become a means to avoid having to work to fix incompatibilities and to disable the competition. The suggestion of Mozilla maintaining a central list of incompatibilities seemed like a good way to go.

Install without restart

No surprise that this was seen as useful, particularly in non-Firefox applications that don’t have the wonder of session restore. We also started talking about the potential for lower impact extensions that just had a small sandbox to work in with a little piece of UI which could make loading and unloading dynamically far easier.

Install/uninstall hooks

We discussed how the current mechanisms for running setup and cleanup code for extensions were poor, there is no way to deal properly with uninstall in all situations. It was felt that add-ons should be able to provide a full uninstall script that would even run if uninstalled when disabled or in safe mode, but would not run if the add-on had been blocklisted. We also talked a little about sandboxing what this script might be able to access.

Replace RDF

I was surprised that there seemed little to no resistance to the prospect of introducing a new plainer XML form for install.rdf and update.rdf given some of the comments I had when I blogged about it last. We would of course still support the old forms for a time and make converters between the forms available. There didn’t appear to be any benefit in using JSON instead of XML.

Much of the rest of the slides were skimmed over as we had unfortunately run out of time by this point, but thankfully that was most of the more important stuff completed anyway. For the immediate future I think introducing replacements for RDF into Firefox 3.1 is potentially on the cards. None of the other items were ruled out, far from it, though the locale packs seemed less interesting to everyone, and I think there needs to be some very careful thought about the prospect of running an uninstall hook for an add-on in safe mode or when disabled.

The Extension Manager is pretty complex and so it can be difficult to gather the sort of information needed in a bug report to really diagnose what is going on. When the problem is related to extension installation, upgrade, uninstall or enable/disable, these suggestions should help get as much information as possible into a bug report.

Be specific in your description

While it may appear that your problem happens for “every add-on you try to install” or for “every website” the reality is that add-ons are complex things. A feature of the 20 add-ons you tried might not be present in the one add-on that the developer tests with. If you say precisely what add-ons you tried then the developers will test with those add-on which gives us a better chance of reproducing your problem.

The other part of this is to be specific in the steps to reproduce. Try to include what windows appear, what buttons you click on, when you restarted Firefox. Too much information is not a bad thing here, the more closely a developer can follow your exact steps the better.

Attach the cache files

In your profile directory are a set of files that the extension manager uses. Attaching copies of them after the problem has occured gives us an idea of what state the extension manager got itself into, ideally shut down Firefox straight after the problem and copy them out of the profile before starting Firefox again, then attach them to the bug report you file.

You want the files extensions.log (only in Firefox 3), extensions.ini, extensions.cache and extensions.rdf. If any don’t exist then say that in the report.

Turn on logging

If the problem is reproducible then you can turn on additional logging to get some more information on what the extension manager is doing. Type about:config into the address bar and look for javascript.options.showInConsole and extensions.logging.enabled. Make sure they both have the value true (double click to change).

Now open the Error Console from the Tools menu, clear it to start fresh and then perform whatever action causes the problem. Hopefully some messages will appear in the error console. Include these in the bug report.

With Firefox 3 getting ever closer to release it is time (well, past time) to start thinking about the future. The Extension Manager has seen quite a number of improvements since Firefox 2. Many were designed to be invisible, generally improving stability and fixing oddities. Some are extremely visible such as the new security requirements and the addons.mozilla.org integration. The question is, what’s the next step for add-ons management?

Here are a few ideas that are floating around my mind to get you started:

• Installing add-ons without restarts
• Presenting more detail in the install dialog
• Simplifying the UI
• Deprecating install.rdf and replacing with a simpler xml or json format
• Add-on dependencies with automatic resolution

So what have I missed? Please keep any comments restricted to add-ons management, either as a user installing and using add-ons or as a developer writing add-ons. While I am interested in the future of Firefox as a whole I don’t want this to be a mass feature request for the product.

After a fair bit of work (feels like longer than 2 months) I’ve finally managed to get bug 382752 landed. What this gives us in simple terms is a set of functions that we can use in order to do unit testing on the extension manager. Alongside I have checked in the first unit test. Now if anything regresses bug 257155 we should know about it immediately.

Ignoring the regression detection, I’ve always found unit tests to be fantastically useful when developing new code or fixing bugs. Zipwriter is a prime example, with a large number of tests that I can run by typing a single command I can test whether the changes I have made have solved the problem and not introduced any other errors.

The next step of course is to start adding unit tests for the extension manager. I have some already in progress and hopefully soon some of the key parts of the EM should be getting checked on a daily basis.

Since the disclosure of potential vulnerabilities in the way Firefox (and other Mozilla applications) automatically update your add-ons we have been discussing how to tighten up the system in a way that is hopefully unnoticeable to users and not much extra work for add-on authors.

After a process of listening to authors on the newsgroups, forums and by email we now have a rough proposal of what changes we are looking to make. There’s still a few minor details to be ironed out of course. This is mainly of interest to add-on authors since there is an impact depending on how you host your updates. I’ve started threads on the newsgroup and forums so if you want to discuss the proposal there then that’d be good. I’d prefer it if you didn’t edit the main page of the wiki but feel free to stick small comments onto the discussion page.