Nightly Tester Tools is being brought back to life

After my last post about how I no longer have the time to maintain Nightly Tester Tools I am happy to say that someone has stepped forward to take over development and maintenance. I’m even more happy that it is one of the Mozilla teams so finally NTT is going to be officially owned and supported by Mozilla.

I sat down with them last week to talk over what I saw as the key goals of NTT and some ideas I had had suggested to me for the future but they also want to hear from you so go and read about their plans and give them your thoughts.

How to extend the new Add-ons Manager (or how I built a simple greasemonkey clone in an evening)

One of the goals of the new add-ons manager API was to create something that was itself extensible. A couple of times in the past we’ve had to add new types of add-ons to the UI like Plugins and Personas. In both cases squeezing them into the UI was something of a kludge involving a bunch of custom code for each case. We already have a number of new types of add-ons that we want to add, things like search plugins which are currently managed by their own custom UI.

To help simplify this the API is largely type-agnostic. Internally it uses a number of so-called add-on “providers”, each of which may make information about add-ons available. Each provider is basically a JavaScript object with functions defined that the API can call to ask for information about add-ons that the provider knows about. The provider then just has to pass back JavaScript objects to represent each add-on. Each of these must have at minimum a set of required properties and functions and may also include a set of optional properties. The full set is defined in the API documentation.

With this design the user interface doesn’t need to care about implementation details of any of the providers, how they store their data or what exactly their add-ons are and do. Because each gives objects that obeys the same interface it can just display and manipulate them.

To try to show this all off I recently put together a small demo extension for the Mozilla Summit that registers a new type of add-on to be displayed in the main add-ons manager. This is a short overview of some of the highlights and I’ll make the code available for people to look at and take examples from. The add-on was a basic implementation of Greasemonkey allowing user scripts to be installed, managed through the add-ons manager and do it all as a restartless add-on.

Making a restartless add-on

Add-ons don’t have to be developed with Jetpack to make them restartless, although the Jetpack SDK certainly makes things easier on you, at the expense of less access to the internals of the platform.

The first thing to learn about making a restartless add-on is that you can forget about using XUL overlays or registering XPCOM components to be called at startup. Neither are supported at the moment, and maybe never will. Instead you have to provide a bootstrap script. This is a simple “bootstrap.js” file in the root of the extension that should include a “startup” and “shutdown” function. These are called whenever Firefox wants to start or stop your add-on either because the application is starting up or shutting down or the add-on is being enabled or disabled. You can also provide “install” and “uninstall” methods to be notified of those cases but that is probably unnecessary in most cases.

At startup the demo extension does some basic things. It registers for some observer notifications, registers a new add-on provider (I’ll talk more about that below) and does a little work to include itself in the add-ons manager UI (again, see below).

The rule is this. Anything your add-on does after being started must be undone by the shutdown function. The shutdown function often ends up being the inverse of startup, here it removes observer notification registrations, unregisters the add-on provider and removes itself from the UI. It also shuts down a database if it was opened.

Implementing a new provider

This extension implements probably the simplest possible provider. As far as the API goes all it supports is requesting a list of add-ons by type or a single add-on by ID. These functions pass add-on objects to the callbacks. For this add-on these objects are held in a database so that code does some fairy uninteresting (and horribly synchronous) sql queries and generates objects that the API expects.

Adding new types to the UI

Perhaps the hardest part of this extension is getting the new type of add-on to display in the UI. Unfortunately one thing that we haven’t implemented so far is any kind of auto-discovery of add-on types. Instead the UI works from a mostly hardcoded list. This is something that we think it would be nice to change but at the moment it seems unlikely that we wiull get time to before Firefox 4, unless someone wants to volunteer to do some of the work.

The demo extension works around this restriction by inserting some elements into the add-ons manager window whenever it detects it opening. In particular it adds an item to the category list with a value attribute “addons://list/user-script”. The add-ons manager UI uses this kind of custom URL to decide what to display when a category is selected. In this case it means displaying the normal list view (that plugins and extensions currently use) and to ask the API for add-ons of the type “user-script”. There is also some code there that overrides the normal string bundle that the manager uses to localize the text in the UI to allow adding in some additional strings. The code I am showing is of course badly written in that it is hardcoded and so could not be localized, please forgive me for cutting corners with the demo.

That is basically all that is needed to have the UI work to display the new add-ons from the registered provider however the demo also throws in some style rules to pretty things up with a custom icon

Notifying the UI of changes

When you implement your own provider you have to be sure to send out appropriate notifications whenever changes to the add-ons you manager happen so that any UI can update accordingly. I won’t go into too much detail here, hopefully the AddonListener and InstallListener API covers the events you need to know about enough. You can see the script database send out some of these notifications.

Get the full code

This has been a very short overview of the highlights of this demo, hopefully enough for the interested to pick up the code and make use of it themselves. The full source of the extension is available from the mercurial repository. Right now I wouldn’t really release this as an extension. As I’ve mentioned it uses synchronous sql queries (on every page load no less!) and cannot be localized. These things can be fixed but this was just made as a demo in basically one evening to show off the sorts of things that are possible with the new add-ons manager.

Where is the updated Nightly Tester Tools?

Many of you nightly testers may have noticed that Nightly Tester Tools’ compatibility override feature doesn’t work with the new add-ons manager and may be wondering when I’m planning to issue an update to fix that. The more astute of you may have noticed that there hasn’t actually been a real code update to Nightly Tester Tools in 2 years, barring a couple of simple app compatibility fixes. Those with a sharp memory will remember that I said just under 2 years ago that I was ceasing work on my extensions in my spare time. I suggested that Nightly Tester Tools might still receive the odd update but obviously that hasn’t happened and the truth is that I can’t see it happening anytime soon. I’m too busy with that whole real life thing to even be able to work on projects I do enjoy, let alone maintain old stuff that no longer really interests me.

This unfortunately leaves a sizable number of users losing a feature that they liked and still potentially have a need for. I can only really see a couple of possible roads to follow from here:

  1. Do nothing. Users will be annoyed for a time but eventually find ways around what they needed NTT for.
  2. Find someone else to pick up and maintain NTT. I’ve had numerous requests for the source code for many of my extensions over the past two years, none have ever apparently tried to do anything though. Perhaps someone out there will pick up the torch this time?
  3. Point all the users to something else, like the Add-on Compatibility Reporter (once that is updated to work on trunk). While nothing else I know of works quite like NTT at least something is better than nothing, and ACR has the benefit of being Mozilla supported, provides Mozilla with valuable information about add-on compatibility and may be rolled into Firefox at some point.

Option 3 is the only one available that involves any work on my part but probably the choice that leads to less user annoyance, unless someone reading this wants to take up the challenge or has a better idea?

Update: Part of the Mozilla QA team are going to take over development and maintenance of Nightly Tester Tools, let them know what you want to see!

How extensions can slow down Firefox (my dirty little secret)

Tab Sidebar is probably my favourite extension that I’ve created. It is certainly the most polished, thanks mostly to other people pushing me to make it so. For those that haven’t used it it creates a thumbnail preview of all of your tabs in the sidebar. The thumbnails automatically update whenever the page changes, even things like popup menus generally show up. This automatic updating comes at a cost though.

In order to detect changes to the page content the extension mostly uses DOM mutation events. These are in theory sent out whenever a change to the page’s structure happens, which covers adding/removing content, style changes, text entry and all manner of things. What I wasn’t aware of when I first used this technique was that Gecko performs certain optimisations when there are no DOM mutation listeners registered for a document (the normal case). Simply the presence of a mutation listener impacts the speed of any DOM manipulation by the page. How much? Well nothing like a graph to illustrate things.

Performance impact of mutation listeners

Performance impact of mutation listeners

These are numbers gathered from the Dromaeo test suite, averaging 5 runs with and without Tab Sidebar installed. A little perf loss is inevitable as the extension must perform some checks on the mutation and occasionally repaint the thumbnail, but nowhere near the sort of regression that actually occurs. I’ve combined some of the test results for simplicity but it is pretty clear that while the regular JS tests are essentially unaffected, the DOM tests can have wildly different results. Don’t ask me why DOM Queries get faster with the extension installed, but DOM modifications are a nice round 4 times slower.

That isn’t to say that using mutation listeners is a complete sin to be avoided at all costs. In some cases they are the only safe option. I discovered this problem some time ago and never before found a better way to detect the changes for Tab Sidebar. Last I checked Firebug uses them in similar ways. The point I think is that there are many features in the platform that can have unexpected side effects.

Of course I guess I wouldn’t be making this post if I hadn’t finally come up with a solution. While I’m no longer actively working on my extensions, I’m afraid I couldn’t resist the opportunity of the new MozAfterPaint event that landed on trunk recently. Quite simply whenever an area of the page is repainted on the screen this event is dispatched with details of what the area painted was. This is absolutely perfect for Tab Sidebar, which isn’t surprising since the use case it was written for is essentially what Tab Sidebar is doing anyway. After a few hours of ripping out most of the event handling code I now have a working prototype that redraws solely based on this event, only painting the areas that might have changed.

Performance impact of the MozAfterPaint listener

Performance impact of the MozAfterPaint listener

Boy does it work well. As you can see from the graph essentially all of the performance loss is gone. For some reason the faster DOM queries are still there (who am I to complain if my extension makes Firefox run faster!). And what’s more the previews are now updated not just for DOM changes, on OSX at least it even sees repaints of plugins. It is quite bizarre to be watching a movie on youtube and see the little thumbnail at the side showing the movie as well.

I guess now I have whetted your appetite it would be unfair not to make the test version available. Use at your own risk of course, the sheer amount of code change means there are likely some problems lurking, but you can install Tab Sidebar 2.5a1 and see how it goes. If you set the preference extensions.tabsidebar.selecteddelay to -1 then it won’t even delay before repainting and just repaint the thumbnail as soon as the main page is painted, this actually seems to work out best for me, particularly on sites with fast animations.

Throwing in the Towel

I guess it isn’t going to come as much of a surprise to people when I say I’ve been having a hard time finding the time to maintain the extensions I develop. I’ll go over some of the reasons why shortly, but I guess the highlight (or lowlight depending on your point of view) of this post is that I’m no longer going to really put in any effort to keep them updated.

Nightly Tester Tools will probably continue to get the odd update here and there where I find something interesting to add, and Update Channel Selector is due to get an update to make it actually work on most platforms, but otherwise I would not expect to see any further updates for any of my other extensions.

I came to this conclusion last week when doing some updates and I realised that I really wasn’t enjoying it. The things I generally enjoy doing in my spare time are things that are new and interesting. All of my main extensions were born out of finding something cool to do with the platform. Fast ways to perform regular expressions on large html documents, painting thumbnails of webpages (back when it was actually new and exciting), taking enough control of the toolbar customisation service to be able to dynamically add many instances of the same widgets. All of this stuff was interesting enough to make me knock up a quick extension to play with it. When they seemed useful I would go through and try to iron out some of the rough edges to make it something I’d consider release-worthy. Only Tab Sidebar had real effort put into it to try to make it into a real user friendly product.

I’ve also had quite a large change in my life since I last really actively worked on my extensions. Back then I was a small business owner, most of my day to day work was IT support and web development. Getting home and sitting down for a few hours hacking away on extensions worked as quite a diversion, completely taking my mind off the stresses at work. Now I work for Mozilla. Working on extensions is basically no different from what I do during the day. It is no longer a diversion, it is just like work. How many of you like to get home from work, kick back and then go back over more work to relax?

These things, as well as other issues, have left me really uninterested in doing anything except the exciting new stuff. Turning that into a usable tool, fixing bugs, making things work in new releases, all of that is just dull and no longer worth the spare time it loses me.

And that is pretty much that. I’m not totally out of the extensions game, while I might still create a few experimental add-ons in the future, they probably won’t be fully useful, just little things for fun. I’m not really going to be interested in hearing about bugs in my existing stuff or that it hasn’t yet been updated to the latest version of Firefox. I’m not going to do anything about it. Of course all of my add-ons are open source, under the MPL/GPL/LGPL tri-license, and I’d happily talk to anyone thinking of taking over official development of them for the future.

To those of you who have kept using my extensions and let me know how much you liked them I’d like to say thank you, and I’m sorry that there will be nothing more even though I know you were hoping. It’s taken me a while to reach this decision and I’m sure some will feel a little left in the lurch by it, but I can’t keep fooling myself that I’ll find the time to put in the effort you all deserve.

Extension Updates

Obviously for those keeping up with this rather minimal blog I’m somewhat behind where I hoped to be since my last post. Still no new release of Nightly Tester Tools so I thought it worth giving a quick roundup of the few extension’s I’m still working actively on and what’s happening with them:

Nightly Tester Tools is getting the main development work right now. My previous few weekends have been spent working on coding zip support so that it can update an extension’s compatibility info on the fly in the xpi file. This will allow the Mozilla extension manager to do it’s job, which is handling the safe install of the extension itself. Right now I’ve slowed down because I’m about to start talking to Mozilla about getting the zip writing component into the Mozilla platform which will undoubtedly require some API changes. That and I don’t have a Linux or Windows platform to do a build of the component on. If anyone wants to volunteer to assist (you need to already have experience of building a mozilla app) then please get in touch.

Tab Sidebar is an extension that many are waiting for the updates that I currently have in development. A lot of it is there and working, unfortunately there are also some broken bits so it’s really a case of finding the time to get those fixed.

/Find Bar/ is now totally broken on trunk builds of Firefox, but should still be ok on 2.0.0.x builds, though I have had a few error reports and suggestions on how to proceed. The next steps are to make this handle block content in pages as paragraphs and then to try to find and fix what’s causing a rare crash. I also need to work out how toadd to the new find bar widget on trunk without duplicating too much of it, but so far that’s proving to be tough.

Toolbar Thinger is not receiving a lot of attention. I have had some good reports that people are finding it very useful, also a couple of reports of problems unfortunately not in enough detail for me to be able to track the issues down. At some point I want to find a nice icon for it, do a full track through for any issues I’ve missed and then do a first proper release.

Update Channel Selector was always a very simple extension and so unsurprisingly it’s not needed a lot of work. Unfortunately it seems that Vista has changed that. In order to change the channel, the extension has to overwrite a file in the application’s installation directory. Vista doesn’t let you do that as easily as previous versions of Windows did. Unfortunately I do not have a Vista machine to play with so resolving that could take time.

Finally I have taken the step of marking JavaScript Options and Cookie Extras as officially discontinued. It’s unfortunate but I do all this work in my spare time and as you can see from above, I simply do not have enough of it to be able to keep up with the extensions that I want to spend time on, let alone those that I have lost interest in. I will continue to make the extension’s available on my site and I have decided that if enough people let me know that the extension works flawlessly in newer versions of the application then I will update the compatibility information (I am about to do this for Cookie Extras). However I will be releasing no new versions of these extensions.

Since both those extensions are open source (as are all my extensions) if anyone particularly wants to take over development of them then I am happy to discuss that possibility.