Oxymoronical

The extension I’ve been working on in my spare time for the past couple of weeks is now available as a first (hopefully not too buggy) release. It lets you open WebApps in Thunderbird, properly handling loading new links into Firefox and making all features like spellchecking work in Thunderbird (most other extensions I found didn’t do this). You can read more about the actual extension at its homepage.

Mostly I’ve been really encouraged during the development of this at just how far our platform has come for developing restartless add-ons. When we first made it possible in Firefox 4 there was a whole list of things that were quite difficult to do but we’ve come a long way since then. While there are still things that are difficult there are lots of things that are now pretty straightforward. My add-on loads simple XUL overlays, style overlays, installs JS XPCOM components with category manager registration, all similar to older add-ons. In fact I’m struggling to think of things that it is still hard to do though I’m sure other more prolific developers will have plenty of comments on that!

The other thing I’ve been doing with this extension is experimenting with git and GitHub. I think it’s been an interesting experience, there are continual arguments over which is better between git and mercurial with many pros and cons listed. I think most of these were done some time ago before mercurial and git really matured because from what I’ve seen there is really little difference between the two. They have slightly different default branching styles, but both can do the same kind of branching that the other can if you want and there are a few other minor differences but nothing that would really make me all that bothered over deciding which to use. I think the only place where git has a bonus is with GitHub, and really as far as I can see there isn’t a reason why someone couldn’t develop a similar site backed by mercurial repositories, it’s just that no-one really has.

GitHub is pretty nice with built-in basic issue tracking and documentation though it still has some frustrating issues. It seems odd for example that you can’t fork your own project, only someone else’s, but that’s only a minor niggle really. As project hosting goes I can’t say I’ve come across anything better that I can remember.

After my last post about how I no longer have the time to maintain Nightly Tester Tools I am happy to say that someone has stepped forward to take over development and maintenance. I’m even more happy that it is one of the Mozilla teams so finally NTT is going to be officially owned and supported by Mozilla.

I sat down with them last week to talk over what I saw as the key goals of NTT and some ideas I had had suggested to me for the future but they also want to hear from you so go and read about their plans and give them your thoughts.

Many of you nightly testers may have noticed that Nightly Tester Tools’ compatibility override feature doesn’t work with the new add-ons manager and may be wondering when I’m planning to issue an update to fix that. The more astute of you may have noticed that there hasn’t actually been a real code update to Nightly Tester Tools in 2 years, barring a couple of simple app compatibility fixes. Those with a sharp memory will remember that I said just under 2 years ago that I was ceasing work on my extensions in my spare time. I suggested that Nightly Tester Tools might still receive the odd update but obviously that hasn’t happened and the truth is that I can’t see it happening anytime soon. I’m too busy with that whole real life thing to even be able to work on projects I do enjoy, let alone maintain old stuff that no longer really interests me.

This unfortunately leaves a sizable number of users losing a feature that they liked and still potentially have a need for. I can only really see a couple of possible roads to follow from here:

1. Do nothing. Users will be annoyed for a time but eventually find ways around what they needed NTT for.
2. Find someone else to pick up and maintain NTT. I’ve had numerous requests for the source code for many of my extensions over the past two years, none have ever apparently tried to do anything though. Perhaps someone out there will pick up the torch this time?
3. Point all the users to something else, like the Add-on Compatibility Reporter (once that is updated to work on trunk). While nothing else I know of works quite like NTT at least something is better than nothing, and ACR has the benefit of being Mozilla supported, provides Mozilla with valuable information about add-on compatibility and may be rolled into Firefox at some point.

Option 3 is the only one available that involves any work on my part but probably the choice that leads to less user annoyance, unless someone reading this wants to take up the challenge or has a better idea?

Update: Part of the Mozilla QA team are going to take over development and maintenance of Nightly Tester Tools, let them know what you want to see!

Tab Sidebar is probably my favourite extension that I’ve created. It is certainly the most polished, thanks mostly to other people pushing me to make it so. For those that haven’t used it it creates a thumbnail preview of all of your tabs in the sidebar. The thumbnails automatically update whenever the page changes, even things like popup menus generally show up. This automatic updating comes at a cost though.

In order to detect changes to the page content the extension mostly uses DOM mutation events. These are in theory sent out whenever a change to the page’s structure happens, which covers adding/removing content, style changes, text entry and all manner of things. What I wasn’t aware of when I first used this technique was that Gecko performs certain optimisations when there are no DOM mutation listeners registered for a document (the normal case). Simply the presence of a mutation listener impacts the speed of any DOM manipulation by the page. How much? Well nothing like a graph to illustrate things.

Performance impact of mutation listeners

These are numbers gathered from the Dromaeo test suite, averaging 5 runs with and without Tab Sidebar installed. A little perf loss is inevitable as the extension must perform some checks on the mutation and occasionally repaint the thumbnail, but nowhere near the sort of regression that actually occurs. I’ve combined some of the test results for simplicity but it is pretty clear that while the regular JS tests are essentially unaffected, the DOM tests can have wildly different results. Don’t ask me why DOM Queries get faster with the extension installed, but DOM modifications are a nice round 4 times slower.

That isn’t to say that using mutation listeners is a complete sin to be avoided at all costs. In some cases they are the only safe option. I discovered this problem some time ago and never before found a better way to detect the changes for Tab Sidebar. Last I checked Firebug uses them in similar ways. The point I think is that there are many features in the platform that can have unexpected side effects.

Of course I guess I wouldn’t be making this post if I hadn’t finally come up with a solution. While I’m no longer actively working on my extensions, I’m afraid I couldn’t resist the opportunity of the new MozAfterPaint event that landed on trunk recently. Quite simply whenever an area of the page is repainted on the screen this event is dispatched with details of what the area painted was. This is absolutely perfect for Tab Sidebar, which isn’t surprising since the use case it was written for is essentially what Tab Sidebar is doing anyway. After a few hours of ripping out most of the event handling code I now have a working prototype that redraws solely based on this event, only painting the areas that might have changed.

Performance impact of the MozAfterPaint listener

Boy does it work well. As you can see from the graph essentially all of the performance loss is gone. For some reason the faster DOM queries are still there (who am I to complain if my extension makes Firefox run faster!). And what’s more the previews are now updated not just for DOM changes, on OSX at least it even sees repaints of plugins. It is quite bizarre to be watching a movie on youtube and see the little thumbnail at the side showing the movie as well.

I guess now I have whetted your appetite it would be unfair not to make the test version available. Use at your own risk of course, the sheer amount of code change means there are likely some problems lurking, but you can install Tab Sidebar 2.5a1 and see how it goes. If you set the preference extensions.tabsidebar.selecteddelay to -1 then it won’t even delay before repainting and just repaint the thumbnail as soon as the main page is painted, this actually seems to work out best for me, particularly on sites with fast animations.

I guess it isn’t going to come as much of a surprise to people when I say I’ve been having a hard time finding the time to maintain the extensions I develop. I’ll go over some of the reasons why shortly, but I guess the highlight (or lowlight depending on your point of view) of this post is that I’m no longer going to really put in any effort to keep them updated.

Nightly Tester Tools will probably continue to get the odd update here and there where I find something interesting to add, and Update Channel Selector is due to get an update to make it actually work on most platforms, but otherwise I would not expect to see any further updates for any of my other extensions.

I came to this conclusion last week when doing some updates and I realised that I really wasn’t enjoying it. The things I generally enjoy doing in my spare time are things that are new and interesting. All of my main extensions were born out of finding something cool to do with the platform. Fast ways to perform regular expressions on large html documents, painting thumbnails of webpages (back when it was actually new and exciting), taking enough control of the toolbar customisation service to be able to dynamically add many instances of the same widgets. All of this stuff was interesting enough to make me knock up a quick extension to play with it. When they seemed useful I would go through and try to iron out some of the rough edges to make it something I’d consider release-worthy. Only Tab Sidebar had real effort put into it to try to make it into a real user friendly product.

I’ve also had quite a large change in my life since I last really actively worked on my extensions. Back then I was a small business owner, most of my day to day work was IT support and web development. Getting home and sitting down for a few hours hacking away on extensions worked as quite a diversion, completely taking my mind off the stresses at work. Now I work for Mozilla. Working on extensions is basically no different from what I do during the day. It is no longer a diversion, it is just like work. How many of you like to get home from work, kick back and then go back over more work to relax?

These things, as well as other issues, have left me really uninterested in doing anything except the exciting new stuff. Turning that into a usable tool, fixing bugs, making things work in new releases, all of that is just dull and no longer worth the spare time it loses me.

And that is pretty much that. I’m not totally out of the extensions game, while I might still create a few experimental add-ons in the future, they probably won’t be fully useful, just little things for fun. I’m not really going to be interested in hearing about bugs in my existing stuff or that it hasn’t yet been updated to the latest version of Firefox. I’m not going to do anything about it. Of course all of my add-ons are open source, under the MPL/GPL/LGPL tri-license, and I’d happily talk to anyone thinking of taking over official development of them for the future.

To those of you who have kept using my extensions and let me know how much you liked them I’d like to say thank you, and I’m sorry that there will be nothing more even though I know you were hoping. It’s taken me a while to reach this decision and I’m sure some will feel a little left in the lurch by it, but I can’t keep fooling myself that I’ll find the time to put in the effort you all deserve.

I’ve finally taken the plunge and switched my website to a more modern blogging software (WordPress) and a dedicated media gallery (Gallery 2). Hopefully through the magic of redirects most shouldn’t notice much difference. I just hope planet hasn’t decided to dump all my posts onto the front page, if it has then I apologise.

I’ve also taken this opportunity to move all my extensions to addons.mozilla.org. Most are currently still in the sandbox, hopefully they will come public soon. I’m also using the nice new AMO API service to populate details on the add-on homepages, means there is only one place I need to make most changes to.

One of the main parts of my work for Mozilla at the moment is about securing add-on updates. The spec is now pretty near complete and the implementation is also pretty much complete so hopefully we can start pushing out the necessary tools to add-on authors real soon then land the work shortly after.

Of course it wouldn’t be right for me to push this out without first making my own extensions comply with the new requirements. So today I am rolling out updates to all of them, mostly just changing the update url to an SSL one, though a couple of the extensions (Nightly Tester Tools and /Find Bar/) have some additional updates.

Using SSL really will be the easiest way of hosting secure updates for your extensions and I urge you to use it. Assuming you have a sensible hosting package, adding SSL is really not as expensive as many expect. Godaddy offer SSL certificates for $18 per year (minimum of 2 years) and if you are like me and hosting open source extensions then you can get the first year for free (though that seems to take a few weeks longer than if you pay). It’s also pretty simple to set up assuming you have a decent webhost, Dreamhost just has one form to fill in.

It turns out that the hardest part of getting SSL was fixing the bugs in my CMS since it’s current version had never been used in an SSL environment before

For some time now I’ve promised myself that I’d sort out a simple system to view stats about how many people are using my extensions. The idea is a simple one, on a daily basis Firefox (or whatever app) will ping my site checking for an update for the extension. Counting the number of checks in a day gives you a rough idea of the number of users. You can’t take the numbers literally of course but as ballpark figures go it’s probably not bad.

Finally I have got around to doing it and there have been some interesting results. Not surprisingly Nightly Tester Tools is my most popular extension. However it’s distressing to see how many updated to the broken 1.3b1 and still haven’t gone to 1.3b2. Not surprising of course,  but it makes me wonder when these 30,000 people or so will update again.

The most surprising result is that of the second most popular extension. Turns out it’s JavaScript Options, something I had never considered to be popular at all based on the number of questions/complaints I get about it.I may well have to reconsider my decision to cease further work on it.

Then of course you get the freaky results. Who’d have though that people on Solaris are using my extensions. And even more bizarre, why has someone installed Tab Sidebar into Thunderbird?

You can peruse the stats yourself if you like, there’s a few different views to play with. Some of the old data isn’t highly accurate, my back-filling script needs some work, and many of the extensions simply aren’t reporting the more detailed information about OS and version so some of those graphs are a little misleading, still there it is.

I understand that AMO are planning on rolling out some stats for add-on authors based on the same update pings that I use. I urge all authors to take a good look at them when they do, you never know what you might find that surprises you and makes you re-evaluate your priorities for your extensions.

I don’t normally do announcements of new versions of my extensions here but unfortunately there was a problem with the packaging of the last version of Nightly Tester Tools so if you want to get the latest and greatest version then you’ll have to download it manually.

Sorry, I’ll try not to let it happen again.

David Baron recently put out the call for there to be more leak testing on trunk builds.  We have a number of regressions that we need to find and fix before Firefox 3 goes out the door. The best way to do leak testing has always been to use his excellent leak-gauge script which catches more leaks than the leak monitor extension. Unfortunately the leak-gauge script is not the easiest of beasts to play around with.

Thankfully for some time now Nightly Tester Tools has had the leak-gauge script included in it to hopefully make this process a bit easier. I discovered today that that functionality has actually been broken on trunk builds so make sure you pick up the latest version 1.3b1 if you want to use it (don’t worry i should be pretty stable as betas go). Once installed just go to Tools – Nightly Tester Tools – Analyse Leak Log. Tell it where to find your log and it will create a leak report that you can copy to the clipboard at the touch of a button.

Unfortunately you still need to actually create the leak log, I suggest a script something like this (will need adapting for other platforms):

cp -f nspr.log lastnspr.log

export NSPR_LOG_MODULES=DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5

export NSPR_LOG_FILE=/Users/dave/nspr.log

/Applications/Minefield.app/Contents/MacOS/firefox-bin

Then you can analyse lastnspr.log while still logging the current run. In Firefox you can even set the log analyser as the home page so it automatically loads the last log on startup.

Note that you can’t analyse the log being generated by a currently running app, nor one from a crashed session in an app, both will look hideously leaky. Please be sure to file good memory leak bugs and good leak hunting.