Planning for the future

For some time now I’ve been throwing around ideas for new features that we may want for the add-ons manager. After lots of thought and trying to take on board comments from anyone that would pipe up I decided it was high time to put together an actual plan for what features I have decided should be pursued and what order to start tackling them.

The roadmap itself should be pretty self explanatory, just remember that nothing is ever set in stone. I’ll try to keep it updated as things change, features seem less important or new items are added. But for now this is how I see the add-ons manager evolving over the course of about the next year or so, which may be the next two versions after Firefox 3.1.

Obviously the ordering of the goals may not be quite to everyone’s taste, but I’ve tried to strike a balance between getting new features done and getting groundwork laid to make things better in the future. I’m happy to take any comments people might have into consideration for altering the plan, but chances are I won’t be making any big changes unless you have some pretty damning evidence that a change is necessary 🙂

Dividing the labour

Faaborg has started a discussion on the newsgroups on the relative importance of polish and blocker bugs (I’d provide a link bug Google Groups seems to be refusing to acknowledge existence of the post). I have been for quite some time now very focused on the in-depth blocker issues, partly because that is where all the interesting work is for me, but also because I think it makes more sense for me or one of the other guys with good understandings of how the extension manager works to be making these changes rather than throwing others in at the deep end.

This has of course left little time for me to fix the trivial issues that Faaborg talks about. The things that users probably notice more and make the product look complete or half-done. This is why I’m really pleased that when I posted a list of simple bugs that anyone should be able to tackle there was a fairly good response. Of the 12 bugs on the original list, 9 have been fixed. I wanted to extend my thanks to those guys that took the time to work on those, some of them probably ended up being a little harder work than anticipated.

Of course the polish list is almost never ending. I’ve been adding more to the list of good first bugs, if anyone is interested in helping out then don’t be afraid to just email me or find me on IRC as Mossop and ask how much work something really involves.

Finding API Changes

I started a small discussion on the newsgroups the other day wondering about the numbers of API changes that had landed on trunk since 1.9.0. It was a short discussion, maybe partly because all I had to go on were the vague metrics that mercurial could give me. It also evolved a little into a discussion of how add-on developers we’re meant to find out about API changes. It’s true that as developers we are getting better at trying to announce our changes as much as possible, but inevitably we miss some, and if you don’t track the announcements what then?

Yesterday on IRC ted jokingly made the comment that I could use the idl parser than bsmedberg wrote in python a short time ago to find out more about the API changes. I’ve been hard pressed on other work lately so it wasn’t long before the prospect of knocking together a little side project had me too interested to do anything else. In a short space of time I had something that could parse all the in-tree idl files into a sqlite database that I could then query. It wasn’t long after that that I got a small web app written to display the content of the database, but I spent some more time today getting it a bit more useful before I showed it to the rest of the world.

So here it is, the API viewer. You can look at any interface in any of the platform versions that I’ve bothered to scan. And here is the more interesting bit, you can compare two platforms to see what interfaces changed. Not happy there? Well you can also compare an interface in different platform versions in a sort of diff like display.

I should note that it is only indexing the core and Firefox interfaces for now, and it’s close to indexing all of the idl files from the sources, many of which don’t make it into the final build (which makes life interesting as it turns out some of them are invalid xpidl!). So you can’t trust that interfaces you find are necessarily useful etc.

The web app is really a prototype right now. Navigation is probably a bit clunky and it’ll need more work to be really useful but I wondered what other people thought of it. So what do you think? Is it useful? What other interesting information might we be able to grab automatically like this?

Update: I’ve added details of the current trunk, 1.9.1b2pre as of today so you can see what changes are coming since the last beta. Not sure how often I’ll keep that up to date though.

How extensions can slow down Firefox (my dirty little secret)

Tab Sidebar is probably my favourite extension that I’ve created. It is certainly the most polished, thanks mostly to other people pushing me to make it so. For those that haven’t used it it creates a thumbnail preview of all of your tabs in the sidebar. The thumbnails automatically update whenever the page changes, even things like popup menus generally show up. This automatic updating comes at a cost though.

In order to detect changes to the page content the extension mostly uses DOM mutation events. These are in theory sent out whenever a change to the page’s structure happens, which covers adding/removing content, style changes, text entry and all manner of things. What I wasn’t aware of when I first used this technique was that Gecko performs certain optimisations when there are no DOM mutation listeners registered for a document (the normal case). Simply the presence of a mutation listener impacts the speed of any DOM manipulation by the page. How much? Well nothing like a graph to illustrate things.

Performance impact of mutation listeners
Performance impact of mutation listeners

These are numbers gathered from the Dromaeo test suite, averaging 5 runs with and without Tab Sidebar installed. A little perf loss is inevitable as the extension must perform some checks on the mutation and occasionally repaint the thumbnail, but nowhere near the sort of regression that actually occurs. I’ve combined some of the test results for simplicity but it is pretty clear that while the regular JS tests are essentially unaffected, the DOM tests can have wildly different results. Don’t ask me why DOM Queries get faster with the extension installed, but DOM modifications are a nice round 4 times slower.

That isn’t to say that using mutation listeners is a complete sin to be avoided at all costs. In some cases they are the only safe option. I discovered this problem some time ago and never before found a better way to detect the changes for Tab Sidebar. Last I checked Firebug uses them in similar ways. The point I think is that there are many features in the platform that can have unexpected side effects.

Of course I guess I wouldn’t be making this post if I hadn’t finally come up with a solution. While I’m no longer actively working on my extensions, I’m afraid I couldn’t resist the opportunity of the new MozAfterPaint event that landed on trunk recently. Quite simply whenever an area of the page is repainted on the screen this event is dispatched with details of what the area painted was. This is absolutely perfect for Tab Sidebar, which isn’t surprising since the use case it was written for is essentially what Tab Sidebar is doing anyway. After a few hours of ripping out most of the event handling code I now have a working prototype that redraws solely based on this event, only painting the areas that might have changed.

Performance impact of the MozAfterPaint listener
Performance impact of the MozAfterPaint listener

Boy does it work well. As you can see from the graph essentially all of the performance loss is gone. For some reason the faster DOM queries are still there (who am I to complain if my extension makes Firefox run faster!). And what’s more the previews are now updated not just for DOM changes, on OSX at least it even sees repaints of plugins. It is quite bizarre to be watching a movie on youtube and see the little thumbnail at the side showing the movie as well.

I guess now I have whetted your appetite it would be unfair not to make the test version available. Use at your own risk of course, the sheer amount of code change means there are likely some problems lurking, but you can install Tab Sidebar 2.5a1 and see how it goes. If you set the preference extensions.tabsidebar.selecteddelay to -1 then it won’t even delay before repainting and just repaint the thumbnail as soon as the main page is painted, this actually seems to work out best for me, particularly on sites with fast animations.

Ruminations on a fortnight

It’s been a hectic couple of weeks for me and I wanted to touch on a few of the highs and lows before I forget them all.

It started with flights to get me to Toronto (for those of you that don’t know I live and work out of the UK). I was speaking at the Toronto Developer Day as well as attending a Firefox team work week (two things that conflicted more than I would have liked unfortunately). I normally manage to find direct flights but this time I had to connect through Amsterdam which wasn’t too bad, even if they seem even more mad for security than Heathrow. Incidentally terminal 4 at Heathrow is miles nicer than 3 where I normally come from, even if the 6 police officers armed with automatic weapons was a little disconcerting.

The Developer Day seemed to be a success and I hope all the attendees got something out of it. I’d encourage those that didn’t to let us know where we went wrong of course, not point in us running these things if they don’t give you anything. I think the people at my session found it useful, though I suspect I really need to work on my presentation style for such things. I feel kinda bad though that I’m not more confident about speaking and so could have taken some of the massive workload that Mark Finkle took on off his shoulders.

The rest of the week was spent with the rest of the Firefox team either figuring out concrete plans for the immediate future or working on those plans, interspersed with pizza, barbecue and poutine (Who’d have thought that Canadians would mash together some of the quick takeaway foods I used to get here and serve it in a restaurant!).

Next stop was Mountain View, via Detroit this time. I have to say that Detroit is a really nice airport to change at, even if it does mean that you’re probably flying Northwest if you do so.

I had a weekend to kill so on the Saturday a friend took me to a Renaissance Faire. I thought it might be a bit too geeky even for my tastes at first but it turned out to be a lot of fun, not sure you’d ever get me dressing up to attend like many do though. Fairly fun to watch Americans celebrating a history they never had.

Sunday I headed to the Winchester mystery house. I probably wasn’t in a good mood for it to be honest, jet lag had hit me and then the tour guide was a stereotypical over the top American tour guide trying to make every little oddity out to be spooky. Still it was an interesting place to walk around and the gardens were quite spectacular.

Tuesday I headed into San Francisco proper and went to meet up with the guys at Songbird. One of the things I really want to do is avoid writing the add-ons manager for just Firefox so it was good to chat with them about what they would like to see and how important they find the future plans that we’ve already toyed with.

That evening I had a great meal with some new friends at a German restaurant (yes I had to go to California in order to eat at a German place for the first time). I also got to experience the fun of driving out of the city back to MV at eight in the morning. I think I’ll be avoiding that in the future.

Thursday it was fun to attend one of the monthly labs meetups. Pretty cool to see plans for what is coming both in experimental stuff and in Firefox itself. I think it’d be really useful to stream these events so more people can get an insight in what is coming and why. Too often we can land things in the nightlies seemingly without any discussion when in actual fact a lot of thought does go into pretty much everything.

Finally Friday morning was painful. I maybe took people too seriously when they advised me on how long it would take to get to the airport and to check in, and then I added some for safety leaving me getting up at 5am and then ending up waiting at the gate for an hour and a half.

Seriously, why are so many people carrying on multple bags rather than checking them. I suppose I understand the risk of lost luggage but it is getting really annoying when I have to stick my single carry-on bag under my feet, decreasing the already sparse space there, because the overhead lockers are filled with trolley cases.

Northwest has to be one of the worst airlines I have flown. Poor food, very little movie selection, no tv episodes available at all. Thank god I have taken to putting movies onto my iPod so I always have something decent to watch.

Now I’m sat in a hotel room trying to stay awake for a bit longer. After I get back to Heathrow, after some 20 odd hours travelling (and I can never sleep on planes) it seems to be a good idea for me to sleep before taking the final 3 hour drive back home. But I also need to start the shift back to UK time. I shall probably be waking myself at around 3am for a leisurely drive back to Wales and then slowly shifting myself a couple of hours a day for the rest of this week.

Toronto Developer Day

In case you hadn’t heard Mozilla are running a developer day in a week or so’s time in Toronto. The first day is aimed at giving a good overview of the Mozilla platform and what technologies it offers. The second day will be a more hands on workshop where you can try your hand at extensions, applications and testing with friendly experienced people around to lend a hand when you hit problems. It isn’t far off now so get yourself signed up if you’re coming.

I’m in the midst of trying to put together what I’ll be talking about so if you have anything you particularly want to hear then make a suggestion.

The great bug triage

I was very excited to learn at the Firefox Summit that Rob Strong was handing over ownership of the Extension Manager module to me. He did great work making the extension manager what it is today but has lately had to be more focused on Installer and windows integration issues.

Last week I spent a large part of my time trying to get myself up to speed on all the old filed bugs, clearing out things that clearly aren’t going to happen and trying to consolidate others. In particular I made the effort to go over every single unconfirmed bug and either resolve as appropriate or request any additional information from the reporter. It is rather sad that many of these were issues that noone commented in after the initial report or even worse the reporter responded with additional information but people (including me) dropped the ball and nothing further happened.

This big triage has been quite useful. Firstly the number of unconfirmed bugs for the add-ons manager has dropped from the mid 70s to under 30 (I would give you exact figures and pretty graphs but bugzilla’s charting is broken). I expect it to drop even further next week when I resolve the issues where the reporter is no longer responding to bugmail. Secondly it has allowed me to spot a pattern amongst 6 or so filed bugs which strongly suggests a new issue that can break add-on installations. While I have yet to be able to reproduce it in a test environment I certainly have more information to go on than previously.

Finally I have been marking all the trivial little issues that I think we could fix, but are unlikely to be priorities. These are the kinds of things that if you were interested in helping out and getting started in developing for Mozilla you could start out on. Many will end up being a few lines of patch, just need to figure out where to put it and test it.

So if you are interested in tackling any of these please give it a go. You can find me on IRC (for reasons best left alone I appear as Mossop) or by email if you need any guidance on getting started. I’m going to try to keep this list updated as I find new bugs that fit there, I still have yet to go through all of the 263 confirmed bugs, maybe a job for next week.

Add-ons Manager session notes

My head is slightly less fuzzy now and I’m almost back on UK time so I thought it would be a good opportunity to give a quick review of how the Add-ons Manager session I ran at the summit went. I won’t go into too much detail, just a look at the main points of discussion and how I felt the room reacted to the ideas. You can also see the slides I prepared or the notes that Basil kindly took for me without me needing to remember to ask. Most of the session was about things we might want to implement and what problems/benefits we might encounter.

Add-on locale packs

The idea of developing the UI for locale packs specific to extensions received a less enthusiastic response than I had expected. This may have been because there were more add-on developers and less localisers in the room, but the general feeling seemed to be that there was not a large benefit and the potential complications, particularly around updating, could cause some large problems.

Dependency resolution

The current add-on dependency implementation is a very rough start. Being able to automatically find and install dependant add-ons for the user, with clear UI to show what is happening, seemed to be something that didn’t have many downsides. There were also suggestions that AMO could handle dependencies and deliver sets of add-ons as multi-xpi packages, though this could cause issues in upgrades.

Add-on conflicts

The moment I mentioned the suggestion of a system to mark two add-ons as incompatible with each other half the room said they wanted it. It was pointed out that some add-ons were already disabling things they don’t like on install (if you hear of that happening without appropriate warning please let us know). It was also quickly realised that under the author’s control it could just become a means to avoid having to work to fix incompatibilities and to disable the competition. The suggestion of Mozilla maintaining a central list of incompatibilities seemed like a good way to go.

Install without restart

No surprise that this was seen as useful, particularly in non-Firefox applications that don’t have the wonder of session restore. We also started talking about the potential for lower impact extensions that just had a small sandbox to work in with a little piece of UI which could make loading and unloading dynamically far easier.

Install/uninstall hooks

We discussed how the current mechanisms for running setup and cleanup code for extensions were poor, there is no way to deal properly with uninstall in all situations. It was felt that add-ons should be able to provide a full uninstall script that would even run if uninstalled when disabled or in safe mode, but would not run if the add-on had been blocklisted. We also talked a little about sandboxing what this script might be able to access.

Replace RDF

I was surprised that there seemed little to no resistance to the prospect of introducing a new plainer XML form for install.rdf and update.rdf given some of the comments I had when I blogged about it last. We would of course still support the old forms for a time and make converters between the forms available. There didn’t appear to be any benefit in using JSON instead of XML.

Much of the rest of the slides were skimmed over as we had unfortunately run out of time by this point, but thankfully that was most of the more important stuff completed anyway. For the immediate future I think introducing replacements for RDF into Firefox 3.1 is potentially on the cards. None of the other items were ruled out, far from it, though the locale packs seemed less interesting to everyone, and I think there needs to be some very careful thought about the prospect of running an uninstall hook for an add-on in safe mode or when disabled.

Thanks Canada (no, really)

It’s 6:30am Sunday UK time (though in my weird method of getting over the jet lag that works out as just after lunchtime) and I’ve just got home to Swansea. As many others are I’ve been thinking back over the great times I’ve had over the past two weeks. Needless to say I got out alive, as I believe everyone did (though apparently some are still stranded in Vancouver), despite Canada’s attempts to break us.

After a short visit with my sister trying to avoid getting licked to death by her new puppy I spent a week in Toronto at the Mozilla offices there. Thanks guys for having me, it was a fun week and I’m sorry I brought the Welsh rain over with me. Although I do have a shared office that I work from for most of the week I forget how good it is to actually be in a place where others are working on the same stuff. Maybe I’ll give in to Beltzner and move out there.

The summit was nothing short of amazing. So much larger than the last one and so much more enjoyable for it. As with all of these events I never got chance to do the things I had planned on and missed meeting many of those I wanted to, but met others I had never known and did more interesting things anyway. Dan and the rest of the organising team should be seriously proud with how the event went and how they coped with the upsets.

I think the Add-ons Management session I ran on Tuesday went pretty well all things considered. I have never really done interactive sessions before and combined with the hangover I was sporting I was a little nervous about it. Thankfully you can always rely on the Mozilla community to be talkative so in the end we actually ran out of time to talk over some of the ideas for the future of the Add-ons manager. Hopefully those there found it useful, I know I will take a lot from being able to get feedback and suggestions on my plans.

I guess the only minor disappointment (which is always the case since it would be impossible to avoid) is that there were many sessions I wanted to attend but conflicted with others. I hope there are some notes around for some of them so I can skim over what I missed.

Somehow I managed to find the journey home easier than normal. I suspect it helps that there were so many of us at the airport and I did the journey in a couple of stages to avoid the drive home immediately after getting off a plane which always turns into a dangerous race between traffic and my body trying to fall asleep.

I forgot how cool it is to drive down the motorways at 4am in the pitch black with nothing but good music and your thoughts to enjoy. Now, if only there were some shops open so I could actually get some lunch and a decent cup of English tea…

Keeping Track

In the past year or so that I’ve been working for Mozilla I’ve found myself slowly working my way through a bunch of different ways to keep track of all of the work on my plate. I still don’t think I’ve found the best way so I wondered what other people do to manage such things?

For a rough idea of my work, most of my work is bug oriented. Either bugs that I want to work on when I get some time, bugs I am actively working on, bugs I am waiting for review on, bugs I am reviewing, bugs I am waiting to check in, bugs I want to backport to a branch… and so on. Then there is other work like planning future work and working with extension authors to help resolve their problems.

A few different approaches I’ve tried:

Bug ASSIGNED status

Simplest of all, gives me a long list of stuff I am looking at. Totally fails for prospective work though and the list gets somewhat daunting.

Status whiteboard flags

I tried for a while sticking special flags into the status whiteboard that could then be spotted in special bug queries so I could see what state each bug was in. This kind of worked, though I imagine irritated some with bugspam as I maintained the flags and a couple of times other people changed the flags.

Bugzilla tagging

Bugzilla has this tagging feature that basically lets you create named buglists. Basically like using the whiteboard stuff without having to use the whiteboard. Sadly Bugzilla’s tagging UI sucks really hard.

Bookmark tagging

With Firefox 3 I can bookmark my bugs and tag them, and use smart bookmark folders as my buglists right? Well sort of, the folder list isn’t really enough detail and there are issues with manually created smart folders that kept losing my bookmarks.

Task management software

Finally bit the bullet and stuck all of this stuff into some task managent software (Omnifocus as it happens). This is so far going ok but seems to require more dedication to keep the list up to date, I think because of having to switch to a different application so much.

So what are other people doing?