Oxymoronical

One of the nice new features in the fast evolving HTML 5 spec is support for specific video and audio tags, replacing the more generic object and embed tags that have been used in the past. They have many benefits over the object tag such as a well defined JavaScript interface, controls provided by the browser and support for multiple formats for the browser to choose between for display. The current beta version of Firefox 3.1 supports this and includes support for playing the Ogg format with Theora and Vorbis codecs included. Opera has support underway as well and it looks like the latest Safari releases also have it (though it seems broken in some respects).

I’ve been trying to find a suitable video format for putting my fractal animations online for some time. Pretty much whatever format you choose you suffer because some platform or other doesn’t have a plugin for it by default. If all browsers are slowly getting ogg support though then that seems to be the ideal choice, so I have re-encoded all the animations as ogg files. Unfortunately it seems that my media gallery software (Gallery2) doesn’t support Ogg, and of course it doesn’t use the new video tag either. This is the world of open source though, so after a bit of tinkering I have it all working nicely. It can generate thumbnails for the animations using ffmpeg (which is already used for other video formats anyway) and will display the video using the video tag.

At some point I might try to push this upstream to the Gallery2 guys, but I after spending some time on their site and finding nothing helpful like a bug tracker or a way to pass patches to them I’m not sure when I’ll get round to it. Until then here are the rough instructions for applying the patch and some other changes I had to do. I’m sure there is some way to automate it all but for this one off case it wasn’t really worth my time.

• Apply the patch in your gallery install directory: “patch -p0 <patchfile“
• Run the following sql in the gallery database: “INSERT INTO g2_MimeTypeMap (g_extension, g_mimetype, g_viewable) VALUES ('ogg', 'video/ogg', 1);” (adjust for the table and column prefixes that you use).
• Deactivate the ffmpeg plugin (if it is enabled), and then activate it (assuming you want thumbnail generation to work).
• Delete the database cache in the maintenance section of the gallery admin.

Of course this means that now even fewer people can see the animations I guess. So if you want to see them then you’ll just have to try out the latest Firefox betas!

Oh, watch out for old versions of ffmpeg. The default version on my webhost (dreamhost) crashes when attempting to make a thumbnail of an ogg file. A build of the latest version of ffmpeg works though.

Apparently I’ve been talking a little too much about interfaces lately

Word cloud

I’ve just rolled out a small update to the interface cross reference, nothing major, just fixing a few bugs and I’ve put up what looks like the final set of interfaces for 1.9.1b2.

I’ve now figured out the best way to gather the interface lists and so the cross reference now includes all interfaces used in all 3 major platforms of Firefox. It is relatively simple for me to add the interfaces for other applications now but this got me thinking about what kind of uses people are making of this and how the multi-OS, multi-app interfaces should be presented. A few ideas came to mind:

• App developers probably just want to see the interfaces in their app, either only those that appear for all operating systems or those for a specific OS in some cases.
• Add-on developers probably want to see some kind of shared set of interfaces (what you might call gecko/toolkit) that appear for all apps they are targeting, unfortunately this changes depending on the set of apps. Again they probably want for all OS or for a specific OS.
• Maybe I shouldn’t even bother doing anything and just include a note in the interface display about where the interface actually exists.

I’m sure other people have other ideas of how it might work, any suggestions?

I forgot to add to my last post some information about what I currently had in mind to improve about the API reference I’ve been working on. Any further suggestions people have are very welcome, but here is the current few ideas I have.

Currently it only indexes Firefox and Gecko interfaces, in fact it only indexes those used by an OSX build of Firefox. It would be nice to be able to index interfaces for all the main applications and for all the platforms. There are a couple of issues with this. Firstly I’d want the UI to work so people viewing it could select what set of interfaces they wanted to see. Secondly actually building the lists of interfaces for each application and OS is going to be pretty tricky I think. So far I’ve resorted to building Firefox then looking in dist/idl. This works but it is time consuming. Possibly I might be able to come up with some clever nonsense to make the build system only compile idls but we’ll see.

It would be nice to be able to include a list of the component contract ids available in the platform and what interfaces they implement. This is I think pretty difficult to do just by parsing the source files. Potentially a little extension could run in the app and do some looking through Components.classes to find this out. One issue here though is there is no way to tell whether components are services or not so instantiating them is potentially problematic. This is a problem that could be solved by brute force, or maybe doing a special instrumented build.

So those are my main two ideas, if anyone has any suggestions on how to make those tasks easier or for other things that should be there then shout them out.

Since my first announcement about my little api reference tool I’ve slowly been working on updates to make it more useful and easier to navigate around. I’ve now gone and pushed the latest version live. A few of the new features:

• The main and platform lists of interfaces are split by the module where the interface lives.
• Interface lists have a quick filter box, type in it to filter the list quickly.
• Any interfaces used as return types or parameters are now links to get you straight to the information about them.
• Full IDL support so it now lists all the special attributes and for the uninitiated they have tooltips that explain what they mean.
• You can now list all usage of an interface by other interfaces.
• Constants are ordered the same as in the source IDL since they tend to make more sense that way.
• Includes direct links to the source IDL file.

I’ve also updated the database with the latest APIs in 1.9.1b2pre and as soon as 1.9.1b2 is frozen I’ll update it again just to be sure it has the latest versions.

You can also search the API for interfaces. There isn’t any UI for this right now but it is perfectly set up for a smart bookmark. Just use http://www.oxymoronical.com/experiments/apidocs/search/interface/%s for the url and an appropriate keyword and you can do quick searches for substrings of the interface name.

For some time now I’ve been throwing around ideas for new features that we may want for the add-ons manager. After lots of thought and trying to take on board comments from anyone that would pipe up I decided it was high time to put together an actual plan for what features I have decided should be pursued and what order to start tackling them.

The roadmap itself should be pretty self explanatory, just remember that nothing is ever set in stone. I’ll try to keep it updated as things change, features seem less important or new items are added. But for now this is how I see the add-ons manager evolving over the course of about the next year or so, which may be the next two versions after Firefox 3.1.

Obviously the ordering of the goals may not be quite to everyone’s taste, but I’ve tried to strike a balance between getting new features done and getting groundwork laid to make things better in the future. I’m happy to take any comments people might have into consideration for altering the plan, but chances are I won’t be making any big changes unless you have some pretty damning evidence that a change is necessary

Faaborg has started a discussion on the newsgroups on the relative importance of polish and blocker bugs (I’d provide a link bug Google Groups seems to be refusing to acknowledge existence of the post). I have been for quite some time now very focused on the in-depth blocker issues, partly because that is where all the interesting work is for me, but also because I think it makes more sense for me or one of the other guys with good understandings of how the extension manager works to be making these changes rather than throwing others in at the deep end.

This has of course left little time for me to fix the trivial issues that Faaborg talks about. The things that users probably notice more and make the product look complete or half-done. This is why I’m really pleased that when I posted a list of simple bugs that anyone should be able to tackle there was a fairly good response. Of the 12 bugs on the original list, 9 have been fixed. I wanted to extend my thanks to those guys that took the time to work on those, some of them probably ended up being a little harder work than anticipated.

Of course the polish list is almost never ending. I’ve been adding more to the list of good first bugs, if anyone is interested in helping out then don’t be afraid to just email me or find me on IRC as Mossop and ask how much work something really involves.

I started a small discussion on the newsgroups the other day wondering about the numbers of API changes that had landed on trunk since 1.9.0. It was a short discussion, maybe partly because all I had to go on were the vague metrics that mercurial could give me. It also evolved a little into a discussion of how add-on developers we’re meant to find out about API changes. It’s true that as developers we are getting better at trying to announce our changes as much as possible, but inevitably we miss some, and if you don’t track the announcements what then?

Yesterday on IRC ted jokingly made the comment that I could use the idl parser than bsmedberg wrote in python a short time ago to find out more about the API changes. I’ve been hard pressed on other work lately so it wasn’t long before the prospect of knocking together a little side project had me too interested to do anything else. In a short space of time I had something that could parse all the in-tree idl files into a sqlite database that I could then query. It wasn’t long after that that I got a small web app written to display the content of the database, but I spent some more time today getting it a bit more useful before I showed it to the rest of the world.

So here it is, the API viewer. You can look at any interface in any of the platform versions that I’ve bothered to scan. And here is the more interesting bit, you can compare two platforms to see what interfaces changed. Not happy there? Well you can also compare an interface in different platform versions in a sort of diff like display.

I should note that it is only indexing the core and Firefox interfaces for now, and it’s close to indexing all of the idl files from the sources, many of which don’t make it into the final build (which makes life interesting as it turns out some of them are invalid xpidl!). So you can’t trust that interfaces you find are necessarily useful etc.

The web app is really a prototype right now. Navigation is probably a bit clunky and it’ll need more work to be really useful but I wondered what other people thought of it. So what do you think? Is it useful? What other interesting information might we be able to grab automatically like this?

Update: I’ve added details of the current trunk, 1.9.1b2pre as of today so you can see what changes are coming since the last beta. Not sure how often I’ll keep that up to date though.

Dear web (I’m not going to insult you and call you lazy like some people I’d mention),

Why is it that even if I’m just adding a tag to a wordpress post it deems it necessary to update the date of the post causing places like planet to republish the post? Surely there must be some way around this that I’m missing?

Tab Sidebar is probably my favourite extension that I’ve created. It is certainly the most polished, thanks mostly to other people pushing me to make it so. For those that haven’t used it it creates a thumbnail preview of all of your tabs in the sidebar. The thumbnails automatically update whenever the page changes, even things like popup menus generally show up. This automatic updating comes at a cost though.

In order to detect changes to the page content the extension mostly uses DOM mutation events. These are in theory sent out whenever a change to the page’s structure happens, which covers adding/removing content, style changes, text entry and all manner of things. What I wasn’t aware of when I first used this technique was that Gecko performs certain optimisations when there are no DOM mutation listeners registered for a document (the normal case). Simply the presence of a mutation listener impacts the speed of any DOM manipulation by the page. How much? Well nothing like a graph to illustrate things.

Performance impact of mutation listeners

These are numbers gathered from the Dromaeo test suite, averaging 5 runs with and without Tab Sidebar installed. A little perf loss is inevitable as the extension must perform some checks on the mutation and occasionally repaint the thumbnail, but nowhere near the sort of regression that actually occurs. I’ve combined some of the test results for simplicity but it is pretty clear that while the regular JS tests are essentially unaffected, the DOM tests can have wildly different results. Don’t ask me why DOM Queries get faster with the extension installed, but DOM modifications are a nice round 4 times slower.

That isn’t to say that using mutation listeners is a complete sin to be avoided at all costs. In some cases they are the only safe option. I discovered this problem some time ago and never before found a better way to detect the changes for Tab Sidebar. Last I checked Firebug uses them in similar ways. The point I think is that there are many features in the platform that can have unexpected side effects.

Of course I guess I wouldn’t be making this post if I hadn’t finally come up with a solution. While I’m no longer actively working on my extensions, I’m afraid I couldn’t resist the opportunity of the new MozAfterPaint event that landed on trunk recently. Quite simply whenever an area of the page is repainted on the screen this event is dispatched with details of what the area painted was. This is absolutely perfect for Tab Sidebar, which isn’t surprising since the use case it was written for is essentially what Tab Sidebar is doing anyway. After a few hours of ripping out most of the event handling code I now have a working prototype that redraws solely based on this event, only painting the areas that might have changed.

Performance impact of the MozAfterPaint listener

Boy does it work well. As you can see from the graph essentially all of the performance loss is gone. For some reason the faster DOM queries are still there (who am I to complain if my extension makes Firefox run faster!). And what’s more the previews are now updated not just for DOM changes, on OSX at least it even sees repaints of plugins. It is quite bizarre to be watching a movie on youtube and see the little thumbnail at the side showing the movie as well.

I guess now I have whetted your appetite it would be unfair not to make the test version available. Use at your own risk of course, the sheer amount of code change means there are likely some problems lurking, but you can install Tab Sidebar 2.5a1 and see how it goes. If you set the preference extensions.tabsidebar.selecteddelay to -1 then it won’t even delay before repainting and just repaint the thumbnail as soon as the main page is painted, this actually seems to work out best for me, particularly on sites with fast animations.