Simplifying

The big project that I have been working on for quite some time now is a complete change to the architecture of the add-ons manager backend. It’s a big scary prospect since (IMHO) the code is pretty crucial to the success of Firefox and many other Mozilla based applications. Without extensions I don’t think we’d be where we are today, in fact it was because of extensions that I got involved in the project in the first place.

The current incarnation of the extension manager has served us well over the past years but for some time it has been under strain. There are always more features we want to add but certain aspects of its design make it hard to do that. Things like changing from an RDF based persistent storage to anything else are hard because the concepts are so ingrained in the code. It’s now at the point where changing anything (particularly in the startup code) is very difficult since unexpected side effects are almost sure to happen. I tried various approaches to slowly iterating the current code to something better but ultimately changing anything required changing everything so I finally bit the bullet and concluded that a rewrite was the way forward (I know, normally they aren’t the right choice but I’m, pretty confident that it is in this case).

It is that rewrite that I’ve been working on on and off for something like half a year, if not longer. Some of the nice benefits that we should see:

  • Switching from RDF to a SQLite storage model (and making it easier to switch in the future)
  • Startup and other performance improvements (probably not immediately though)
  • Less bothersome dialogs for the user
  • Vastly simpler APIs for application/extension developers including:
    • Proper separation of the backend and UI code
    • Support for pluggable add-on types

I didn’t actually mean for this post to be this long actually (after all it is about simplification), it was just this last point that I felt like demonstrating. I just wrote the basic part of the code that implements the automatic daily background update checks. It uses the new API that will be available and as well as checking for updates it also downloads and installs any it finds silently (currently, I’ll be adding the notification and controls etc. soon). It struck me just how simple it was to write this:

AddonManager.getAddonsByType(null, function(addons) {
  addons.forEach(function(addon) {
    if (addon.permissions & AddonManager.PERM_CANUPGRADE) {
      addon.findUpdates({
        onUpdateAvailable: function(addon, install) {
          install.install();
        }
      }, AddonManager.UPDATE_WHEN_PERIODIC_UPDATE);
    }
  });
});

That is some 11 lines of code to perform an update check to the add-on’s update server, retrieve information about the newest available update and start downloading and installing it.

Testing the background update checks

Firefox does a fair number of things in the background to help keep things up to date. This includes checking for updates to Firefox itself, checking for updates to add-ons you have installed and checking for updates to a blocklist that disables known unstable add-ons. Normally it does these things quite happily, but for developers and QA, trying to verify that these background tasks are doing what they are supposed to can be annoying. After all who wants to leave Firefox running for a day just to see if it finds a new update?

I’ve written a small extension that helps test these background tasks. It is fairly simple, just gives you a menu option to trigger one of the background checks. It saves delving through preferences and changing the timers to run faster. Most users shouldn’t use it but I figure it is useful for other developers and add-on authors so I’ve put it in the sandbox on AMO and you can go download it from there.

Making Gallery2 understand ogg and use video tags

One of the nice new features in the fast evolving HTML 5 spec is support for specific video and audio tags, replacing the more generic object and embed tags that have been used in the past. They have many benefits over the object tag such as a well defined JavaScript interface, controls provided by the browser and support for multiple formats for the browser to choose between for display. The current beta version of Firefox 3.1 supports this and includes support for playing the Ogg format with Theora and Vorbis codecs included. Opera has support underway as well and it looks like the latest Safari releases also have it (though it seems broken in some respects).

I’ve been trying to find a suitable video format for putting my fractal animations online for some time. Pretty much whatever format you choose you suffer because some platform or other doesn’t have a plugin for it by default. If all browsers are slowly getting ogg support though then that seems to be the ideal choice, so I have re-encoded all the animations as ogg files. Unfortunately it seems that my media gallery software (Gallery2) doesn’t support Ogg, and of course it doesn’t use the new video tag either. This is the world of open source though, so after a bit of tinkering I have it all working nicely. It can generate thumbnails for the animations using ffmpeg (which is already used for other video formats anyway) and will display the video using the video tag.

At some point I might try to push this upstream to the Gallery2 guys, but I after spending some time on their site and finding nothing helpful like a bug tracker or a way to pass patches to them I’m not sure when I’ll get round to it. Until then here are the rough instructions for applying the patch and some other changes I had to do. I’m sure there is some way to automate it all but for this one off case it wasn’t really worth my time.

  • Apply the patch in your gallery install directory: “patch -p0 <patchfile
  • Run the following sql in the gallery database: “INSERT INTO g2_MimeTypeMap (g_extension, g_mimetype, g_viewable) VALUES ('ogg', 'video/ogg', 1);” (adjust for the table and column prefixes that you use).
  • Deactivate the ffmpeg plugin (if it is enabled), and then activate it (assuming you want thumbnail generation to work).
  • Delete the database cache in the maintenance section of the gallery admin.

Of course this means that now even fewer people can see the animations I guess. So if you want to see them then you’ll just have to try out the latest Firefox betas!

Oh, watch out for old versions of ffmpeg. The default version on my webhost (dreamhost) crashes when attempting to make a thumbnail of an ogg file. A build of the latest version of ffmpeg works though.

On Timezones, Testing and Deals with the Devil

Timezones make my head hurt. Not the concept, that is easy, but writing code that works correctly in different timezones. Throw daylight savings correction into the mix and you’ll often find me in a corner rocking gently. The problem of course is that I frequently have to deal with data that isn’t just the standard unix timestamp (or various orders of magnitude away from). A number of times a load of information has been thrown away, never to be recovered which makes life interesting. You end up having to decide which is the best path to take when all of them are wrong in various ways.

So why bring this up? Well America has just jumped to daylight savings time. And wouldn’t you know it, a few of Mozilla’s automated tests failed. Which is awesome of course. Always good to know something is up. The tricky thing with test failures is figuring out why they are failing. The automated tinderbox are remote machines you generally don’t have access to to debug on. This can be pretty unhelpful if the test doesn’t provide enough logged information. I’ve been pretty guilty of this myself, while a test that spots problems is fantastic, what is even better is one that provides enough information in the log to figure out why it failed. Think of it this way. If you write a test that (months, years) later fails then you’ll likely be asked why. What information do you need to find out? A few suggestions I’ve learned the hard way:

  • Always log the found and expected values when comparing (do_check_eq and similar does it for you but there are cases where you don’t use that).
  • If you have a loop in the test be sure to log what iteration the test failed in. A message that “false wasn’t true” isn’t a lot of use when that check is performed a hundred times.
  • If it is a long test why not just log a few checkpoints along the way, if the test crashes somewhere at least you’ll have a rough idea of where.

I will probably come back with more suggestions by the end of the week. I needed approval for a late fix I needed to land on Saturday and shaver was the only driver around at the time. He is often getting on at people (quite rightly) for not writing tests for the code they submit, so I made a deal with him that in return for the approval that I’d spend 2 days this week writing tests. I expect the majority of them to be testing for the UI side of the add-ons manager which is essentially untested right now and right now I have no clue how to go about testing. Should be fun!

Trimming the Fat

As I mentioned in my last post and as I know everyone is well aware, tinderbox is starting to feel the strain. You already can’t just glance at it to see what is going on. The Firefox tree is currently showing build and test results from a total of 23 machines. The interesting thing is that not all of these machines are actually doing any compiling. The rest are merely running tests on builds produced by the other machines. That isn’t to say that those test results are less important but I wonder whether it is worth treating these differently.

Thinking about this difference I’ve spent a short time knocking together an experimental view of tinderbox. The idea is simple, the test boxes that aren’t actually compiling don’t get a dedicated column. We still want those results though, and luckily there is somewhere to put them, on the build that they are testing. So every actual compile ends up as a box within which can be multiple results from testers. This leaves us a very manageable 9 columns, 3 per platform.

This is quite early stages, the physical perf and test numbers aren’t visible yet, there are a couple of bugs I know about, it doesn’t auto update and I wouldn’t recommend you really use this, but I am interested to know what people think before I proceed any further with it. There are a few questions in my mind like how to colour the main box of the build (currently the worse result from the compile and all the test results), how to colour the header (I suspect the worst of the most recent results from all of the machines reporting in that column), where to even put the full perf/test numbers (there isn’t really room in some of the fast nightly boxes). More importantly is this even readable? I think it is good for some things, often tinderbox gets confusing when you see a talos result fail long after the nightly box went green, here that talos red would be back on the nightly box that failed. I can imagine problems trying to figure out why the header is red when it’s actually the talos results from build a couple of times ago that is causing it.

Anyway without further ado, take a look at Tindermerge and let me know your thoughts.

Visualising our Labours

Well it has been a mad few weeks. Between landing the new Get Add-ons pane, sheriffing the Firefox 3 beta 3 freeze, decimating old parts of the code and working through follow-up issues from the Get Add-ons pane and not to mention two 10 hour flights it really has been busy. Thankfully I’ve not just been totally buried under code, I’ve had some small spare time to tinker with a few other things (I need to to keep my sanity).

We’re getting better and better at adding quality control to our code. The QA team do their awesome work, we have some 50,000 automated unit tests now (depending how you count) and our tinderbox now has more machines running performance tests than it does machines building the Fox. One problem that all this brings though is a bit of information overload. The tinderbox is now so large that you can no longer glance to see if you can check in or not. Goodness knows how many pages you need to view to truly monitor all of the performance graphs. So, spurred on by Jonathan’s performance dashboard, I’ve been tinkering with some ways to present some of this data.

MultiperfFirst off there is the old-style performance machines, pre-talos. These often find regressions first, because they cycle fast. Apparently that is going to improve but until then these old machines have useful data. Unfortunately the graphs they produce are … well lets just say pig ugly! It didn’t take too long to knock up something very basic, but nicer looking and allowing multiple results on the same graph. You can see it graphing the live data (yes it will appear blank for the age it takes to draw). Unfortunately the graph kit I’m using seems a little slow and not all that functional, but that could be changed. Then again I have just found out that new work on the graphs server pretty much blows this out of the water!

So that’s all pretty and good but doesn’t bring a much new to the table. Graphs are good but what we want to be able to do is relate the performance changes to the code that caused it. At this point Johnathan was kind enough to point me at Timeplot. This is a rather neat tool that lets you visualise numerical data on the same graph as defined events. That’s kinda perfect for this. It took me a little longer to hook this one up given the multiple sources and the kinda weird API that can’t normally load from anything other than a remote text file, but here is the result. You can see the performance data of course (Ts from the linux perf box), then each vertical band is a checkin which you can click on for more details.

CheckinplotIn the end the timeplot is not massively useful. When you get down to it we tend to check in far more frequently than we gather performance information. I’ll be interested to see if the fast run talos boxes will improve matters. If we had many of them then we can start correlating the regressions across them and thus far reduce the range in which to look for potential causes. I’m interested in other ideas people might have for visualising these kinds of data, I think finding the right way to visualise our information is the key to managing things in the future.

Another thing that needs help is the tinderbox of course. While I was in Mountain View last week Johnathan and I did a quick brainstorm of what the tinderbox is used for. We think it is being used for too many different things and if we split the display out into just those tasks that individuals needed then it would be far more manageable. Maybe you have your own comments on uses we didn’t come up with.

Just finally, after playing with timeplot I was still in a tinkering mood. One thing I periodically have to do is look back over what work I did on specific days. Bugmail is a good indicator for this. Now though I am working in my own Mercurial repository which means I have a full checkin log of my own to look over. Given that it is interleaved with the full Mozilla CVS checkin log it still a little arduous to scan through. But know I have the tools and after not long I can know skim over my previous checkins and even see in depth what each involved (yes, sorry boss it doesn’t look like I did much coding today!).

ZipWriter is Here!

Yes, finally after months of twiddling my thumbs waiting for approval to land the zipwriter component has gone into trunk, and is enabled for all applications (some apps might need to add the interface to their packages list). If you want to use check out the interface, it’s fairly well documented I think. That magic contract ID you need is “@mozilla.org/zipwriter;1″.

It’s been quite a long process both writing the code, getting reviewed and getting agreement for it to appear in Firefox so I hope all you extension authors and application developers are going to make good use out of it in your projects. Let me know what you come up with.

Let the Testing Commence

After a fair bit of work (feels like longer than 2 months) I’ve finally managed to get bug 382752 landed. What this gives us in simple terms is a set of functions that we can use in order to do unit testing on the extension manager. Alongside I have checked in the first unit test. Now if anything regresses bug 257155 we should know about it immediately.

Ignoring the regression detection, I’ve always found unit tests to be fantastically useful when developing new code or fixing bugs. Zipwriter is a prime example, with a large number of tests that I can run by typing a single command I can test whether the changes I have made have solved the problem and not introduced any other errors.

The next step of course is to start adding unit tests for the extension manager. I have some already in progress and hopefully soon some of the key parts of the EM should be getting checked on a daily basis.

Zip Writer Goodness

I can’t believe it’s been over a month since I wrote something here. Well I kinda can, lots of hectic stuff going on at my work right now which has been making finding time for Mozilla stuff tricky. Hopefully not for much longer.

I’m glad to say that I have managed to make great progress on the zip writer component. I have decided that dealing with multiple platform compiles for Nightly Tester Tools is just a bad idea, so instead I have pushed on with submitting the zip writer to Mozilla for review. Hopefully that will make it into tree where I (and of course anyone else) can just use it. There’s been a bunch of changes between the version I posted earlier and that that’s gone up for review, not least of which is a set of testcases that have made sure I didn’t break the old by making some cleanups.

Lots of people have been bugging me about when new versions of my extensions are going to be done. Sadly I can’t really do this. The old adage of “It’ll be done when it’s done” certainly applies. For most I don’t have a good handle on how much work is left and I certainly don’t know how much time will have to spend on them in the near future which sort of messes up any planning.

Oh and I’d just like to say hello to all you people reading planet out there. Assuming I haven’t broken my atom feed you should be able to see everything I write here from now on, you lucky lucky people!

Zip Writer Update

I spent some more time this weekend hacking on my Zip Writer component. It’s now pretty capable. It can open existing zip files and remove entries and append to them, happily rewriting the headers and everything exactly as it encountered them. And the other major win is that I have deflate code up and running which makes compression possible.

All this has allowed me to reintroduce making extensions compatible during the install process in a far safer way than was the case before. NTT can now watch the EM datasource, spot that an extension has finished being downloaded, then open that extension’s xpi file and if necessary rewrite the metedata to claim compatibility with the current version of whatever app you happen to be running in.

Depending on time I may have the next release of NTT out the door by the end of next weekend, but things are hectic as usual and I have to be various places over the coming week.