Oxymoronical

Back in the mists of time I wrote some code to make nightly testers’ lives easier by giving them a simple preference to flip if they wanted to be able to install and use incompatible extensions. It’s been more than three years since then and the use of this preference has grown beyond its original use. It is now something recommended to regular users everywhere from forums to comments in news articles as a way to use their extensions in the new major Firefox releases.

Don’t get me wrong, letting users upgrade sooner than they otherwise might is a good thing, but the preference is a dangerous beast. It is pretty simple for a user to set the preference and then forget about it leaving them able to install incompatible extensions that break their Firefox without realising it. This costs Mozilla time as well since we get quite a number of bug and crash reports to look at that turn out to be caused by extensions that are dutifully marked incompatible with the user’s Firefox.

We’ve been mulling over ways to change this for a while but now we’ve actually gone and done something about it. We still want nightly testers and early adopters to be able to use incompatible extensions if they need to but we also want to make the preference not be a one shot deal that lasts till the end of time. The plan we’ve come up with is to change the preference’s name with the Firefox version, so for Firefox 3.6 (and all security releases) the preference will be extensions.checkCompatibility.3.6. When switching to a future 3.7 testers and users will have to set a new pref extensions.checkCompatibility.3.7 to say they still accept the risks of running with incompatible stuff.

Nightly users will have to make the changes slightly more often since the preference will also track whether the version is one of the development alphas or betas, so for the 3.6 betas the preference would be extensions.checkCompatibility.3.6b, for the current trunk extensions.checkCompatibility.3.7a. These are just normal preferences of course, if you frequently switch between different Firefox versions you can just set both necessary preferences. The change should land on the trunk in the next day or so and then the 3.6 builds a day or so after that.

There is just one oddity, if you’re testing nightlies and you update to a build with the change then you likely won’t notice any of your extensions disabling themselves, that won’t happen till either you toggle the pref or you switch to a build with a different Firefox version number in it.

Update: For more in-depth information this change was made in bug 521905.

Firefox does a fair number of things in the background to help keep things up to date. This includes checking for updates to Firefox itself, checking for updates to add-ons you have installed and checking for updates to a blocklist that disables known unstable add-ons. Normally it does these things quite happily, but for developers and QA, trying to verify that these background tasks are doing what they are supposed to can be annoying. After all who wants to leave Firefox running for a day just to see if it finds a new update?

I’ve written a small extension that helps test these background tasks. It is fairly simple, just gives you a menu option to trigger one of the background checks. It saves delving through preferences and changing the timers to run faster. Most users shouldn’t use it but I figure it is useful for other developers and add-on authors so I’ve put it in the sandbox on AMO and you can go download it from there.

Timezones make my head hurt. Not the concept, that is easy, but writing code that works correctly in different timezones. Throw daylight savings correction into the mix and you’ll often find me in a corner rocking gently. The problem of course is that I frequently have to deal with data that isn’t just the standard unix timestamp (or various orders of magnitude away from). A number of times a load of information has been thrown away, never to be recovered which makes life interesting. You end up having to decide which is the best path to take when all of them are wrong in various ways.

So why bring this up? Well America has just jumped to daylight savings time. And wouldn’t you know it, a few of Mozilla’s automated tests failed. Which is awesome of course. Always good to know something is up. The tricky thing with test failures is figuring out why they are failing. The automated tinderbox are remote machines you generally don’t have access to to debug on. This can be pretty unhelpful if the test doesn’t provide enough logged information. I’ve been pretty guilty of this myself, while a test that spots problems is fantastic, what is even better is one that provides enough information in the log to figure out why it failed. Think of it this way. If you write a test that (months, years) later fails then you’ll likely be asked why. What information do you need to find out? A few suggestions I’ve learned the hard way:

• Always log the found and expected values when comparing (do_check_eq and similar does it for you but there are cases where you don’t use that).
• If you have a loop in the test be sure to log what iteration the test failed in. A message that “false wasn’t true” isn’t a lot of use when that check is performed a hundred times.
• If it is a long test why not just log a few checkpoints along the way, if the test crashes somewhere at least you’ll have a rough idea of where.

I will probably come back with more suggestions by the end of the week. I needed approval for a late fix I needed to land on Saturday and shaver was the only driver around at the time. He is often getting on at people (quite rightly) for not writing tests for the code they submit, so I made a deal with him that in return for the approval that I’d spend 2 days this week writing tests. I expect the majority of them to be testing for the UI side of the add-ons manager which is essentially untested right now and right now I have no clue how to go about testing. Should be fun!

After a fair bit of work (feels like longer than 2 months) I’ve finally managed to get bug 382752 landed. What this gives us in simple terms is a set of functions that we can use in order to do unit testing on the extension manager. Alongside I have checked in the first unit test. Now if anything regresses bug 257155 we should know about it immediately.

Ignoring the regression detection, I’ve always found unit tests to be fantastically useful when developing new code or fixing bugs. Zipwriter is a prime example, with a large number of tests that I can run by typing a single command I can test whether the changes I have made have solved the problem and not introduced any other errors.

The next step of course is to start adding unit tests for the extension manager. I have some already in progress and hopefully soon some of the key parts of the EM should be getting checked on a daily basis.