How Crashplan breaks xpcshell tests on Windows

I recently switched to a Windows laptop and have been going through the usual teething pains related. One thing that confused me though was that when I was running xpcshell tests on my new machine they would frequently fail with access denied errors. I’ve seen this sort of thing before so I know some service was monitoring files and opening them after they had changed, when this happens they can’t be deleted or edited until the service closes them again and often tests open, close and delete files so fast that there isn’t time for that to happen.

It took me a little while to remember that I can just use Process Monitor to track down the offending service. Just fire it up, set a filter to only include results to a particular directory (the temp directory in this case) and go create a file there and see what shows up. I was quite surprised to see Crashplan, the backup software I (and probably many people in Mozilla) use. Surprised because Crashplan isn’t set to backup my temp directory and really I shudder to think what the performance cost is of something continually accessing every file that changes in the temp directory.

Turns out you can turn it off though. Hidden in the depths of Crashplan’s advanced backup settings is an option to disable real-time filesystem watching. From what I can see online the downside to this is that files will only be backed up once a day, but that’s a pretty fine tradeoff to  having functioning xpcshell tests for me. There is also an option to put crashplan to sleep for an hour or so, that seems to work too but I don’t know exactly what that does.

It confuses me a little why Crashplan monitors files it never intends to backup (even when the backup server isn’t connected and backups aren’t in progress) and it is quite a lot of file accesses it does too. Seems likely to be a bug to me but at least I can workaround it for now.

Changing the checkCompatibility preference

Back in the mists of time I wrote some code to make nightly testers’ lives easier by giving them a simple preference to flip if they wanted to be able to install and use incompatible extensions. It’s been more than three years since then and the use of this preference has grown beyond its original use. It is now something recommended to regular users everywhere from forums to comments in news articles as a way to use their extensions in the new major Firefox releases.

Don’t get me wrong, letting users upgrade sooner than they otherwise might is a good thing, but the preference is a dangerous beast. It is pretty simple for a user to set the preference and then forget about it leaving them able to install incompatible extensions that break their Firefox without realising it. This costs Mozilla time as well since we get quite a number of bug and crash reports to look at that turn out to be caused by extensions that are dutifully marked incompatible with the user’s Firefox.

We’ve been mulling over ways to change this for a while but now we’ve actually gone and done something about it. We still want nightly testers and early adopters to be able to use incompatible extensions if they need to but we also want to make the preference not be a one shot deal that lasts till the end of time. The plan we’ve come up with is to change the preference’s name with the Firefox version, so for Firefox 3.6 (and all security releases) the preference will be extensions.checkCompatibility.3.6. When switching to a future 3.7 testers and users will have to set a new pref extensions.checkCompatibility.3.7 to say they still accept the risks of running with incompatible stuff.

Nightly users will have to make the changes slightly more often since the preference will also track whether the version is one of the development alphas or betas, so for the 3.6 betas the preference would be extensions.checkCompatibility.3.6b, for the current trunk extensions.checkCompatibility.3.7a. These are just normal preferences of course, if you frequently switch between different Firefox versions you can just set both necessary preferences. The change should land on the trunk in the next day or so and then the 3.6 builds a day or so after that.

There is just one oddity, if you’re testing nightlies and you update to a build with the change then you likely won’t notice any of your extensions disabling themselves, that won’t happen till either you toggle the pref or you switch to a build with a different Firefox version number in it.

Update: For more in-depth information this change was made in bug 521905.

Testing the background update checks

Firefox does a fair number of things in the background to help keep things up to date. This includes checking for updates to Firefox itself, checking for updates to add-ons you have installed and checking for updates to a blocklist that disables known unstable add-ons. Normally it does these things quite happily, but for developers and QA, trying to verify that these background tasks are doing what they are supposed to can be annoying. After all who wants to leave Firefox running for a day just to see if it finds a new update?

I’ve written a small extension that helps test these background tasks. It is fairly simple, just gives you a menu option to trigger one of the background checks. It saves delving through preferences and changing the timers to run faster. Most users shouldn’t use it but I figure it is useful for other developers and add-on authors so I’ve put it in the sandbox on AMO and you can go download it from there.

On Timezones, Testing and Deals with the Devil

Timezones make my head hurt. Not the concept, that is easy, but writing code that works correctly in different timezones. Throw daylight savings correction into the mix and you’ll often find me in a corner rocking gently. The problem of course is that I frequently have to deal with data that isn’t just the standard unix timestamp (or various orders of magnitude away from). A number of times a load of information has been thrown away, never to be recovered which makes life interesting. You end up having to decide which is the best path to take when all of them are wrong in various ways.

So why bring this up? Well America has just jumped to daylight savings time. And wouldn’t you know it, a few of Mozilla’s automated tests failed. Which is awesome of course. Always good to know something is up. The tricky thing with test failures is figuring out why they are failing. The automated tinderbox are remote machines you generally don’t have access to to debug on. This can be pretty unhelpful if the test doesn’t provide enough logged information. I’ve been pretty guilty of this myself, while a test that spots problems is fantastic, what is even better is one that provides enough information in the log to figure out why it failed. Think of it this way. If you write a test that (months, years) later fails then you’ll likely be asked why. What information do you need to find out? A few suggestions I’ve learned the hard way:

  • Always log the found and expected values when comparing (do_check_eq and similar does it for you but there are cases where you don’t use that).
  • If you have a loop in the test be sure to log what iteration the test failed in. A message that “false wasn’t true” isn’t a lot of use when that check is performed a hundred times.
  • If it is a long test why not just log a few checkpoints along the way, if the test crashes somewhere at least you’ll have a rough idea of where.

I will probably come back with more suggestions by the end of the week. I needed approval for a late fix I needed to land on Saturday and shaver was the only driver around at the time. He is often getting on at people (quite rightly) for not writing tests for the code they submit, so I made a deal with him that in return for the approval that I’d spend 2 days this week writing tests. I expect the majority of them to be testing for the UI side of the add-ons manager which is essentially untested right now and right now I have no clue how to go about testing. Should be fun!

Let the Testing Commence

After a fair bit of work (feels like longer than 2 months) I’ve finally managed to get bug 382752 landed. What this gives us in simple terms is a set of functions that we can use in order to do unit testing on the extension manager. Alongside I have checked in the first unit test. Now if anything regresses bug 257155 we should know about it immediately.

Ignoring the regression detection, I’ve always found unit tests to be fantastically useful when developing new code or fixing bugs. Zipwriter is a prime example, with a large number of tests that I can run by typing a single command I can test whether the changes I have made have solved the problem and not introduced any other errors.

The next step of course is to start adding unit tests for the extension manager. I have some already in progress and hopefully soon some of the key parts of the EM should be getting checked on a daily basis.