Broken executables in extensions in Firefox 3.6

If you are an extension developer and include executable files in your XPI package (binary or shell scripts) then you may be seeing problems in Firefox 3.6.

Back between Firefox 3.6 beta and Firefox 3.6 RC we took a small fix to the extension manager that changed how we extract the files from the XPI package. The fix involved adjusting how we accessed files to avoid hitting problems with certain anti-virus tools that would occasionally lock files in the middle of extraction making us fail to install the add-on. A side effect to this fix leaves us setting file permissions on the extracted files in a slightly different way to previously. This side effect means that the executable permission is getting stripped from all extracted files. If you try to execute these files with nsIProcess it will likely fail.

There is a bug on file and I have a patch almost complete so hopefully this should be fixed in Firefox 3.6.1 but until then you can workaround this in your extension by setting the executable permissions on the file yourself. Assuming you have a file variable that is an nsIFile pointed at the executable, just do:

file.permissions = 0755;

In case you were wondering why we enforce file permissions at all, it is because it turns out there are quite a lot of different zip tools that developers use to build add-ons. Some of them are unfortunately broken and embed bogus permissions into the generated XPI. After extraction it leaves us with files that are unreadable/unwritable which makes the add-on fail to work correctly. This most commonly affects developers on windows (where permissions are a little laxer in general) who get odd bug reports from users on Linux which is respectfully refusing to use the files. The easy fix is to enforce read/write permissions on the extracted file in the first place.

Do we need extension dependencies?

Ever since Firefox 2 we have vaguely supported a form of extension dependency. That is marking an extension as requiring particular versions of another extension.

The support is currently very limited and when a user tries to use an add-on that depends on something they don’t have they are pretty much left in the cold. While we tell them it requires something we don’t tell that what it requires or give them any easy way to download and install it.

Given this poor implementation it is perhaps no surprise that very few extensions make use of the feature (something like 50 listed on AMO, many of which are outdated). This raises questions over the cost of continuing to support this feature particularly now when we are undergoing large scale changes to the add-ons manager architecture to give better support to personas and jetpacks.

Before I made any final decisions I wanted to gather some feedback on what people thought about completely dropping the support for dependencies for now. We will always have the option to add it back again (even in a more functional state) at a later time should we decide it is worthwhile. Unfortunately it’s certainly not on the cards right now to complete the implementation as it stands and even leaving it as it is is costing significant work during the refactoring project.

Extensions can always handle dependencies themselves at runtime by testing for the presence of whatever they depend on. They can even then give the user a way to get what they need which makes for a far better user experience than the platform currently offers so I’m not sure I see a convincing reason to keep the feature as it is around for now.

Changing the checkCompatibility preference

Back in the mists of time I wrote some code to make nightly testers’ lives easier by giving them a simple preference to flip if they wanted to be able to install and use incompatible extensions. It’s been more than three years since then and the use of this preference has grown beyond its original use. It is now something recommended to regular users everywhere from forums to comments in news articles as a way to use their extensions in the new major Firefox releases.

Don’t get me wrong, letting users upgrade sooner than they otherwise might is a good thing, but the preference is a dangerous beast. It is pretty simple for a user to set the preference and then forget about it leaving them able to install incompatible extensions that break their Firefox without realising it. This costs Mozilla time as well since we get quite a number of bug and crash reports to look at that turn out to be caused by extensions that are dutifully marked incompatible with the user’s Firefox.

We’ve been mulling over ways to change this for a while but now we’ve actually gone and done something about it. We still want nightly testers and early adopters to be able to use incompatible extensions if they need to but we also want to make the preference not be a one shot deal that lasts till the end of time. The plan we’ve come up with is to change the preference’s name with the Firefox version, so for Firefox 3.6 (and all security releases) the preference will be extensions.checkCompatibility.3.6. When switching to a future 3.7 testers and users will have to set a new pref extensions.checkCompatibility.3.7 to say they still accept the risks of running with incompatible stuff.

Nightly users will have to make the changes slightly more often since the preference will also track whether the version is one of the development alphas or betas, so for the 3.6 betas the preference would be extensions.checkCompatibility.3.6b, for the current trunk extensions.checkCompatibility.3.7a. These are just normal preferences of course, if you frequently switch between different Firefox versions you can just set both necessary preferences. The change should land on the trunk in the next day or so and then the 3.6 builds a day or so after that.

There is just one oddity, if you’re testing nightlies and you update to a build with the change then you likely won’t notice any of your extensions disabling themselves, that won’t happen till either you toggle the pref or you switch to a build with a different Firefox version number in it.

Update: For more in-depth information this change was made in bug 521905.

Lightweight themes UI landed

As part of the ongoing work to bring basic support for lightweight themes (based on the ideas from the Personas extension) into Firefox 3.6 I’ve today landed the main UI parts that allow users to see and select between lightweight themes they have used recently. o landed most of the backend last week  but we’re still waiting on the support for installing new lightweight themes before this feature will be truly usable in the development builds. For the time being here is a shot of what the UI looks like in the add-ons manager after you have used some lightweight themes.

Lightweight themes UI

The idea is that users will install lightweight themes from websites. The add-ons manager will include the most recent lightweight themes used to allow the user to use them again. For this release they are offered as a straight alternative instead of other installed themes. We’re going to replace the default icon there with something more themey.

Third-party extension installation status

I have been remiss in not posting a status update about this in two weeks, but that is mostly because we have unfortunately had to slow down work on this feature. The problem is that a string freeze became necessary for all toolkit code (the code shared with the Firefox mobile browser and where this feature would have lived). Unfortunately this all came up over a small period when I was travelling long distances and having to take time out to satisfy immigration authorities that I wasn’t a terrorist intent on bringing down the U.S. government.

In the end there just wasn’t enough time to finalise the UI and be confident that it was correct before the freeze. This means that this feature isn’t going to make Firefox 3.6, however the intention is to continue working on it to get it into Firefox 3.7. It will just be on hold for a short time since I need to devote myself to Firefox 3.6 blockers.

Third-party add-on notification progress, the lite edition

This week has been the Firefox work week where almost all of the team, including me, made our way to Mountain View. This pretty much means that you spend the entire week in meetings since when you’re remote it can be hard to keep in sync on everything. Of course this means that the amount of coding is pretty low for the week so there isn’t a great deal of progress to report here.

Status

This week, Alex and I met to come up with a rough design for what this is going to look like now that we know that the doorhanger notifications we were hoping for aren’t going to make Firefox 3.6.

Whiteboard mockup
Whiteboard mockup

Next steps

  • Implement the planned design

Progress on notifying users about third-party add-ons

This is a weekly status update on the feature to notify users about add-ons that third-party installers have added to Firefox. You can read more about it in last weeks blog post or on the project wiki.

Status

There has been little progress this week mostly due to waiting to see whether the doorhanger notifications UI was going to arrive in time for Firefox 3.6. It looks like it isn’t so instead we are discussing using an in-content page to describe the add-ons that have been installed and allowing users to disable them.

Regular readers of my blog will probably spot that this is very similar to an idea I had a couple of months ago. What we’ll probably try to do is integrate both new third-party add-ons and new add-ons installed by the user into the same in-content page. It’s possible that we won’t be able to go the full stretch and allow add-ons to add their own content into this page in the first pass, that will depend on time constraints. I think that even without that this is still a very worthwhile feature.

Next steps

  • Develop more detailed UI mockups.
  • Implement the in-content page to read the already available data on new add-ons.

Supporting icons for disabled extensions

I’ve just landed code that allows extensions to include their icon in the add-ons manager view even when disabled. Currently extensions provide their icon by giving a chrome iconURL for us to load. This can only work when the extension is enabled since we don’t register the chrome otherwise. Themes on the other hand provide their icon as a simple file called “icon.png” alongside the install.rdf. Well extensions can do this too now and it will be used in preference to the iconURL but more importantly works at all times (well not before installation yet, but that is at least feasible this way).

We’ll continue to support iconURL for a time to come I imagine, but if you want to keep your icon when users disable you then just put your icon.png alongside the install.rdf (keep the iconURL if you want to work in Firefox 3.5 and lower of course).

This is only in trunk builds right now (today’s Linux nightly has it, tomorrow’s OSX and Windows nightlies will have it). It will be making its way to the Firefox 3.6 nightlies soonish.

Notifying users about third-party add-ons

You may have noticed many of the Firefox team starting to blog progress reports on Fridays. This is a part of our new plan to clearly define the main projects we are all working on and communicate a much as possible about them rather than just having users surprised to see them turn up in nightlies. So here is my report for this week:

I’ve been working on improving the level of information and control we give users over add-ons installed by other applications on the system (think Skype, Java, AV tools etc.). The work for this is being tracked in bug 476430 and on the project wiki.

The basic problem is that sometimes when another application installs an extension into Firefox the user either isn’t told so by the application, or more likely the information is missed in installer pages that users click straight through. It is then a surprise when users later find this thing in the add-ons manager, sometimes when it is causing problems. The goal is to make sure that this never happens by telling users that something has been installed and giving them the option not to use it.

We considered a couple of principles for the design here:

  • Firstly we should not delay starting Firefox. If we present UI asking a question before startup then users will just click through it.
  • Secondly we shouldn’t just run Firefox with the new extensions enabled, users should be given the option whether to use them before they are run for the first time.
  • Thirdly the eventual result of a user taking no additional action should be that the new extensions should be enabled. If we default to disabling these items then the majority of users will never enable them. This would make other applications stop using these mechanisms to integrate with Firefox and seek out more hacky, less controllable ways to do it, something that no-one wins from really.

The resultant plan is a hybrid approach. The first time Firefox runs after a new extension is detected it will not enable it. Firefox will tell the user what has been installed and tell them that the extension will be enabled the next time Firefox restarts. The user can then choose to disable it permanently (or at least until they choose to re-enable it in the extension manager). The user will probably also be presented with a quick way to restart Firefox to use the new extension.

Status

The backend code that disables detected extensions during the first run is complete, as is the frontend code that marks them to be enabled after a restart. The UI still needs to be connected but that will require the doorhanger notification API to be completed.

Next steps

  • Finalise the UI
  • Wait for doorhanger notifications to be implement
  • Wire up the detection code to the UI
  • Figure out whether we can test the UI

Farewell contents.rdf

This is mainly of interest to add-on and application developers and I should stress from the outset that this is talking about changes in Gecko 1.9.2 which will first be released in whatever version of Firefox comes in 6-12 months time. Firefox 3.5 is unaffected by this change.

What was it?

Contents.rdf was the old way of performing chrome registration for add-ons. It was replaced by chrome.manifest back in the mists of time in Gecko 1.8 and Firefox 1.5 (back in 2005 as it happens). We’ve continued to support reading contents.rdf for those developers who hadn’t had the chance to make the switch but after 4 years it seems time to remove that support and clean up the code that dealt with it.

What does this mean?

Chances are not a lot. If you are an add-on developer then look in your add-on’s source code. Is there a chrome.manifest file there? If so then you don’t need to do anything, you are already using the modern method of chrome registration. If not then look for contents.rdf files. If you don’t have any then you just have an odd add-on with no chrome.

How do I cope?

If your add-on actually doesn’t have a chrome.manifest file and needs one then you need to do one of two things. One option is simply to read the documentation on chrome.manifest and learn how to replicate what you have in the contents.rdf files into the manifest. Another is to install your add-on into a toolkit application since Gecko 1.8. You’ll find that when it installs it it also creates a chrome.manifest file for you from your contents.rdf files. Just copy it.

If you really need to support both an old version of an application (before Gecko 1.8, or not toolkit based) and a new version (Gecko 1.9.2 or newer) then you will need to keep both contents.rdf and chrome.manifest, replicating the information across the two.

When does this happen?

I have just landed the patch that removed support so trunk nightlies from tomorrow (12th June) should be affected.