WebApp Tabs, version control and GitHub

The extension I’ve been working on in my spare time for the past couple of weeks is now available as a first (hopefully not too buggy) release. It lets you open WebApps in Thunderbird, properly handling loading new links into Firefox and making all features like spellchecking work in Thunderbird (most other extensions I found didn’t do this). You can read more about the actual extension at its homepage.

Mostly I’ve been really encouraged during the development of this at just how far our platform has come for developing restartless add-ons. When we first made it possible in Firefox 4 there was a whole list of things that were quite difficult to do but we’ve come a long way since then. While there are still things that are difficult there are lots of things that are now pretty straightforward. My add-on loads simple XUL overlays, style overlays, installs JS XPCOM components with category manager registration, all similar to older add-ons. In fact I’m struggling to think of things that it is still hard to do though I’m sure other more prolific developers will have plenty of comments on that!

The other thing I’ve been doing with this extension is experimenting with git and GitHub. I think it’s been an interesting experience, there are continual arguments over which is better between git and mercurial with many pros and cons listed. I think most of these were done some time ago before mercurial and git really matured because from what I’ve seen there is really little difference between the two. They have slightly different default branching styles, but both can do the same kind of branching that the other can if you want and there are a few other minor differences but nothing that would really make me all that bothered over deciding which to use. I think the only place where git has a bonus is with GitHub, and really as far as I can see there isn’t a reason why someone couldn’t develop a similar site backed by mercurial repositories, it’s just that no-one really has.

GitHub is pretty nice with built-in basic issue tracking and documentation though it still has some frustrating issues. It seems odd for example that you can’t fork your own project, only someone else’s, but that’s only a minor niggle really. As project hosting goes I can’t say I’ve come across anything better that I can remember.

Overlays without overlays in restartless add-ons

Perhaps the most common way of making changes to Firefox with an extension has always been using the overlay. For a window’s UI you can make changes to the underlying XUL document, add script elements to be executed in the context of the normal window’s code and add new stylesheets to the window to change how the UI looks.

Restartless add-ons change this around completely, the normal overlay and style-overlay mechanisms just aren’t available to restartless add-ons and this is likely to remain true for a while, these methods don’t clean up after themselves when the add-on is uninstalled.

This can make things hard, particularly for porting older add-ons to become restartless. I was in this situation earlier this weekend. I was working on porting David Ascher‘s WebTabs for Thunderbird to be restartless. I could have just moved all the script code over to bootstrap.js but in many ways it is nice to keep the code that works on the main UI separate to the code that runs for the preferences UI etc. Plus I like to play around with new ways of doing things so I developed a JS module I’m calling the OverlayManager.

The OverlayManager watches for new windows being opened and for every new window it can run JS script and apply CSS stylesheets to the window in a way that is easy to undo if the add-on is disabled at runtime. Although it can’t do any XUL modifications right now (I didn’t need any for this particular extension) it would be pretty easy to extend this to support a minimum about of XUL overlays.

Stylesheets are loaded by adding a HTML style tag to the XUL document, so they can be removed easily when the add-on is disabled. Scripts are handled in a way that may even be better than normal overlays. In the old system extension scripts all run in the same context as the window they overlay giving rise to the possibility of conflicts. Restartless add-ons shouldn’t do this since it makes removing the script code again much more troublesome. The OverlayManager handles it by creating a sandbox to run the script in. The sandbox’s prototype is set to the window the script is being run for meaning the script sees all the functions and objects of the window directly in its own scope but as long as it doesn’t modify any of the objects in the main window’s code all we have to do is throw away the sandbox to get rid of its JS.

There are a few things different of course. The script shouldn’t use load and unload event handlers for the window as it may get loaded well after the window does or unloaded well before. Instead the OverlayManager looks for an OverlayListener object in the script and calls load and unload methods on it, these are called either with the window’s real load and unload events or while the window is open normally. You also can’t reference code in the script from JS string blocks, so if you set onclick="myfunc()" on a XUL element it wouldn’t work because that would run in the main window scope which can’t see the sandbox code at that point. This tends to be pretty simple to get around by using addEventListener for all your events though.

You can see the existing state of the code on github and an example of the structure you’d pass to OverlayManager.addOverlays is in the bootstrap script for the same project. It is appropriately licensed so go nuts!

Update: I changed the stylesheets to use XML processing instructions to be more like they work currently and just for fun I implemented the very basics of document overlaying, almost totally untested though so YMMV.

Adding add-on preferences to the Add-ons Manager

For some time now Firefox for mobile has had this nice feature where add-ons could embed their preferences right into the list of add-ons, no need to open a whole a new window like add-ons for desktop have to. During the development of Firefox 4 we were a little jealous of what the mobile team had done and so we drew up some ideas for how the same functionality would look on desktop. We didn’t get time to implement them then but I’m excited that someone from the community stepped up and implemented it for us. Not just that but he made the code shared between mobile and desktop, added some new option types and made it work fine for restartless add-ons which are unable to register their own chrome.

The basic idea is simple. Create a XUL file containing a list of <setting> elements. Different types of settings are possible, checkboxes, input boxes, menulists, buttons, etc. Each one shows up as a row in the details view for an add-on in the add-ons manager. The XUL file can either be just added to your XPI (call it options.xul) or referenced by the optionsURL option in your install.rdf.

Get it right and you’ll see something like this:

I want to thank Geoff Lankow (darktrojan on IRC) for his awesome work getting this done. This feature is now in the Aurora builds and it’d be great to get add-on developers playing with it. Geoff even wrote up some detailed docs to help you out.

As a bonus Geoff also implemented support for in-tab preferences. This makes Firefox load an add-ons options UI in a new tab instead of a new window. Setting the optionsType property to 3 enables this.

Unloading JS modules

One of the problems with writing a restartless add-on is that you have to be careful about undoing anything your add-on does when it is told to shutdown. This means that right now some features of the platform can’t be used as we have no way to undo them. Recently I made this list a little shorter by making it possible to unload JS modules loaded with Components.utils.import().

Just call Components.utils.unload(uri) and the module loaded from that URI will be unloaded. Take care when you do this because something might still have references into it which will stop working. Firefox also caches the module’s code for fast loading the next time around. The add-ons manager clears this cache when your add-on is updated or uninstalled so you mostly don’t have to worry about it but if you do something strange like unload a module, manually alter the file and then import it again you won’t get the latest code.

This support is in the upcoming Aurora build.

Playing with windows in restartless (bootstrapped) extensions

Lots of people seem to be playing with the new support for restartless extensions (also known as bootstrapped extensions) coming in Firefox 4. Nothing could make me happier really. I’m not sure I can remember ever helping implement something which is will hopefully turn out to be a game changer for extension development in the future. The technical details of restartless extensions are talked about in a few places but one thing that is missing is what I think is probably going to be the most common code pattern in these extensions.

When you’re developing an extension it’s almost certain that you’re going to want to interact with the main browser window in some way. With the XUL based extensions this was a pretty simple process, normally your extension would be overlaying browser.xul and so any script you add there would automatically run for every new browser window and would have direct access to it (in fact this sometimes confused new developers who assumed their script would run just once and be globally available, not for every browser window opened).

In restartless extensions things are different. You have one script that runs when your extension is started and runs in its own sandbox so you have to take steps to get access to windows. Complicating matters your extension may be started when Firefox starts up and there are no browser windows or after when some already exist. The best way to go is when your extension is started to look for any existing browser windows and make your necessary modifications and then wait for new windows to open and modify those too.

Since this is such a common case (and vlad was trying to figure it out on IRC) I put together a very simple bootstrap script that just does this. It calls the setupBrowserUI function for every window that needs to be initialised by your extension and tearDownBrowserUI for every window that must be cleaned up when your extension is being disabled. This code isn’t the only way to do all this but it is fairly straightforward and works, feel free to take it and modify it for use in your own extensions. I’ll probably get it put onto MDC too.

const Cc = Components.classes;
const Ci = Components.interfaces;

var WindowListener = {
  setupBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to add UI or anything to the browser window
    // document.getElementById() etc. will work here
  },

  tearDownBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to remove UI or anything from the browser window
    // document.getElementById() etc. will work here
  },

  // nsIWindowMediatorListener functions
  onOpenWindow: function(xulWindow) {
    // A new window has opened
    let domWindow = xulWindow.QueryInterface(Ci.nsIInterfaceRequestor)
                             .getInterface(Ci.nsIDOMWindow);

    // Wait for it to finish loading
    domWindow.addEventListener("load", function listener() {
      domWindow.removeEventListener("load", listener, false);

      // If this is a browser window then setup its UI
      if (domWindow.document.documentElement.getAttribute("windowtype") == "navigator:browser")
        WindowListener.setupBrowserUI(domWindow);
    }, false);
  },

  onCloseWindow: function(xulWindow) {
  },

  onWindowTitleChange: function(xulWindow, newTitle) {
  }
};

function startup(data, reason) {
  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.setupBrowserUI(domWindow);
  }

  // Wait for any new browser windows to open
  wm.addListener(WindowListener);
}

function shutdown(data, reason) {
  // When the application is shutting down we normally don't have to clean
  // up any UI changes made
  if (reason == APP_SHUTDOWN)
    return;

  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.tearDownBrowserUI(domWindow);
  }

  // Stop listening for any new browser windows to open
  wm.removeListener(WindowListener);
}

How to extend the new Add-ons Manager (or how I built a simple greasemonkey clone in an evening)

One of the goals of the new add-ons manager API was to create something that was itself extensible. A couple of times in the past we’ve had to add new types of add-ons to the UI like Plugins and Personas. In both cases squeezing them into the UI was something of a kludge involving a bunch of custom code for each case. We already have a number of new types of add-ons that we want to add, things like search plugins which are currently managed by their own custom UI.

To help simplify this the API is largely type-agnostic. Internally it uses a number of so-called add-on “providers”, each of which may make information about add-ons available. Each provider is basically a JavaScript object with functions defined that the API can call to ask for information about add-ons that the provider knows about. The provider then just has to pass back JavaScript objects to represent each add-on. Each of these must have at minimum a set of required properties and functions and may also include a set of optional properties. The full set is defined in the API documentation.

With this design the user interface doesn’t need to care about implementation details of any of the providers, how they store their data or what exactly their add-ons are and do. Because each gives objects that obeys the same interface it can just display and manipulate them.

To try to show this all off I recently put together a small demo extension for the Mozilla Summit that registers a new type of add-on to be displayed in the main add-ons manager. This is a short overview of some of the highlights and I’ll make the code available for people to look at and take examples from. The add-on was a basic implementation of Greasemonkey allowing user scripts to be installed, managed through the add-ons manager and do it all as a restartless add-on.

Making a restartless add-on

Add-ons don’t have to be developed with Jetpack to make them restartless, although the Jetpack SDK certainly makes things easier on you, at the expense of less access to the internals of the platform.

The first thing to learn about making a restartless add-on is that you can forget about using XUL overlays or registering XPCOM components to be called at startup. Neither are supported at the moment, and maybe never will. Instead you have to provide a bootstrap script. This is a simple “bootstrap.js” file in the root of the extension that should include a “startup” and “shutdown” function. These are called whenever Firefox wants to start or stop your add-on either because the application is starting up or shutting down or the add-on is being enabled or disabled. You can also provide “install” and “uninstall” methods to be notified of those cases but that is probably unnecessary in most cases.

At startup the demo extension does some basic things. It registers for some observer notifications, registers a new add-on provider (I’ll talk more about that below) and does a little work to include itself in the add-ons manager UI (again, see below).

The rule is this. Anything your add-on does after being started must be undone by the shutdown function. The shutdown function often ends up being the inverse of startup, here it removes observer notification registrations, unregisters the add-on provider and removes itself from the UI. It also shuts down a database if it was opened.

Implementing a new provider

This extension implements probably the simplest possible provider. As far as the API goes all it supports is requesting a list of add-ons by type or a single add-on by ID. These functions pass add-on objects to the callbacks. For this add-on these objects are held in a database so that code does some fairy uninteresting (and horribly synchronous) sql queries and generates objects that the API expects.

Adding new types to the UI

Perhaps the hardest part of this extension is getting the new type of add-on to display in the UI. Unfortunately one thing that we haven’t implemented so far is any kind of auto-discovery of add-on types. Instead the UI works from a mostly hardcoded list. This is something that we think it would be nice to change but at the moment it seems unlikely that we wiull get time to before Firefox 4, unless someone wants to volunteer to do some of the work.

The demo extension works around this restriction by inserting some elements into the add-ons manager window whenever it detects it opening. In particular it adds an item to the category list with a value attribute “addons://list/user-script”. The add-ons manager UI uses this kind of custom URL to decide what to display when a category is selected. In this case it means displaying the normal list view (that plugins and extensions currently use) and to ask the API for add-ons of the type “user-script”. There is also some code there that overrides the normal string bundle that the manager uses to localize the text in the UI to allow adding in some additional strings. The code I am showing is of course badly written in that it is hardcoded and so could not be localized, please forgive me for cutting corners with the demo.

That is basically all that is needed to have the UI work to display the new add-ons from the registered provider however the demo also throws in some style rules to pretty things up with a custom icon

Notifying the UI of changes

When you implement your own provider you have to be sure to send out appropriate notifications whenever changes to the add-ons you manager happen so that any UI can update accordingly. I won’t go into too much detail here, hopefully the AddonListener and InstallListener API covers the events you need to know about enough. You can see the script database send out some of these notifications.

Get the full code

This has been a very short overview of the highlights of this demo, hopefully enough for the interested to pick up the code and make use of it themselves. The full source of the extension is available from the mercurial repository. Right now I wouldn’t really release this as an extension. As I’ve mentioned it uses synchronous sql queries (on every page load no less!) and cannot be localized. These things can be fixed but this was just made as a demo in basically one evening to show off the sorts of things that are possible with the new add-ons manager.

How do restartless add-ons work?

I blogged a short time ago about how we’re adding support for a new form of add-on to Firefox that can install and uninstall without needing to restart the application. Since then I’ve been finalizing a specification for how the platform will load these add-ons, trying to keep it simple but still give developers everything they commonly need. The planned specification is now available and if developers have comments then I’d like to hear them. Currently there isn’t a version of Firefox that implements it but that should change in the next day or so when I make the changes to the add-ons manager project branch and very soon when it all lands on trunk.

My hope is that once on trunk this spec won’t change but obviously this is quite new so we may see changes for a short time if add-on developers come across problems.