An update for hgchanges

Nearly a year ago I showed off the first version of my webapp for displaying recent changes in mercurial repositories. I’ve heard directly from a number of people that use it but it’s had a few problems that I’ve spent some of my spare time trying to solve. I’m now pleased to say that a new version is up and running. I’m not sure it’s accurate to call this a rewrite since it was entirely incremental but when I look back over of the code changes there really isn’t much that wasn’t touched in some way or other. So what’s new? For you, a person using the site absolutely nothing! So what on earth have I been doing rewriting the code?

Hopefully I’ve been making the thing a whole lot more stable. Eagle eyed users may have noticed that periodically the site would stop updating, sometimes for days at a time. The site is split into two pieces. One is a django website to display the changes. The other is the important bit that reads the change data out of mercurial and puts it into a database that the website can use. I do this caching because it would be too expensive to work it out on the fly. The problem is that the process of reading the data out of mercurial meant keeping a local version of every repository on the server (some 5GB for the four currently tracked) and reading the changes using mercurial’s python API was slow and used a lot of memory. The shared hosting environment that I run this out of kept killing the process when it used too many resources. Normally it could recover but occasionally it would need a manual kick. Worse there is some mercurial bug where sometimes pulling a repository will end up with an inconsistent database. It’s rare so many people don’t see it but when you’re pulling repositories every ten minutes it starts happing every couple of weeks, again requiring manual work to fix the problem.

I did dabble with getting this into shape so I could run it on a mozilla server. PAAS seemed like the obvious option but after a lot of work it turned out that the database size restrictions there made this impossible. Since then I’ve seem so many horror stories about PAAS falling over that I think I’m glad I never got there. I also tried getting a dedicated server from Mozilla to run this on but they were quite rightly wary when I mentioned that the process used a bunch of memory and wanted harder figures before committing, unfortunately I never found out a way to give those figures.

The final solution has been ripping out the mercurial API dependency. Now rather than needing local mercurial repositories the import process instead talks to them over the web. The default mercurial web APIs aren’t great but luckily all of the repositories I care about have pushlog installed which has a good JSON API for retrieving recently pushed changesets. So now the process is to pull the recent pushlog, get the patches for every changeset and parse them to work out which files have changed. By not having to load the mercurial structures there is far less memory used and since the process now has to delay as it downloads each patch file it keeps the CPU usage low.

I’ve also made the database itself a lot more efficient. Previously I recorded every changeset for every repository, but many repositories have the same changeset in them. So by noting that the database gets a little more complicated but uses much less space and since you only have to parse the changeset once the import process becomes faster. It took me a while to get the website performing as fast as before with these changes but I think it’s there now (barring some strange slowness in Django that I can’t identify). I’ve also turned on some quite aggressive caching both on the client and server side, if someone has visited the same page as you’re trying to access recently then it should load super fast.

So, the new version is up and running at the same location as the old. Chances are you’re using it already so if you’re noticing any unexpected slowness or something not looking right please let me know.

Developer Tools meet-up in Portland

Two weeks ago the developer tools teams and a few others met in the Portland office for a very successful week of discussions and hacking. The first day was about setting the stage for the week and working out what everyone was going to work on. Dave Camp kicked us off with a review of the last six months in developer tools and talked about what is going to be important for us to focus on in 2014. We then had a little more in-depth information from each of the teams. After lunch a set of lightning talks went over some projects and ideas that people had been working on recently.

After that everyone got started prototyping new ideas, hacking on features and fixing bugs. The amount of work that happens at these meet-ups is always mind-blowing and this week was no exception, even one of our contributors got in on the action. Here is a list of the things that the team demoed on Friday:

This only covers the work demoed on Friday, a whole lot more went on during the week as a big reason for doing these meet-ups is so that groups can split off to have important discussions. We had Darrin Henein on hand to help out with UX designs for some of the tools and Kyle Huey joined us for a couple of days to help work out the final kinks in the plan for debugging workers. Lot’s of work went on to iron out some of the kinks in the new add-on SDK widgets for Australis, there were discussions about memory and performance tools as well as some talk about how to simplify child processes for Firefox OS and electrolysis.

Of course there was also ample time in the evenings for the teams to socialise. One of the downsides of being a globally distributed team is that getting to know one another and building close working relationships can be difficult over electronic forms of communication so we find that it’s very important to all come together in one place to meet face to face. We’re all looking forward to doing it again in about six months time.

Firefox now ships with the add-on SDK

It’s been a long ride but we can finally say it. This week Firefox 21 shipped and it includes the add-on SDK modules.

We took all the Jetpack APIs and we shipped them in Firefox!What does this mean? Well for users it means two important things:

  1. Smaller add-ons. Since they no longer need to ship the APIs themselves add-ons only have to include the unique code that makes them special. That’s something like a 65% file-size saving for the most popular SDK based add-ons, probably more for simpler add-ons.
  2. Add-ons will stay compatible with Firefox for longer. We can evolve the modules in Firefox that add-ons use so that most of the time when changes happen to Firefox the modules seamlessly shift to keep working. There are still some cases where that might be impossible (when a core feature is dropped from Firefox for example) but hopefully those should be rare.

To take advantage of these benefits add-ons have to be repacked with a recent version of the SDK. We’re working on a plan to do that automatically for existing add-ons where possible but developers who want to get the benefits right now can just repack their add-ons themselves using SDK 1.14 and using cfx xpi --strip-sdk, or using the next release of the SDK, 1.15 which will do that by default.

Get notifications about changes to any directory in mercurial

Back in the old days when we used CVS of all things for our version control we had a wonderful tool called bonsai to help query the repository for changes. You could list changes on a per directory basis if you needed which was great for keeping an eye on certain chunks of code. I recall there being a way of getting an RSS feed from it and I used it when I was the module owner of the extension manager to see what changes were landed that I hadn’t noticed in bugs.

Fast forward to today and we use mercurial instead. When we switched there was much talk of how we’d get tool parity with CVS but bonsai is something that has never been replaced fully. Oh hgweb is decent at looking at individual files and browsing the tree, but you can’t get that list of changes per directory from it. I believe you can use the command line to do it but who wants to do that? Lately I’ve been finding need of those directory RSS feeds more and more. We’re now periodically uplifting the Add-on SDK repository to mozilla-central, it’s really important to spot changes that have been made to that directory in mozilla-central so we can also land them in our git repository and not clobber them the next time we uplift. I’m also the module owner of toolkit, which is a pretty big sprawling set of files. It seems like everytime I look I find something that landed without me noticing. I don’t make for a good module owner if I’m not keeping an eye on things so I’d really like to see when new files are added there.

So I introduce the Hg Change Feed, the result of mostly just a few days of work. Every 10 minutes it pulls new changes from mozilla-central and mozilla-inbound. A mercurial hook looks over the changes and adds information about them to a MySQL database. Then a simple django app displays that information. As you browse through the directories in the tree it shows only changesets that affected files beneath that directory. For any directory you can also get an RSS feed of the same. Plug that into IFTTT and you have an automated system to notify you in pretty much any way you’d like about new changes you’d be interested in.

Some simple examples. For tracking changes to Add-on SDK I’m watching http://hgchanges.fractalbrew.com/mozilla-inbound/file/addon-sdk/source. For toolkit I’m looking at http://hgchanges.fractalbrew.com/mozilla-inbound/file/toolkit?types=added. Types takes a comma separated list of “added”, “removed” and “modified” to filter which changes you’re interested in. There’s no UI on the site for changing that right now, you’re welcome to add some!

One other neat trick that this does is mostly ignore merge changesets. Only if a merge actually makes a change not already present in either of the merge parents (mostly happens when resolving merge conflicts) will it show up in the list of changes, because really you don’t need to hear about changes twice.

So play with it, let me know if you find it useful or if you think things are missing. I can also add other mercurial repositories if people want. Some caveats:

  • It only retains the last 2000 changesets from any repository in an effort to keep the DB size small and fast, it also only shows the last 200 changesets for each page, or just the last 20 in the feeds. These can be tweaked easily enough and I’ve done basically no benchmarking to say those are the right values.
  • The site isn’t as fast as I’d like, particularly listing changes for the top level directory takes nearly 5 seconds. I’ve thrown some basic caching in place to help alleviate that for now. I bet someone who has more MySQL and django experience than me could tell me what I’m doing wrong.
  • I’m off on vacation tomorrow so I guess I’m announcing this then running away, sorry if that means it takes me a while to respond to comments.

Want to help out and make it better? Go nuts with the source. There’s a readme that hopefully explains how to set up your own instance.

Hacking on Tilt

Tilt, or 3D view as it is known in Firefox, is an awesome visual tool that really lets you see the structure of a webpage. It shows you just how deep your tag hierarchy goes which might give signs of your page being too complex or even help you spot errors in your markup that you wouldn’t otherwise notice. But what if it could do more? What if there were different ways to visualise the same page? What if even web developers could create their own visualisations?

I’ve had this idea knocking around in my head for a while now and the devtools work week was the perfect time to hack on it. Here are some results, click through for larger images:

Normal 3D view

Normal 3D view

Only give depth to links

Only give depth to links

Only give depth to links going off-site

Only give depth to links going off-site

Only give depth to elements that have a different style on hover

Only give depth to elements that have a different style on hover

This is all achieved with some changes to Firefox itself to make Tilt handle more generic visualisations along with an extension that then overrides Tilt’s default visualisation. Because the extension has access to everything Firefox knows about the webpage it can use some interested sources of data about the page, including those not found in the DOM. This one I particularly like, it makes the element’s depth proportional to the number of attached DOM event listeners:

Give depth to elements based on the number of attached event listeners

Give depth to elements based on the number of attached event listeners

Just look at that search box, and what’s up with the two buttons having different height?

The code just calls a JS function to get the height for each element displayed in 3D view. It’s really easy to use DOM functions to highlight different things about the elements and while I think some of the examples I made are interesting I think it will be more interesting to just let web-devs come up with and share their own visualisations. To that end I also demoed using scratchpad to write whatever function you like to control the visualisation. You can see a screencast of it in action.

Something that struck me towards the end of the week is that it could be awesome to pair this up with external sources of data like analytics. What about being able to view your page with links given depth proportional to how often users click them? Seems like an awesome way to really understand where your users are going and maybe why.

I’m hoping to get the changes to Firefox landed soon maybe with an additional patch to properly support extensibility of Tilt, right now the extension works by replacing a function in a JSM which is pretty hacky, but it wouldn’t be difficult to make it nicer than that. After that I’ll be interested to see what visualisation ideas others come up with.

Let’s just put it in Toolkit!

Toolkit is fast turning into the dumping ground of mozilla-central. Once upon a time the idea was simple. Any code that could be usefully shared across multiple applications (and in particular code that wasn’t large enough to deserve a module of its own) would end up in Toolkit. The rules were pretty simple, any code in there should work for any application that wants to use it. This didn’t always work exactly according to plan but we did our best to fix Seamonkey and Thunderbird incompatibilities as they came along.

This worked great when there was only one Firefox. Shared code went into m-c/toolkit, Firefox specific code went into m-c/browser. There were always complaints that more of the code in browser should be moved to toolkit so Seamonkey and other projects could make use of it but otherwise no big issues.

Now we have more than one Firefox: Firefox for desktop, Firefox for Android, B2G, Metro and who knows what else to come. Suddenly we want to share code across different Firefoxen, often different sets depending on the code, often depending on other pieces of code like services that aren’t available in all other applications. Keeping the rules as they stand now means that Toolkit isn’t the correct place for this code. So what do we do about this?

There only seem to be two sensible choices. Either we change the Toolkit rules to allow code that may not work in some applications, or we create a new catch-all module for this sort of code. And really those are just the same but with the need to find a new module owner for the new module. I’m going to ignore the third option which is to create a new module for each new piece of code like this as being hopelessly bureaucratic.

So, I’m proposing that we (by which I mean I as module owner) redefine the rules for Toolkit to be a little broader:

  • Any code in Toolkit should be potentially useful to multiple applications but it isn’t up to the author to make it work everywhere.
  • Patches to make code work in other applications will be accepted if not too invasive.
  • Any code in Toolkit that is called automatically by Gecko (like the add-ons manager) must work in all applications.

Any strong objections?

WebApp Tabs, version control and GitHub

The extension I’ve been working on in my spare time for the past couple of weeks is now available as a first (hopefully not too buggy) release. It lets you open WebApps in Thunderbird, properly handling loading new links into Firefox and making all features like spellchecking work in Thunderbird (most other extensions I found didn’t do this). You can read more about the actual extension at its homepage.

Mostly I’ve been really encouraged during the development of this at just how far our platform has come for developing restartless add-ons. When we first made it possible in Firefox 4 there was a whole list of things that were quite difficult to do but we’ve come a long way since then. While there are still things that are difficult there are lots of things that are now pretty straightforward. My add-on loads simple XUL overlays, style overlays, installs JS XPCOM components with category manager registration, all similar to older add-ons. In fact I’m struggling to think of things that it is still hard to do though I’m sure other more prolific developers will have plenty of comments on that!

The other thing I’ve been doing with this extension is experimenting with git and GitHub. I think it’s been an interesting experience, there are continual arguments over which is better between git and mercurial with many pros and cons listed. I think most of these were done some time ago before mercurial and git really matured because from what I’ve seen there is really little difference between the two. They have slightly different default branching styles, but both can do the same kind of branching that the other can if you want and there are a few other minor differences but nothing that would really make me all that bothered over deciding which to use. I think the only place where git has a bonus is with GitHub, and really as far as I can see there isn’t a reason why someone couldn’t develop a similar site backed by mercurial repositories, it’s just that no-one really has.

GitHub is pretty nice with built-in basic issue tracking and documentation though it still has some frustrating issues. It seems odd for example that you can’t fork your own project, only someone else’s, but that’s only a minor niggle really. As project hosting goes I can’t say I’ve come across anything better that I can remember.

Overlays without overlays in restartless add-ons

Perhaps the most common way of making changes to Firefox with an extension has always been using the overlay. For a window’s UI you can make changes to the underlying XUL document, add script elements to be executed in the context of the normal window’s code and add new stylesheets to the window to change how the UI looks.

Restartless add-ons change this around completely, the normal overlay and style-overlay mechanisms just aren’t available to restartless add-ons and this is likely to remain true for a while, these methods don’t clean up after themselves when the add-on is uninstalled.

This can make things hard, particularly for porting older add-ons to become restartless. I was in this situation earlier this weekend. I was working on porting David Ascher‘s WebTabs for Thunderbird to be restartless. I could have just moved all the script code over to bootstrap.js but in many ways it is nice to keep the code that works on the main UI separate to the code that runs for the preferences UI etc. Plus I like to play around with new ways of doing things so I developed a JS module I’m calling the OverlayManager.

The OverlayManager watches for new windows being opened and for every new window it can run JS script and apply CSS stylesheets to the window in a way that is easy to undo if the add-on is disabled at runtime. Although it can’t do any XUL modifications right now (I didn’t need any for this particular extension) it would be pretty easy to extend this to support a minimum about of XUL overlays.

Stylesheets are loaded by adding a HTML style tag to the XUL document, so they can be removed easily when the add-on is disabled. Scripts are handled in a way that may even be better than normal overlays. In the old system extension scripts all run in the same context as the window they overlay giving rise to the possibility of conflicts. Restartless add-ons shouldn’t do this since it makes removing the script code again much more troublesome. The OverlayManager handles it by creating a sandbox to run the script in. The sandbox’s prototype is set to the window the script is being run for meaning the script sees all the functions and objects of the window directly in its own scope but as long as it doesn’t modify any of the objects in the main window’s code all we have to do is throw away the sandbox to get rid of its JS.

There are a few things different of course. The script shouldn’t use load and unload event handlers for the window as it may get loaded well after the window does or unloaded well before. Instead the OverlayManager looks for an OverlayListener object in the script and calls load and unload methods on it, these are called either with the window’s real load and unload events or while the window is open normally. You also can’t reference code in the script from JS string blocks, so if you set onclick="myfunc()" on a XUL element it wouldn’t work because that would run in the main window scope which can’t see the sandbox code at that point. This tends to be pretty simple to get around by using addEventListener for all your events though.

You can see the existing state of the code on github and an example of the structure you’d pass to OverlayManager.addOverlays is in the bootstrap script for the same project. It is appropriately licensed so go nuts!

Update: I changed the stylesheets to use XML processing instructions to be more like they work currently and just for fun I implemented the very basics of document overlaying, almost totally untested though so YMMV.

Playing with windows in restartless (bootstrapped) extensions

Lots of people seem to be playing with the new support for restartless extensions (also known as bootstrapped extensions) coming in Firefox 4. Nothing could make me happier really. I’m not sure I can remember ever helping implement something which is will hopefully turn out to be a game changer for extension development in the future. The technical details of restartless extensions are talked about in a few places but one thing that is missing is what I think is probably going to be the most common code pattern in these extensions.

When you’re developing an extension it’s almost certain that you’re going to want to interact with the main browser window in some way. With the XUL based extensions this was a pretty simple process, normally your extension would be overlaying browser.xul and so any script you add there would automatically run for every new browser window and would have direct access to it (in fact this sometimes confused new developers who assumed their script would run just once and be globally available, not for every browser window opened).

In restartless extensions things are different. You have one script that runs when your extension is started and runs in its own sandbox so you have to take steps to get access to windows. Complicating matters your extension may be started when Firefox starts up and there are no browser windows or after when some already exist. The best way to go is when your extension is started to look for any existing browser windows and make your necessary modifications and then wait for new windows to open and modify those too.

Since this is such a common case (and vlad was trying to figure it out on IRC) I put together a very simple bootstrap script that just does this. It calls the setupBrowserUI function for every window that needs to be initialised by your extension and tearDownBrowserUI for every window that must be cleaned up when your extension is being disabled. This code isn’t the only way to do all this but it is fairly straightforward and works, feel free to take it and modify it for use in your own extensions. I’ll probably get it put onto MDC too.

const Cc = Components.classes;
const Ci = Components.interfaces;

var WindowListener = {
  setupBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to add UI or anything to the browser window
    // document.getElementById() etc. will work here
  },

  tearDownBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to remove UI or anything from the browser window
    // document.getElementById() etc. will work here
  },

  // nsIWindowMediatorListener functions
  onOpenWindow: function(xulWindow) {
    // A new window has opened
    let domWindow = xulWindow.QueryInterface(Ci.nsIInterfaceRequestor)
                             .getInterface(Ci.nsIDOMWindow);

    // Wait for it to finish loading
    domWindow.addEventListener("load", function listener() {
      domWindow.removeEventListener("load", listener, false);

      // If this is a browser window then setup its UI
      if (domWindow.document.documentElement.getAttribute("windowtype") == "navigator:browser")
        WindowListener.setupBrowserUI(domWindow);
    }, false);
  },

  onCloseWindow: function(xulWindow) {
  },

  onWindowTitleChange: function(xulWindow, newTitle) {
  }
};

function startup(data, reason) {
  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.setupBrowserUI(domWindow);
  }

  // Wait for any new browser windows to open
  wm.addListener(WindowListener);
}

function shutdown(data, reason) {
  // When the application is shutting down we normally don't have to clean
  // up any UI changes made
  if (reason == APP_SHUTDOWN)
    return;

  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.tearDownBrowserUI(domWindow);
  }

  // Stop listening for any new browser windows to open
  wm.removeListener(WindowListener);
}

How to extend the new Add-ons Manager (or how I built a simple greasemonkey clone in an evening)

One of the goals of the new add-ons manager API was to create something that was itself extensible. A couple of times in the past we’ve had to add new types of add-ons to the UI like Plugins and Personas. In both cases squeezing them into the UI was something of a kludge involving a bunch of custom code for each case. We already have a number of new types of add-ons that we want to add, things like search plugins which are currently managed by their own custom UI.

To help simplify this the API is largely type-agnostic. Internally it uses a number of so-called add-on “providers”, each of which may make information about add-ons available. Each provider is basically a JavaScript object with functions defined that the API can call to ask for information about add-ons that the provider knows about. The provider then just has to pass back JavaScript objects to represent each add-on. Each of these must have at minimum a set of required properties and functions and may also include a set of optional properties. The full set is defined in the API documentation.

With this design the user interface doesn’t need to care about implementation details of any of the providers, how they store their data or what exactly their add-ons are and do. Because each gives objects that obeys the same interface it can just display and manipulate them.

To try to show this all off I recently put together a small demo extension for the Mozilla Summit that registers a new type of add-on to be displayed in the main add-ons manager. This is a short overview of some of the highlights and I’ll make the code available for people to look at and take examples from. The add-on was a basic implementation of Greasemonkey allowing user scripts to be installed, managed through the add-ons manager and do it all as a restartless add-on.

Making a restartless add-on

Add-ons don’t have to be developed with Jetpack to make them restartless, although the Jetpack SDK certainly makes things easier on you, at the expense of less access to the internals of the platform.

The first thing to learn about making a restartless add-on is that you can forget about using XUL overlays or registering XPCOM components to be called at startup. Neither are supported at the moment, and maybe never will. Instead you have to provide a bootstrap script. This is a simple “bootstrap.js” file in the root of the extension that should include a “startup” and “shutdown” function. These are called whenever Firefox wants to start or stop your add-on either because the application is starting up or shutting down or the add-on is being enabled or disabled. You can also provide “install” and “uninstall” methods to be notified of those cases but that is probably unnecessary in most cases.

At startup the demo extension does some basic things. It registers for some observer notifications, registers a new add-on provider (I’ll talk more about that below) and does a little work to include itself in the add-ons manager UI (again, see below).

The rule is this. Anything your add-on does after being started must be undone by the shutdown function. The shutdown function often ends up being the inverse of startup, here it removes observer notification registrations, unregisters the add-on provider and removes itself from the UI. It also shuts down a database if it was opened.

Implementing a new provider

This extension implements probably the simplest possible provider. As far as the API goes all it supports is requesting a list of add-ons by type or a single add-on by ID. These functions pass add-on objects to the callbacks. For this add-on these objects are held in a database so that code does some fairy uninteresting (and horribly synchronous) sql queries and generates objects that the API expects.

Adding new types to the UI

Perhaps the hardest part of this extension is getting the new type of add-on to display in the UI. Unfortunately one thing that we haven’t implemented so far is any kind of auto-discovery of add-on types. Instead the UI works from a mostly hardcoded list. This is something that we think it would be nice to change but at the moment it seems unlikely that we wiull get time to before Firefox 4, unless someone wants to volunteer to do some of the work.

The demo extension works around this restriction by inserting some elements into the add-ons manager window whenever it detects it opening. In particular it adds an item to the category list with a value attribute “addons://list/user-script”. The add-ons manager UI uses this kind of custom URL to decide what to display when a category is selected. In this case it means displaying the normal list view (that plugins and extensions currently use) and to ask the API for add-ons of the type “user-script”. There is also some code there that overrides the normal string bundle that the manager uses to localize the text in the UI to allow adding in some additional strings. The code I am showing is of course badly written in that it is hardcoded and so could not be localized, please forgive me for cutting corners with the demo.

That is basically all that is needed to have the UI work to display the new add-ons from the registered provider however the demo also throws in some style rules to pretty things up with a custom icon

Notifying the UI of changes

When you implement your own provider you have to be sure to send out appropriate notifications whenever changes to the add-ons you manager happen so that any UI can update accordingly. I won’t go into too much detail here, hopefully the AddonListener and InstallListener API covers the events you need to know about enough. You can see the script database send out some of these notifications.

Get the full code

This has been a very short overview of the highlights of this demo, hopefully enough for the interested to pick up the code and make use of it themselves. The full source of the extension is available from the mercurial repository. Right now I wouldn’t really release this as an extension. As I’ve mentioned it uses synchronous sql queries (on every page load no less!) and cannot be localized. These things can be fixed but this was just made as a demo in basically one evening to show off the sorts of things that are possible with the new add-ons manager.