An update for hgchanges

Nearly a year ago I showed off the first version of my webapp for displaying recent changes in mercurial repositories. I’ve heard directly from a number of people that use it but it’s had a few problems that I’ve spent some of my spare time trying to solve. I’m now pleased to say that a new version is up and running. I’m not sure it’s accurate to call this a rewrite since it was entirely incremental but when I look back over of the code changes there really isn’t much that wasn’t touched in some way or other. So what’s new? For you, a person using the site absolutely nothing! So what on earth have I been doing rewriting the code?

Hopefully I’ve been making the thing a whole lot more stable. Eagle eyed users may have noticed that periodically the site would stop updating, sometimes for days at a time. The site is split into two pieces. One is a django website to display the changes. The other is the important bit that reads the change data out of mercurial and puts it into a database that the website can use. I do this caching because it would be too expensive to work it out on the fly. The problem is that the process of reading the data out of mercurial meant keeping a local version of every repository on the server (some 5GB for the four currently tracked) and reading the changes using mercurial’s python API was slow and used a lot of memory. The shared hosting environment that I run this out of kept killing the process when it used too many resources. Normally it could recover but occasionally it would need a manual kick. Worse there is some mercurial bug where sometimes pulling a repository will end up with an inconsistent database. It’s rare so many people don’t see it but when you’re pulling repositories every ten minutes it starts happing every couple of weeks, again requiring manual work to fix the problem.

I did dabble with getting this into shape so I could run it on a mozilla server. PAAS seemed like the obvious option but after a lot of work it turned out that the database size restrictions there made this impossible. Since then I’ve seem so many horror stories about PAAS falling over that I think I’m glad I never got there. I also tried getting a dedicated server from Mozilla to run this on but they were quite rightly wary when I mentioned that the process used a bunch of memory and wanted harder figures before committing, unfortunately I never found out a way to give those figures.

The final solution has been ripping out the mercurial API dependency. Now rather than needing local mercurial repositories the import process instead talks to them over the web. The default mercurial web APIs aren’t great but luckily all of the repositories I care about have pushlog installed which has a good JSON API for retrieving recently pushed changesets. So now the process is to pull the recent pushlog, get the patches for every changeset and parse them to work out which files have changed. By not having to load the mercurial structures there is far less memory used and since the process now has to delay as it downloads each patch file it keeps the CPU usage low.

I’ve also made the database itself a lot more efficient. Previously I recorded every changeset for every repository, but many repositories have the same changeset in them. So by noting that the database gets a little more complicated but uses much less space and since you only have to parse the changeset once the import process becomes faster. It took me a while to get the website performing as fast as before with these changes but I think it’s there now (barring some strange slowness in Django that I can’t identify). I’ve also turned on some quite aggressive caching both on the client and server side, if someone has visited the same page as you’re trying to access recently then it should load super fast.

So, the new version is up and running at the same location as the old. Chances are you’re using it already so if you’re noticing any unexpected slowness or something not looking right please let me know.

Get notifications about changes to any directory in mercurial

Back in the old days when we used CVS of all things for our version control we had a wonderful tool called bonsai to help query the repository for changes. You could list changes on a per directory basis if you needed which was great for keeping an eye on certain chunks of code. I recall there being a way of getting an RSS feed from it and I used it when I was the module owner of the extension manager to see what changes were landed that I hadn’t noticed in bugs.

Fast forward to today and we use mercurial instead. When we switched there was much talk of how we’d get tool parity with CVS but bonsai is something that has never been replaced fully. Oh hgweb is decent at looking at individual files and browsing the tree, but you can’t get that list of changes per directory from it. I believe you can use the command line to do it but who wants to do that? Lately I’ve been finding need of those directory RSS feeds more and more. We’re now periodically uplifting the Add-on SDK repository to mozilla-central, it’s really important to spot changes that have been made to that directory in mozilla-central so we can also land them in our git repository and not clobber them the next time we uplift. I’m also the module owner of toolkit, which is a pretty big sprawling set of files. It seems like everytime I look I find something that landed without me noticing. I don’t make for a good module owner if I’m not keeping an eye on things so I’d really like to see when new files are added there.

So I introduce the Hg Change Feed, the result of mostly just a few days of work. Every 10 minutes it pulls new changes from mozilla-central and mozilla-inbound. A mercurial hook looks over the changes and adds information about them to a MySQL database. Then a simple django app displays that information. As you browse through the directories in the tree it shows only changesets that affected files beneath that directory. For any directory you can also get an RSS feed of the same. Plug that into IFTTT and you have an automated system to notify you in pretty much any way you’d like about new changes you’d be interested in.

Some simple examples. For tracking changes to Add-on SDK I’m watching http://hgchanges.fractalbrew.com/mozilla-inbound/file/addon-sdk/source. For toolkit I’m looking at http://hgchanges.fractalbrew.com/mozilla-inbound/file/toolkit?types=added. Types takes a comma separated list of “added”, “removed” and “modified” to filter which changes you’re interested in. There’s no UI on the site for changing that right now, you’re welcome to add some!

One other neat trick that this does is mostly ignore merge changesets. Only if a merge actually makes a change not already present in either of the merge parents (mostly happens when resolving merge conflicts) will it show up in the list of changes, because really you don’t need to hear about changes twice.

So play with it, let me know if you find it useful or if you think things are missing. I can also add other mercurial repositories if people want. Some caveats:

  • It only retains the last 2000 changesets from any repository in an effort to keep the DB size small and fast, it also only shows the last 200 changesets for each page, or just the last 20 in the feeds. These can be tweaked easily enough and I’ve done basically no benchmarking to say those are the right values.
  • The site isn’t as fast as I’d like, particularly listing changes for the top level directory takes nearly 5 seconds. I’ve thrown some basic caching in place to help alleviate that for now. I bet someone who has more MySQL and django experience than me could tell me what I’m doing wrong.
  • I’m off on vacation tomorrow so I guess I’m announcing this then running away, sorry if that means it takes me a while to respond to comments.

Want to help out and make it better? Go nuts with the source. There’s a readme that hopefully explains how to set up your own instance.

I must be missing something in the clouds

For a long time now there have been web applications mirroring pretty much all the applications I use locally, email, calendar, spreadsheets, etc. I keep looking at these and feeling like I should jump on the bandwagon, after all lots of the people I work with use them and rave about them so they must be great right? The problem is I can’t figure out what I am actually missing, and most of the time I can spot immediately things I would miss by moving to them.

Obviously one clear benefit is that they are available anywhere in the world, you just need access to any computer with a modern web-browser. But you know what? Wherever I go in the world I take my laptop, or if I don’t it is because I really want to relax and be offline completely. About the only critical thing that I might need to get updates on is my mail, which I do have a webmail access to anyway.

The online services seem to fall down for me in a bunch of ways:

  • I like to be able to run applications separately from my browser. I’ll grant you that tools like Prism make this sort of thing possible so that failure is going away slowly.
  • No matter how good browsers become I don’t believe HTML will ever create as good a UI as a real application can. For the most part they are restricted to a single window interface, with pseudo windows hovering above looking nothing like platform native.
  • They need you to be online (let’s ignore gears and stuff for the moment, I haven’t found the technology to be quite there enough yet). As I said I take my laptop everywhere. I can look at all my mail and calendar without needing to pay for an internet connection in some random hotspot.
  • They simply don’t have the features that my local apps do. I expect this to change over the years but many of the online offerings are basic at best.
  • How do I back up my data? Seriously, if I want to back up my gmail or google calendar what do I do? If I want to back up my local mail and calendar I just plug in a hard disk and let OSX deal with it. Obviously the opposite to this is that if my machine goes down then the online service will still be there and I’ll only have a potentially stale backup, but my backup is never more than a week old, and I can tell you I’ve lost more data over the years due to online systems going down than local machines breaking.

So here I am, wondering (again) what to do about task management. I keep feeling drawn to things like Remember The Milk because they are all online and Web2.0ish, but I’m not sure quite why. I’m sure I must be missing something critical about using the online apps, but I just can’t figure out what it is.

Keeping Track

In the past year or so that I’ve been working for Mozilla I’ve found myself slowly working my way through a bunch of different ways to keep track of all of the work on my plate. I still don’t think I’ve found the best way so I wondered what other people do to manage such things?

For a rough idea of my work, most of my work is bug oriented. Either bugs that I want to work on when I get some time, bugs I am actively working on, bugs I am waiting for review on, bugs I am reviewing, bugs I am waiting to check in, bugs I want to backport to a branch… and so on. Then there is other work like planning future work and working with extension authors to help resolve their problems.

A few different approaches I’ve tried:

Bug ASSIGNED status

Simplest of all, gives me a long list of stuff I am looking at. Totally fails for prospective work though and the list gets somewhat daunting.

Status whiteboard flags

I tried for a while sticking special flags into the status whiteboard that could then be spotted in special bug queries so I could see what state each bug was in. This kind of worked, though I imagine irritated some with bugspam as I maintained the flags and a couple of times other people changed the flags.

Bugzilla tagging

Bugzilla has this tagging feature that basically lets you create named buglists. Basically like using the whiteboard stuff without having to use the whiteboard. Sadly Bugzilla’s tagging UI sucks really hard.

Bookmark tagging

With Firefox 3 I can bookmark my bugs and tag them, and use smart bookmark folders as my buglists right? Well sort of, the folder list isn’t really enough detail and there are issues with manually created smart folders that kept losing my bookmarks.

Task management software

Finally bit the bullet and stuck all of this stuff into some task managent software (Omnifocus as it happens). This is so far going ok but seems to require more dedication to keep the list up to date, I think because of having to switch to a different application so much.

So what are other people doing?