The offending changeset that broke hgchanges yesterday turns out to be a merge from an ancient branch to current tip. That makes the diff insanely huge which is why things like hgweb were tripping over it. Kwierso point out that just ignoring those changesets would solve the problem. It’s not ideal but since in this case they aren’t useful changesets I’ve gone ahead and done that and so hgchanges is now updating again.
My handy tool for tracking changes to directories in the mozilla mercurial repositories is going to be broken for a little while. Unfortunately a particular changeset seems to be breaking things in ways I don’t have the time to fix right now. Specifically trying to download the raw patch for the changeset is causing hgweb to timeout. Short of finding time to debug and fix the problem my only solution is to wait until that patch is old enough that it no longer attempts to index it. That could take a week or so.
Obviously I’ll happily accept patches to fix this problem sooner.
I have been a little lax in my duty of keeping the list of peers for Toolkit up to date and there have been a few notable exceptions. Thankfully we’re good about disregarding rules when it makes sense to and so many people who should be peers have already been doing reviews. Of course that’s no use to new contributors trying to figure out who should review what so I am grateful to someone who prodded me into updating the list.
As I was doing so I came to the conclusion that there is a lot of overlap between Firefox code and Toolkit code. Lots of patches touch both at the same time and it often doesn’t make a lot of sense to require different reviewers there. I also couldn’t think of a reason why someone would be a trusted reviewer of Firefox code and yet not be trusted to review Toolkit code. Looking at the differences in the lists of peers confirmed that all those missing really should be Toolkit peers too.
So going forwards I have decided that Firefox peers will automatically be considered to be Toolkit peers. That means I can announce a whole bunch of new people who are now Toolkit peers, please congratulate them in the usual way, by flooding their review queue:
- Ehsan Akhgari
- Mike de Boer
- Mike Conley
- Georg Fritzsche
- Mark Hammond
- Felipe Gomes
- Gijs Kruitbosch
- Florian Quèze
- Tim Taubert
You might ask if the reverse should hold true, should all Toolkit peers be Firefox peers? i.e. should we just merge the lists. I leave that to the Firefox owner to decide but I will say that there are a few pieces of Toolkit that are very much not front-end and so in some cases I could see a reviewer for that area not needing to be listed in the Firefox list, not because they wouldn’t be trusted to turn down the patches they couldn’t review but just because there would be almost no patches in their area in Firefox. Maybe that case is so rare that it isn’t worth the hassle of two lists though.
Slightly belated in some cases but I’d like to formally welcome four new toolkit peers. Paolo Amadini, Matthew Noorenberghe, Jared Wein and Irving Reid have all shown themselves to be well capable of reviewing patches in any of the toolkit code. Paolo, Matt and Jared actually got added a few months ago but apparently I failed to make an announcement at the time. Irving was added just last week. Please congratulate them all and don’t go too hard on their review queues!
Also if you think there are others who should be peers of Toolkit (or current peers that are no longer relevant) then please let me know.
Nearly a year ago I showed off the first version of my webapp for displaying recent changes in mercurial repositories. I’ve heard directly from a number of people that use it but it’s had a few problems that I’ve spent some of my spare time trying to solve. I’m now pleased to say that a new version is up and running. I’m not sure it’s accurate to call this a rewrite since it was entirely incremental but when I look back over of the code changes there really isn’t much that wasn’t touched in some way or other. So what’s new? For you, a person using the site absolutely nothing! So what on earth have I been doing rewriting the code?
Hopefully I’ve been making the thing a whole lot more stable. Eagle eyed users may have noticed that periodically the site would stop updating, sometimes for days at a time. The site is split into two pieces. One is a django website to display the changes. The other is the important bit that reads the change data out of mercurial and puts it into a database that the website can use. I do this caching because it would be too expensive to work it out on the fly. The problem is that the process of reading the data out of mercurial meant keeping a local version of every repository on the server (some 5GB for the four currently tracked) and reading the changes using mercurial’s python API was slow and used a lot of memory. The shared hosting environment that I run this out of kept killing the process when it used too many resources. Normally it could recover but occasionally it would need a manual kick. Worse there is some mercurial bug where sometimes pulling a repository will end up with an inconsistent database. It’s rare so many people don’t see it but when you’re pulling repositories every ten minutes it starts happing every couple of weeks, again requiring manual work to fix the problem.
I did dabble with getting this into shape so I could run it on a mozilla server. PAAS seemed like the obvious option but after a lot of work it turned out that the database size restrictions there made this impossible. Since then I’ve seem so many horror stories about PAAS falling over that I think I’m glad I never got there. I also tried getting a dedicated server from Mozilla to run this on but they were quite rightly wary when I mentioned that the process used a bunch of memory and wanted harder figures before committing, unfortunately I never found out a way to give those figures.
The final solution has been ripping out the mercurial API dependency. Now rather than needing local mercurial repositories the import process instead talks to them over the web. The default mercurial web APIs aren’t great but luckily all of the repositories I care about have pushlog installed which has a good JSON API for retrieving recently pushed changesets. So now the process is to pull the recent pushlog, get the patches for every changeset and parse them to work out which files have changed. By not having to load the mercurial structures there is far less memory used and since the process now has to delay as it downloads each patch file it keeps the CPU usage low.
I’ve also made the database itself a lot more efficient. Previously I recorded every changeset for every repository, but many repositories have the same changeset in them. So by noting that the database gets a little more complicated but uses much less space and since you only have to parse the changeset once the import process becomes faster. It took me a while to get the website performing as fast as before with these changes but I think it’s there now (barring some strange slowness in Django that I can’t identify). I’ve also turned on some quite aggressive caching both on the client and server side, if someone has visited the same page as you’re trying to access recently then it should load super fast.
So, the new version is up and running at the same location as the old. Chances are you’re using it already so if you’re noticing any unexpected slowness or something not looking right please let me know.
Two weeks ago the developer tools teams and a few others met in the Portland office for a very successful week of discussions and hacking. The first day was about setting the stage for the week and working out what everyone was going to work on. Dave Camp kicked us off with a review of the last six months in developer tools and talked about what is going to be important for us to focus on in 2014. We then had a little more in-depth information from each of the teams. After lunch a set of lightning talks went over some projects and ideas that people had been working on recently.
After that everyone got started prototyping new ideas, hacking on features and fixing bugs. The amount of work that happens at these meet-ups is always mind-blowing and this week was no exception, even one of our contributors got in on the action. Here is a list of the things that the team demoed on Friday:
- Joe Walker showed off CSS and JS code coverage tools.
- Rob Campbell showed us status codes and image previews in the network monitor as well as Mihai’s patches for nicer output and cd in the web console.
- Anton Kovalyov demonstrated a sourceeditor add-on that embeds metrics and contextual information into the code you see.
- Alex Poirot showed UI to select which frames in the page or Firefox OS app to monitor in the developer toolbox.
- Brandon Benvie amazed us with autocompletion and type inference using tern.
- Irakli Gozalishvili developed a plugin for Light Table that allowed it to connect to Firefox through the debugging protocol.
- Victor Porof made game developers happy with a demo of a canvas tracer.
- Patrick Brosset made the inspector show badly formed style entries and synced changes between the inspector and style editor.
- I made the box model view in the inspector editable for on the fly layout tinkering.
- Mike Ratcliffe added events and font tooltips to the inspector as well as creating a test log syntax highlighter for Sublime Text.
- Jordan Santell created a tool for viewing, editing and debugging web audio nodes.
- Erik Vold made Itchpad able to live edit add-ons.
- Brian Grinstead improved itchpad with drag and drop file manipulation and file/folder watching.
- Jeff Griffiths showed off a telemetry dashboard.
- One of our contributors, Optimizer, dialled in to show off his storage inspector.
- Panagiotis Astithas played with some widgets for a small live view of Firefox’s performance.
- Will Bamberg created an add-on to add tooltips to CSS rules in the inspector which showed documentation from MDN.
- Jan Keromnes showed more metrics and settings for the developer HUD in Firefox OS.
- Soledad Penades created app templates to help new developers get started quickly.
- Ryan Stinnett wirelessly debugged of Firefox OS with autodiscovery.
- Jan Odvarko showed off Firebug debugging dynamic scripts like eval.
- Eddy Bruel demonstrated some of his work on making workers debuggable.
- Mateo Ferretti showed us a game that you play by using the developer tools.
This only covers the work demoed on Friday, a whole lot more went on during the week as a big reason for doing these meet-ups is so that groups can split off to have important discussions. We had Darrin Henein on hand to help out with UX designs for some of the tools and Kyle Huey joined us for a couple of days to help work out the final kinks in the plan for debugging workers. Lot’s of work went on to iron out some of the kinks in the new add-on SDK widgets for Australis, there were discussions about memory and performance tools as well as some talk about how to simplify child processes for Firefox OS and electrolysis.
Of course there was also ample time in the evenings for the teams to socialise. One of the downsides of being a globally distributed team is that getting to know one another and building close working relationships can be difficult over electronic forms of communication so we find that it’s very important to all come together in one place to meet face to face. We’re all looking forward to doing it again in about six months time.
This week the whole devtools group has been sequestered in Mozilla’s Portland office having one of our regular meet-ups. As always it’s been a fantastically productive week with lots of demos to show for it. I’ll be writing a longer write-up later but I wanted to post about what I played with over the week.
My wife does the odd bit of web development on the side. For a long time she was a loyal Firebug user and a while ago I asked her what she thought of Firefox’s built in devtools. She quickly pointed out a couple of features that Firebug had that Firefox did not. As I was leaving for this week I mentioned I’d be with the devtools group and she asked whether her features had been fixed yet. It turns out that colour swatches had been but the box model still wasn’t editable. So I figured I could earn myself some brownie points by hacking on that this week.
The goal here is to be able to inspect an element on the page, pull up the box model and be able to quickly play with the margins, borders and padding to tweak the positioning until it looks right. Then armed with the right values you can go update your stylesheets. It saves a lot of trial and error with positioning.
It turned out to be relatively simple to implement a pretty full version. The feature allows you to click one of the box model values and type whatever value you like, in any CSS unit you prefer. If the size had been set in the stylesheet in some specific unit then that is what appears in the input box for you to change. Better yet as you type numbers the element updates in the page on the fly and you can use the arrow keys to increase/decrease the value until you’re happy. It’s a really natural way to play with the element’s position.
The changes made appear on the element so you can find them in the rule view pretty easily. This patch is based on an updated version of the box model view which is why it looks so different to existing Firefox, all my work does is make the numbers editable.
I actually completed this so quickly that I decided to take this one step further. One thing missing from the box model display is information about border colours. So I added some colour swatches for each border and made them editable with the regular devtools colour picker.
Both of these patches are pretty much complete but they’ll have to wait for the new box model highlighter to be complete before they can be reviewed and land.
Two years ago I blogged about how it had been six years since I wrote my first patch for Firefox. Today I get to say that it’s been six years since I started getting paid to do what I love, working for Mozilla. In that time I’ve moved to a new continent, found a wife (through Mozilla no less!), progressed from coder to module owner to manager, seen good friends leave for other opportunities and others join and watched (dare I say helped?) Mozilla, and the Firefox team in particular, grow from the small group that I joined in 2007 to the large company that will soon surpass 1000 employees.
One of the things I’ve always appreciated about Mozilla is how flexible they can be about where you work. Recently my wife and I decided to move away from the bay area to be closer to family. It was a hard choice to make as it meant leaving a lot of friends behind but one thing that made it easier was knowing that I wouldn’t have to factor work into that choice. Unlike many other companies I know there is no strict requirement that everyone work from the office. True it is encouraged and it sometimes makes sense to require it for some employees, particularly when starting out, but I knew that when it came time to talk to my manager about it he wouldn’t have a problem with me switching to working remote. Of course six years ago when I started I was living in the UK and remained so for my first two years at Mozilla so he had a pretty good idea that I could handle it, and at least this time I’m only separated from the main office by a short distance and no time-zones.
The web has changed a lot in the last six years. Back then we were working on Firefox 3, the first release to contain the awesomebar and a built-in way to download extensions from AMO. Twitter and Facebook had only been generally available for about a year. The ideas for CSS3 and HTML5 were barely written, let alone implemented. If you had told me back then that you’d be able to play a 3D game in your browser with no additional plugins, or watch videos without flash I’d have probably thought they were crazy pipe-dreams. We weren’t even Jitting our JS code back then. Mozilla, along with other browser makers, are continuing to prove that HTML, CSS and JS are winning combinations that we can build on to make the future of the web open, performant and powerful. I can’t wait to see what things will be like in another six years.
It’s been a long ride but we can finally say it. This week Firefox 21 shipped and it includes the add-on SDK modules.
What does this mean? Well for users it means two important things:
- Smaller add-ons. Since they no longer need to ship the APIs themselves add-ons only have to include the unique code that makes them special. That’s something like a 65% file-size saving for the most popular SDK based add-ons, probably more for simpler add-ons.
- Add-ons will stay compatible with Firefox for longer. We can evolve the modules in Firefox that add-ons use so that most of the time when changes happen to Firefox the modules seamlessly shift to keep working. There are still some cases where that might be impossible (when a core feature is dropped from Firefox for example) but hopefully those should be rare.
To take advantage of these benefits add-ons have to be repacked with a recent version of the SDK. We’re working on a plan to do that automatically for existing add-ons where possible but developers who want to get the benefits right now can just repack their add-ons themselves using SDK 1.14 and using
cfx xpi --strip-sdk, or using the next release of the SDK, 1.15 which will do that by default.
Back in the old days when we used CVS of all things for our version control we had a wonderful tool called bonsai to help query the repository for changes. You could list changes on a per directory basis if you needed which was great for keeping an eye on certain chunks of code. I recall there being a way of getting an RSS feed from it and I used it when I was the module owner of the extension manager to see what changes were landed that I hadn’t noticed in bugs.
Fast forward to today and we use mercurial instead. When we switched there was much talk of how we’d get tool parity with CVS but bonsai is something that has never been replaced fully. Oh hgweb is decent at looking at individual files and browsing the tree, but you can’t get that list of changes per directory from it. I believe you can use the command line to do it but who wants to do that? Lately I’ve been finding need of those directory RSS feeds more and more. We’re now periodically uplifting the Add-on SDK repository to mozilla-central, it’s really important to spot changes that have been made to that directory in mozilla-central so we can also land them in our git repository and not clobber them the next time we uplift. I’m also the module owner of toolkit, which is a pretty big sprawling set of files. It seems like everytime I look I find something that landed without me noticing. I don’t make for a good module owner if I’m not keeping an eye on things so I’d really like to see when new files are added there.
So I introduce the Hg Change Feed, the result of mostly just a few days of work. Every 10 minutes it pulls new changes from mozilla-central and mozilla-inbound. A mercurial hook looks over the changes and adds information about them to a MySQL database. Then a simple django app displays that information. As you browse through the directories in the tree it shows only changesets that affected files beneath that directory. For any directory you can also get an RSS feed of the same. Plug that into IFTTT and you have an automated system to notify you in pretty much any way you’d like about new changes you’d be interested in.
Some simple examples. For tracking changes to Add-on SDK I’m watching http://hgchanges.fractalbrew.com/mozilla-inbound/file/addon-sdk/source. For toolkit I’m looking at http://hgchanges.fractalbrew.com/mozilla-inbound/file/toolkit?types=added. Types takes a comma separated list of “added”, “removed” and “modified” to filter which changes you’re interested in. There’s no UI on the site for changing that right now, you’re welcome to add some!
One other neat trick that this does is mostly ignore merge changesets. Only if a merge actually makes a change not already present in either of the merge parents (mostly happens when resolving merge conflicts) will it show up in the list of changes, because really you don’t need to hear about changes twice.
So play with it, let me know if you find it useful or if you think things are missing. I can also add other mercurial repositories if people want. Some caveats:
- It only retains the last 2000 changesets from any repository in an effort to keep the DB size small and fast, it also only shows the last 200 changesets for each page, or just the last 20 in the feeds. These can be tweaked easily enough and I’ve done basically no benchmarking to say those are the right values.
- The site isn’t as fast as I’d like, particularly listing changes for the top level directory takes nearly 5 seconds. I’ve thrown some basic caching in place to help alleviate that for now. I bet someone who has more MySQL and django experience than me could tell me what I’m doing wrong.
- I’m off on vacation tomorrow so I guess I’m announcing this then running away, sorry if that means it takes me a while to respond to comments.