blogs > andrew > software > version-control

Thu, 15 May 2008

A Bazaar branch of GTK

Converting a project from one version control system to another is always painful. Doing so is a rather momentous change to contemplate. With organizational & community inertia being what it is, not something easily rushed into. Worse is when you attempt to use a modern tool as a better front end for an old one.

One of the neat things about the 3rd generation distributed version control system projects (ie Bazaar, Git, or Mercurial) is their plugins which allow you to continue to work with a Subversion upstream project but use the modern tool as a front end locally. For those of us who have been very productively using DVCS for some time, this is a godsend.

Using one of these plugins to thence evaluate the usability or performance of one of a DVCS is not really the best approach one could take; none of them were built with using Subversion as a peer in mind, and lord knows Subversion is not built to store revisions with such complex relationships. Thus the idea has turned out to be a very challenging problem space for everyone but most have had their tools mature to the point where Subversion is the bottleneck. And despite my misgiving that starting out your experience with a 3rd generation DVCS by first using it to access some legacy project will not show your new shiny tool in its best light, the reality remains that most people need to get on with being productive, and doing a full blown conversion (or starting use with a brand new projects) are less likely to be your circumstance.

Having been using Bazaar (known best by the abbreviation bzr) for well over 18 months and overall being very happy with the experience, I would like to encourage other GNOME hackers to try it; likewise I have some patches I’d like to contribute to GTK so I thought I’d start by using bzr-svn to get a branch of GTK as a starting point (and to likewise give Bazaar’s Subversion interface a workout). So I created Bazaar branch of GTK.

Initial import was very slow

The first step was to create a Bazaar branch that represents the foreign Subversion repository. This was painful.

I originally attempted to do this while I was at the GTK Hackfest we had in Berlin; that turned out to be a disaster because (as one does at a conference or meeting) we were moving around every few hours connecting and disconnecting from the ‘net, and I was busy trying to do this from my laptop rather than a server somewhere. (Unfortunately bzr-svn relies on a memory fix that is not yet in the released version of Subversion, and not every distro has their Subversion patched with it). So that fell flat on its face.

I was able to restart the process a week or two later and leave it running overnight. Along the way I’d learned a technique for doing a few hundred revisions at a time (do bzr pull -r $i in a loop and increment the variable by a hundred or whatever each cycle) and so was able to cope with the need to disconnect periodically. Which is well, because it took 2 days to do the import.

While I was at first appalled by this, I really can’t blame Bazaar. The GTK codebase is, of course, rather large. First preserved commits were in 1997; there are over 15,000 revisions; a working tree (excluding VCS metadata) is 83 MB in size. The bigger problem however, is that Subversion is not fast at accessing historical data (the time taken to process revisions started at over 32 minutes per 100 but eventually dropped to about 9 minutes per 100 as it got to the most recent history; interesting).

Clearly, I could have picked an easier example for my experiments with bzr-svn, but it’s done now.

Usage is fine once branch created

I wasn’t entirely sure whether the DVCS property that branches are all peers would apply to one that had started life backed by a Subversion repository would hold, but things work just as they are supposed to. Creating a new branch to work on a feature with bzr branch, or use the technique of using a bzr checkout along with bzr switch to do change-in-place, both worked fine. So I’m all set.

The real point, though, was to encourage more GNOME hackers to give Bazaar a try. If everyone had to put up with a endless import process like mentioned above then no one would touch it. But now that the branch is created (and up to date as of yesterday or so), anyone can use it.

With Olav’s concurrence, I put my branch on in my GNOME directory, so you can branch from it as follows:

$ bzr init-repo --rich-root-pack gtk+
$ cd gtk+/
$ bzr branch http://www.gnome.org/~afcowie/bzr/gtk+/trunk/

You do the first step so that you can create a “shared repository” (in Bazaar parlance) which will allow that various branches under that directory will share all the revision data for the project. I need to warn you, though. Modern DVCS tools give you the full history of the project, but for a large project that can be pretty big. If you grab the above branch you’ll be downloading about 180 MB and it takes about 5 minutes. Yes 5 minutes is a long time, but no there’s no way around it* and in any case 180 MB is not unreasonable for such a mature project. Don’t be too worried if it seems like it is taking a bit to do the transfer. You only need to do it once, and look on the bright side. It took me 2 days. Just let it run.

Workin’

Personally, I always do my work in a working branch separate from my mirror of upstream, allowing me to easily compare against the last update of upstream that I have done. So:

$ bzr branch trunk working
$ cd working/
$ ./autogen.sh
$ make

and away we go. Branching takes like 3 seconds. Nice and fast, no problem.

I haven’t got a script running religiously updating my GTK branch; I do a bzr pull locally once every few days in my copy of 'trunk' and have been pushing that up to http://www.gnome.org/~afcowie/gtk+/trunk/. But now that you’ve got the branch, you don’t have to rely on me anymore; this whole distributed thing kicks in and you can do it yourself. Assuming you have a svn.gnome.org account, then you can update with:

$ cd ../trunk/
$ bzr pull --remember svn+ssh://username@svn.gnome.org/svn/gtk+/trunk

(use the --remember argument the first time to change where it updates from by default so you’re not relying on my branch anymore) and push with:

$ bzr push svn+ssh://username@svn.gnome.org/svn/gtk+/trunk

The very first time you update from Subversion bzr-svn will have to build some caches and whatnot locally, but it’ll be pretty fast from there on.

I’m ignoring the pushing and pulling of revisions between your mirror of ‘trunk’ and your local 'working' (or whatever) branch; I’m sure you can figure that part out yourself: but here’s a hint: do your work, test it, and commit it, all in 'working'; repeat; then use, say:

$ bzr diff -r ancestor:../trunk/

to consider what your current patch looks like, then shuttle it to 'trunk' with bzr pull or bzr merge (assuming you run it from 'trunk’ and whether you need to merge or not) and then bzr push to svn.gnome.org. Hope it works for you.

If you need help, poke your head into #bzr on FreeNode or write to their mailing list, or ping me and I’ll try and lend a hand. Make sure you’re using the latest releases; I doubt it will work otherwise and so recommend that you use bzr >= 1.4.0 and bzr-svn >= 0.4.10. Older versions do not contain critical bug fixes. I also encourage you to grab bzr-gtk >= 0.93.0 as the visualize command it adds:

$ bzr viz

is really amazing.

Good luck!

AfC

Update:
You need to have PycURL installed (it’s an optional dependency; used at runtime if detected). There is a bug that will prevent you doing the branch from me if you don’t. Debian and Ubuntu package python-pycurl; Gentoo USE=curl pulls in package dev-python/pycurl; Fedora package is also python-pycurl.

This very moment as the GNOME systems administration team is presently working through the consequences of the Debian SSH key vulnerability; they have for the moment very properly locked down all access to the services provided to GNOME hackers, and that knocks out access to the Subversion server meaning people can’t work with their source code properly. If there was ever a clearer demonstration on the inadequacy of old 1st generation centralized version control systems, I cannot think of it.

bzr-branch-of-gtk, posted by Andrew Cowie in /andrew/software/version-control at 05:35, 15 May 2008.

Wed, 26 Sep 2007

Bazaar 0.91 released!

Congratulations to the Bazaar team on the release of 0.91!

Stop using dumb protocols

Probably long overdue, but we set a bzr serve daemon running on our R&D site.

Hackers who had write access to create their own branches there used the bzr server automatically since they `bzr push bzr+ssh://... and can also use it for branching and pulling. Most other people, however, were doing an initial checkout via http:// because that’s the URL we published. Which wasn’t very nice of us: initial checkout time for java-gnome from using the dumb http:// protocol used to be 20 minutes (serious round-trip penalty when from Australia to Europe. Ouch).

Switching to bzr:// dropped the time to clone a branch to 1 minute 46 seconds… Sweet! And that’s even before the monster revision streaming and repository format performance improvements that are likely to land in the 0.92 - 0.93 time frame! Excuse me while I rush off to update our README and HACKING files :).

Yes, dumb protocol time was abysmal — But http is a convenience for when all else fails, and has the nice side effect of allowing people to surf your public branch’s code. The problem is that people don’t necessarily get around to putting the server in place (most people could if they tried).

AfC

bzr-use-smart-server, posted by Andrew Cowie in /andrew/software/version-control at 07:43, 26 Sep 2007.

Thu, 20 Sep 2007

Incremental performance improvement

In passing in #bzr this afternoon, Robert Collins mentioned that he had dropped commit time for the example case he’s working on from 29.5 seconds to 18.8 seconds. Nice!

I asked him if this was towards improving the time taken to import new (large) projects into Bazaar? He said yes, but pointed out that some of this particular bit of optimization also impacts normal incremental commits and merges elsewhere.

It’s nice to see smart people working on things that benefit in not just one but many places. I wondered, though, how to quantify the sort of improvement that Robert is working on. The obvious target is the time taken to do the task being optimized. The broader question of knock on effects would rely on a broader performance measurement suite. It’s not a simple matter of “just make it run faster” — you have to be concerned that fixing one thing doesn’t make other important cases perform poorly as a result.

That reminded me that this is one of the things that db4objects is really good at. They continuously use their “PolePosition” benchmark system to evaluate the impact of changes and [hoped for] optimizations across a wide range of scenarios (read performance, write performance, writing in highly contended environments, etc) as evidenced by Carl Rosenberger’s recent work on improving transactional performance when db4o is being used in an embedded (single VM) environment.

Obviously, the cost of premature optimization is lost developer time spent working on things that don’t matter (if you’re lucky) and this risk of wrong architectural choices (which have much greater consequence). But design choices are unavoidable; you face them every day and then have to live with the consequences.

This isn’t “bad”; you make such decisions for good reasons. Some of the projects we’re doing for clients at the moment involves assessing the complexity of different architectural options and what impact those choices have on developer effectiveness, future maintainability, present performance and how hard improving it will be down the road. It’s interesting work.

This is one of the things I really respect about the team of people hacking on bzr. They took the time to map out a really sound architecture, and have worked hard on developing really robust algorithms capable of dealing with the complex corner cases that arise when doing merges and working with repositories with wildly different performance characteristics (branching locally versus branching from a repository half way around the world; are you using the dumb http:// protocol, or are you streaming at wire speed using the bzr:// protocol, etc). They took some heat early on because the consequence of their focus on architecture and robustness meant that some operations were rather slow, but they have been improving by leaps and bounds as they turn their attention from internals to performance. It works, it works correctly, and increasingly, it works fast.

AfC

bzr-initial-commit-time, posted by Andrew Cowie in /andrew/software/version-control at 07:05, 20 Sep 2007.

Thu, 13 Sep 2007

Git depends on RCS

Somehow, that says it all.

Apparently Git depended on the RCS package to get at certain merge functions. This is ironic, because RCS is a primitive 1st generation version control tool (sort of) that predates CVS.

AfC

Update:

• Dave O’Neill just pointed out to me that this dependency was actually removed in January ‘06. I looked around further and discovered that it’s the Debian “etch” package of Git 1.4.x that depends on RCS. This has been superceeded in “lenny” which presents 1.5.x.

git-uses-rcs, posted by Andrew Cowie in /andrew/software/version-control at 02:26, 13 Sep 2007.

Tue, 14 Aug 2007

Creating orthogonal patches

All open source projects encourage people to submit patches, but in order to review them you want them to be orthogonal, that is, nice and clean, carrying just the changes relating to one bugfix or enhancement.

The old way of doing things is that I, being someone with commit access to the upstream project, take the bits that I am willing to accept and commit them. But if I commit a patch for someone, it looks like I did the work. No. We don’t have to live with that inequity any longer. One of the hallmarks of the modern distributed revision control systems is that the submitter’s name shows up in the revision history and they get full credit and ownership of their contribution, so I am very adamant that contributors make their own commits. They they bundle up the revision or revisions that make up the contribution they want to make, and send it along.

So in addition to being clean and isolated we also want a contribution to actually be a revision carrying the metadata that makes up a commit — but not depend on unrelated revisions.

It’s this last criteria that is tricky. All of the people working on our projects are proceeding merrily on their way creating numerous or infrequent revisions as is their taste. The problem is that an individual branch history is, in essence, linear. From a sequence A,B,C,D,E someone who wants to contribute can’t just easily pull C out and send C to the upstream project because C depends on A,B.

This is one definition of the cherry picking problem, that is, pulling out only the change set you want and not everything else.

The only tool that ever got this right was Darcs, and it has a magnificent user interface for doing so. Unfortunately Darcs doesn’t scale — add too many files and you’ll hit the red giant bug which does tend to put a crimp into most development styles. This is not the only definition of cherry picking, mind you, but in general the “manually pick one revision out” thing has been tricky. Since the developers of the version control tool of choice know and highly respect Darcs, I took an off hand statement along the lines of “no, no Darcs-like UI to compose changesets yet” to mean that the cherry picking of revisions wasn’t there either. It turns out that this wasn’t quite correct.

So how to create orthogonal patches?

This has had me stuck for a while. I’ve been trying to figure out how to help new participants create clean orthogonal patch bundles, but getting those contributions up to scratch has been somewhat trying.

The obvious quick answer to creating orthogonal patches was:

“Individual micro branches for everything”

which at first makes sense; after all, in the 3rd generation distributed version control tools, branches are cheap. So every time you want to work on a new feature, just create a new branch:

$ bzr clone mainline fix-the-broken-juicer
$ cd fix-the-broken-juicer/

then do the necessary work, make a commit, and finishing by creating a patch for submission:

$ bzr commit
$ bzr bundle ../mainline > fix-the-broken-juicer.patch

resulting in a nice clean orthogonal bundle containing just the revisions needed for that fix which you can happily email off to the maintainer.

But this isn’t really practical, right? The whole point is that we get on with stuff. Nobody thinks in so compartmentalized a way. By the time you even think of fixing this bug you’ve already added two new features, refactored a bunch of stuff, and improved the build system. You’ve made lots of commits already, and you depend on that build system fix to even get the project to compile! But you don’t want to send all that stuff in, not yet anyway. You’re still working on some of it. And even if it was ready to go, it’d be a monster patch highly likely to have something wrong with it resulting it its rejection.

So how to submit code?

On the maintainer’s side, I don’t want to lower our standards, but if I reject their contributions too many times they’ll lose their enthusiasm and leave the project. Sure, lots of give and take, usually resulting in me having to clean up their patches in the end. Initially I was ok with that; it’s a reasonable effort to helping someone get the idea of what was required (and is itself collaboration). But there is often a time lapse between them creating a bit of code and their submitting it, my reviewing it, and then getting back to them with comments, and in the intervening time they’ve no doubt raced ahead adding more revisions, making it all the harder for them to rewind and try to keep everything straight in perfect little micro branches.

The solution turns out to be:

“You can cherry pick after all”

One new contributor, Nat Pryce, wrote in saying:

… the only way I can think of to avoid this is to manually copy the files from my branch into a mainline checkout, commit, generate a bundle, and then discard the mainline checkout again.

I agreed with him that this sounded horrible, and was thinking that this wouldn’t have the desired effect of “transporting the revisions” we want to convey.

It would seem, however, that we’re all missing the point.

The object of the exercise is to get code into the repository. What the revision graph looks like is irrelevant. That part I more or less knew already. But the more important parts are:

• it doesn’t matter which revision gets a bit of code into mainline so long as it has the contributor’s name on it; and

• if multiple revisions have the same code fragment, THAT’S OK.

When the two branches containing these various revisions eventually merge, they will be MERGED and (oops, smack myself in the head) they will not conflict because (duh) they’re they same code!

In other words, where we end up is that we need to create revisions X and perhaps Z that convey code to the project maintainer out of a stream of revisions A,B,C,D,E,F.

If X = B [in the java equals() sense], and if Z = D..E, it’s no problem, because when the day comes that X and Z are merged to some branch which happens to be the public mainline, and you later pull from that public branch, then X and B will “contend” and merge without conflict, and likewise Z will merge without conflict. It’s all the same text. Mission accomplished.

Aside: it turns out there is a good reason to merge X and Z now into your local working branch (ie, what will be) A,B,C,D,E,F,X,Z,G,H,I,… rather than waiting for it to turn up via the mainline branch revision stream: you’ll be giving the merge resolution algorithms a hand. Beyond the text not conflicting you’ll now already have the revision locally which will save you from pulling it over the wire later, which is actually back to what we started with in terms of trying to ship revisions around. On the other hand if you had already diverged and there was a conflict, then you’ll already have had to resolve it locally, which is good. And if upstream changed something in that code, say X,X’ and Z,Z’,Z” the merge delta to be computed when you go to merge them in is small. If the result is a conflict, well that’s ok — indeed, resolving those cases are ultimately what it’s all about, but in general they won’t conflict and we’re done.

Support from Bazaar

So back to or quandary, that manually copying files in

create branch, manually copy file, commit, create bundle & send

would be ugly.

It turns out that Bazaar supports this with a slight twist.

You are still going to have to create a [momentary] branch, make a commit, and bundle that off. But using the capabilities built into bzr’s merge command, we can quickly create branches containing just what we want to submit.

First create a pristine branch of a recent upstream:

    $ ls
    mainline/ working/
    $ bzr clone mainline plumbing-fix
    $ cd plumbing-fix

Now either a) select a few specific revisions, perhaps:

    $ bzr merge -r 292..294 ../working
    $ bzr diff

or more interestingly, b) select specific files or directories, for example:

    $ bzr merge ../working/src/bindings/org/gnome/gtk/Plumbing.java
    $ bzr merge --force ../working/src/bindings/some/other/file.txt
    $ bzr diff

No matter that you might have made 14 commits to get there. When you create this branch with its single additional revision, you can make a proper commit message. And if there’s stuff from pending work that shouldn’t be in there, you can clean it up before you commit. But either way, using the merge command means that you do a merge, and any irreconcilable collisions that you’ve introduced relative to the branch you’re merging into will be dealt with as such.

Finally,

    $ bzr commit
    $ bzr bundle ../mainline > john-plumbing-fix.patch

Email the patch, and you’re done.

Optionally, you can of course:

    $ cd ..
    $ rm -r plumbing-fix/

although you may want to keep the branch around in case any further changes need to be made; should that be necessary you can then just commit the fix-ups in the existing branch. Or you can create a new patch from scratch. Or you can always just reapply your bundle and recreate the branch. Lots of choices.

Eventually your patch will be merged to mainline, and when subsequently the revision that carried it is pulled down from mainline to your local copy of it and you are merging mainline into your primary working branch, there won’t be a conflict because that code was in your working branch all along (unless you’ve further changed something, of course; but that’s why it is advantageous to merge your contributed branch|patch back into your working branch(es) early; then its not just the code text that was once common, but now there are recent revision(s) for common ancestors, and the merge engine will slurp it right up).

Conclusion

This dramatically improves matters, I think. Now I don’t have to feel so bad about rejecting patches that aren’t quite there yet. When I do so I know I’m not inflicting as much pain as I was worried I was.

Contributors meanwhile can create all the private revisions they ever wanted, and don’t have to forecast which are going to be publicly visible and which aren’t. So they can leave off worrying about a proper commit messages, etc, until they actually go to create and submit an orthogonal patch.

But best of all, it restores the flexibility for people to do whatever they want. “No one can tell you no”; you might not get your change into upstream, but no one can keep it out of your code. And that’s what software freedom is all about.

AfC

Thanks to Robert Collins for having reviewed and commented on this essay.

Notes:

1. The bzr merge --force filename one-at-a time thing is a bit cumbersome. I had a word with some of the Bazaar hackers about it, and they’re going to see what they can do. Don’t forget there’s also the bzr merge -r A..B form.
2. Actually, the whole process is still a bit cumbersome. Someday, perhaps, we’ll have a UI to automate much of the changeset selection and bundle creation. Until then, though, this works, and it accomplishes the aim, which is facilitating contribution. That’s important to keep in mind.
3. Most of this discussion is characteristic of the 2nd and 3rd generation distributed revision control tools as a class, although obviously command syntax will differ. The key to all this holding up at scale, however, is ultimately the quality of the merge algorithms in the tool you’re using and its ability to deal with corner cases.
4. At the end of it all, though, if you can create a new branch and just do the work there in an isolated fashion resulting in an orthogonal patch, that is easier. :) It depends on the nature of the project you’re working on, the size of the change you’re contemplating contributing, and they way you work. In our case, the insane refactoring power of an IDE like Eclipse tends to mean that we get a lot of unrelated changes in flight at the same time, thus exacerbating the original question of creating orthogonal patches.

Updates:

1. As of Bazaar 0.90, there will be a bzr send -r command which, when supplied with a revision, will create an orthogonal patch in a single step!

bzr-orthogonal-patches, posted by Andrew Cowie in /andrew/software/version-control at 10:22, 14 Aug 2007.

Fri, 20 Jul 2007

Version control that doesn’t make your eyes bleed

At the GUADEC closing, Federico remarked:

trying to learn git felt just like trying to learn cvs the first time 10 years ago felt — why is this so hard?

That was exactly how I felt about git for the 4-5 months I spent dogfooding it — despite my respect for the underlying notions in Git, I was constantly perplexed at why basic things seemed so complicated — and in the back of my mind was “I’ve been at this for a while; how on earth is someone new to Linux supposed to learn this?”

And then I tried bzr!

We chose bzr (Bazaar-NG, as it was briefly known) as the VCS for the new Java bindings for GNOME for Bazaar’s relative straight-forwardness and because of our faith in the ethos and extreme competence of its developers. Anyone experienced with the old world 1st generation centralized VCS tools like CVS or Subversion will be able to make sense of it and you can learn from there. Bazaar is constantly improving in performance terms, has a vibrant developer community, is widely portable, and most of all the fact that they actually follow test-driven-development practices (unit test suite has over 7266 tests) to keep them honest biases in their favour.

In production use for the last year, we have found Bazaar to be reliable, amazingly easy for newcomers to the Open Source world to learn, and a big contributor to our goal of reducing barriers to entry. One of my most treasured emails this year has been when someone wrote me saying:

Nice HACKING instructions, really easy to follow, even though I never used bzr before. Attached are 2 small fixes for the configure script… builds on OpenSolaris now.

Newcomer to contributed patch in less than an hour. Wow.

Some people have argued that they should wait to see which 3rd generation decentralized version control system will “win” before migrating away from svn or god forbid cvs. This is, in my view, foolish — none of Bazaar, Git or Mercurial are going to go away; the determining factor, rather, is usability — and that makes the choice an easy one.

Congratulations to the Bazaar team on the release of 0.18!

AfC

git-is-like-cvs, posted by Andrew Cowie in /andrew/software/version-control at 15:56, 20 Jul 2007.

Sun, 10 Dec 2006

Pushing a bzr branch with rsync

For some reason bzr rspush, a plugin from the “bzrtools” unstable command collection, doesn’t know how to push a branch that’s in a Bazaar repository.

Repository layouts

What’s a Bazaar repository? Good question. It’s not what you get if you do bzr init. You have to do bzr init-repo --trees . in a directory, then within that directory create and copy branches. Apparently this then allows the branches to share revision delta files in common.

Why oh why can’t they just be immutable like Git does and thus hard-linkable?

Anyway, I’ve ended up with a directory structure as follows. There’s a top level Bazaar repository here:

~/src/andrew/java-gnome

The bzr data actually lives in, as you might expect,

~/src/andrew/java-gnome/.bzr

But that repository is not a branch and not a working directory. In the repository directory, I create branches and clone them:

~/src/andrew/java-gnome/codegen
~/src/andrew/java-gnome/equivalence-fixes
~/src/andrew/java-gnome/mainline
~/src/andrew/java-gnome/primary
~/src/andrew/java-gnome/website

And so on. Each one also has a .bzr directory:

~/src/andrew/java-gnome/mainline/.bzr
~/src/andrew/java-gnome/website/.bzr

Apparently the deal is that if the actual revision data isn’t in the proximate branch’s .bzr, it’ll look up one to see if there’s a .bzr there (and finding itself in a repository, there is, and so the data it needs is in ../.bzr/repository).

Since I used the --trees option to bzr init-repository, I get working copies in each of those directories. I can then point a symlink to it from the place where my Eclipse IDE expects my working directory to be,

~/workspace/BindingsPrototype -> ~/src/andrew/java-gnome/primary

and finally I can get to work. [You go to a lot of effort to configure Eclipse to work on a given Java project in a specific place. Switching the basedir of that path from one place to another is a non-starter, except by doing a full refactoring. So, the symlink]

I’m liking bzr well enough, but doesn’t this strike you as a bit complicated? Granted git is a usability disaster zone, but the core concepts of repository storage and branching seem really rock solid. I really miss the create-a-branch-in-place and switch-between-branches-in-place aspects of Git. Those were killer features for me. Bazaar has a evolved in a few too many parallel directions, and they need a bit of good old GNOME 2.0 style refactoring to get out some of the cruft. Still, though, they have unit tests. That’s worth a lot in my book.

Pushing the wall

So now its time to publish your work. My experience trying to bzr push sftp:// to a server half way around the world was that the sftp implementation is brutally slow. Using their “Smart Server” with a bzr push bzr+ssh:// command was not much better.

Obviously this sort of thing is crying out for a little rsync lovin’, but unfortunately the bzr rspush command which supposedly does just that doesn’t know how to push a branch that’s in a Bazaar repository. Oh, great. So I had to do it manually.

I have a script that does a bunch of switching to figure what I want to send (source, website, photos, whatever) to where. For source code, I use it something like this:

upload source <project> <branch>

You get the idea. The trick is to figure out how to tell rsync to include the repository .bzr, the branch directory (and its `.bzr), and exclude everything else. I could have just done two separate SOURCEs, but I like to see all the traffic reported relative to a single root, so I use rsync’s include/exclude syntax instead. That stuff is notoriously voodoo; this time was no exception.

Here’s the magic smoke I used to get rsync to push a Bazaar branch within a Bazaar repository:

    PROJECT="$2"
    BRANCH="$3"

    # note trailing slash
    SOURCE="/home/andrew/src/andrew/$PROJECT/"
    DEST="andrew@centauri.lhr.operationaldynamics.com:/export/web/com/operationaldynamics/research/bzr/$PROJECT"

    # for some reason the escaping necessary eluded me; workaround:
    s='*'

    # now go to the trouble of including each directory in the chain
    # (otherwise exclude * nukes them) and the three things we care
    # about in <project>: /.bzr, /<branch>, and /.bzr

    OPTIONS="--include=/
        --include=/.bzr
        --include=/$BRANCH
        --include=/.bzr/$s$s
        --include=/$BRANCH/$s$s
        --exclude=/$BRANCH/tmp
        --exclude=/$BRANCH/.config
        --exclude=$s"

    shift 3

…

    exec nice rsync \
        --verbose \
        --recursive \
        --update \
        --links \
        --hard-links \
        --perms \
        --times \
        --sparse \
        -e /usr/bin/ssh \
        --partial \
        --progress \
        --compress \
        $OPTIONS \
        $* \
        $SOURCE $DEST

So with a nice simple:

upload source java-gnome mainline

I publish (only) the “mainline” branch in my ~/src/andrew/java-gnome repository.

Hopefully that will be of help to someone; I can’t be the only person who has found themselves wanting to publish a Bazaar branch from within a project repository. What would be great, though, would be a nice and simple bzr push ssh:// that just assumes rsync-over-ssh and does all this for you. Shouldn’t be too hard for someone to hack in.

AfC

bzr-repository-rsync, posted by Andrew Cowie in /andrew/software/version-control at 14:12, 10 Dec 2006.

Fri, 29 Jul 2005

Gotta hate those Red Giant bugs

In a recent thread on the darcs-user mailing list, this response from David Roundy (the original author of darcs) came along:

What you’ve run into is the infamous O(2^N) behavior of darcs when it encounters certain sorts of conflicts. The code should eventually complete, but it’s possible that our sun will become a red giant before that happens (which would most likely cause darcs to fail). :(

I have to admit that the red giant date is indeed a bit beyond my planning horizon. Of course, I can chuckle at this since I haven’t been burned by this bug, whatever it is :)

Darcs is a 2nd generation distributed version control tool. I’d been rather happy with it for a few months. It’s surprisingly easy to use, very powerful, has brilliantly straight forward cherry picking of patches, and seems to Just Work (tm) in all the use cases that I have for it. The way Darcs facilitates casual users to quickly create and submit patches is awesome.

AfC

Update: Spoke too soon; hit what appeared to be the Red Giant bug in Jan 2006. Decided not to wait around to find out and stopped using Darcs for new work. Switched to evaluating 3rd generation tools. Tried Git for six months. Hated it — its user interface is worse that GPG’s if you can believe such a thing is even possible. Tried Bazaar aka bzr. Great usability, Just Works on incredibly complex corner cases. Awesome developer and user community. Using it for all projects as of Nov 2006.

red-giant-bugs, posted by Andrew Cowie in /andrew/software/version-control at 02:58, 29 Jul 2005.

Category Specific Feeds. Use these links for an RSS or ATOM feed limited to this category and its descendants. Technorati Profile