I just want to thank you for this great API to GTK.

It feels entirely natural in Java, is very comfortable and very approachable. The names and APIs are as you’d think them up when you’d draw a widget set API in your mind, thus they are easy to find, too. They also work as expected (as least so far, I am still new).¹ The API documentation for each class has a nice prose introduction text, which is very helpful and saves most “tutorials”. I also appreciate that it points out just the right level of footangles to be aware of.

All in all a really great piece of work. When you talked about “quality” on your website, I was suspect, as that’s most of the time just words, but you really deliver quality. You show an good example of what quality means in the context of an API.

Sun should take a good example from you and many in the open-source world, too. Such brightness and care is rare.

Hey, thanks!

Quality versus Barrier to Entry

Regardless of the underlying economic circumstances, most Open Source projects are labours of love. But they can go one of two ways.

One tendency is to just accept anything, and hope that “someone” will fix it later. This approach has the beneift of a low barrier to entry, making it easy for people to contribute. But it can result in a muddle; code that is not yet up to scratch but whose authors have moved on to other things. The only time that Open Source fails to live up to its promise (as a software development methodology) is when someone starts something and doesn’t finish it. A situation nobody wants, but it is easy to have this inflicted upon your project if you accept incomplete contributions from other people — it can be hard to incent them to come back and finish the job.

The other way is to maintain a high standard, putting emphasis on internal consistency and clarity and asking people to rise to that level. While setting a high bar does put some people off, the contributions from those who invest the effort to rise to the occasion tend to be really good.

I tend to favour the second way. If your initiative is worth working on at all, then it is the kind of thing you work hard at making perfect because it deserves nothing less. You’re not going to let it down, and you don’t want anyone else to either.

Anyway, it’s all a lot of work. Sometimes I hate it. But when you receive a note like this from someone who appreciates the effort you’ve put in, it really does make your day — and makes you feel that result of staying clearly focused has been worth it.

AfC

¹ I pointed out to him in my reply that there was still plenty of time for him to trip over something :)

This blog post is an extract of the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.9 (13 Oct 2008)

The pen is mightier than the sword

New coverage

This is the first release with coverage of GTK’s powerful TextView/TextBuffer multi-line text display and editing Widget. This has been the result of several months of careful effort to present a clean and self-consistent API while remaining faithful to the underlying implementation. This bulk of this work was done by Stefan Prelle and Andrew Cowie, with contributions from Kenneth Prugh and testing by many people in the #java-gnome community.

The snapshot at right is from ExampleInstantMessenger, included with the sources. It is a somewhat detailed example showing the use of TextView, TextBuffer, and related classes.

Other improvements

Continuous improvement to various classes, especially in our documentation. Incremental changes have occurred in a number of places. In the TreeView/TreeModel APIs, some useful methods for translating TreeIters from one model to another have been added.

Also thanks to the persistent work of Stefan Prelle, we now have nice coverage of GTK’s Assistant (aka druid, wizard, etc), along with better support for doing popup context menus, including some bug fixes. Thanks to Srichand Pendyala for taking care of this and to Owen Taylor for having explained out some of the underlying implementation details.

As usual, incremental improvements to core classes continue. Virtually every class has been touched in one way or another; many changes are cosmetic but they add up to a nice delta overall.

Reducing memory pressure

Internally, java-gnome maintains a lookup table so that pointers coming from the C side can be converted into proxy objects for the case where a proxy has already been created. In any library there a great number of transient and temporary objects and structures allocated, and in wrapping native GNOME libraries we are no different. It turned out that registering these temporary objects was putting pressure on the lookup table. While these objects were properly weak referenced and being garbage collected (and thence freed), there were nevertheless an enormous number of temporary objects being inserted and removed from the lookup table — and that sort of thing causes hash tables to grow overly large.

To do something about this we have split the former hierarchy root into two classes. Only structures which have a persistent identity (which, in practise, means only GObjects and certain Cairo entities) are registered so they can be looked up by address later as necessary. The rest of the Java side proxies aren’t registered, essentially eliminating the transient pressure on the lookup table.

Thanks to Vreixo Formoso for doing the bulk of the leg-work on this one.

Making it easier to run java-gnome programs

Because java-gnome is directly binds to underlying system libraries, it has a native shared library component. This led to the usual development hassle of having to specify where this library is to be found if it were anywhere other than /usr and of course the nightmare of ensuring a VM used the right library in the event you were developing against or hacking on a newer version of java-gnome; in Java this meant:

$ java -classpath /opt/local/share/java/gtk-4.0.jar:. -Djava.library.path=/opt/local/lib com.example.Program

No longer!

The native shared library part of java-gnome is now located deterministically and loaded automatically. You don’t need to faff about with java.library.path on the command line or in your IDE any more!

$ java -classpath /opt/local/share/java/gtk-4.0.jar:. com.example.Program

Our native component is completely coupled to the specific release you are using, so sufficient version information is embedded in the .so name to ensure that the right library (and only the right library) is loaded.

There are no changes if you are simply working against an “in-place” development build of java-gnome, be it from command line, or in an IDE like Eclipse. Things will Just Work™. Again, no -Djava.library.path.

This whole issue turned out to be a real stumbling block for new developers attempting to use the bindings; it’s not something that many Java programmers have had to deal with, so engineering around this problem to make the native library loading transparent is a big win for us.

Build system improvements

Serkan Kaba has contributed a number of internal improvements allowing the top level ./configure script to be precise about the versions of various GNOME dependencies we require.

Thanks to some hard work from Serkan Kaba and new contributor George McLachlan, java-gnome correctly builds against GTK 2.14 without any problems due to deprecations.

Note that java-gnome releases do not set GTK_DISABLE_DEPRECATED (this is a change from 4.0.8); thanks to Mart Raudsepp of the Gentoo Linux desktop team for pointing out why this would be better. These macros are still enabled for builds checked out from version control so hackers working on the bindings so will be able to keep up with ensuring we react to future deprecations (it’s always awesome when downstream is a part of the upstream community; Serkan and Kenneth are also Gentoo packagers, and take care of the java-gnome .ebuild for us).

Looking ahead

We’re pretty happy with the state of the java-gnome right now. Coverage of the most important parts of GTK are in place. Our treatment of the underlying drawing library, Cairo, still has a bit to go, but the basics are there and a firm foundation to build from. More interesting are the remaining areas; the more general GNOME utility libraries and other parts of the Free Desktop stack that might be needed by an end-user application. It’ll be interesting to see how these areas evolve in the coming months.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr checkout bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

AfC

If you’re already in an error handler, then the default status code as far as the web server is concerned is, of course, HTTP 404. So you need to change that. In PHP, that’s as simple as:

<?
    header("HTTP/1.1 200 OK");
?>

which is actually treated a bit magically to tweak the response code. Too easy. This worked fine on our test site and on the production site for, well, years.

We ran into a weird problem that at some point over the last few weeks that suddenly I was seeing garbage appearing at the beginning and end of documents being so rendered on our SourceForge website. Huh?

I still have no idea what is wrong, and won’t detail all the diagnostics it took to track down the problem, but I ended up isolating it to the header() call. If I skipped that, then the document would render and serve correctly. But that’s no good, because then you are serving documents with an HTTP 404 error code, which will inhibit search indexing and caching even if it doesn’t screw with your browser.

The workaround ended up being this:

<?
    header("HTTP/1.0 200 OK");
?>

I have no idea why, but with whatever SourceForge has done to their webservers lately, this made it work fine. Go figure.

The real inanity, though, is that using wget to fetch the document while showing HTTP headers the result is this:

$ wget -x -S http://java-gnome.sourceforge.net/4.0/README.html
--2008-10-13 14:11:28--  http://java-gnome.sourceforge.net/4.0/README.html
Resolving java-gnome.sourceforge.net... 216.34.181.96
Connecting to java-gnome.sourceforge.net|216.34.181.96|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/0.6.31
  Date: Mon, 13 Oct 2008 03:11:29 GMT
  Content-Type: text/html; charset=UTF-8
  Connection: close
  X-Powered-By: PHP/5.2.6
  Last-Modified: Sun, 12 Oct 2008 14:02:37 GMT
Length: unspecified [text/html]
Saving to: `java-gnome.sourceforge.net/4.0/README.html'

    [    <=>                                ] 16,052      22.3K/s   in 0.7s    

2008-10-13 14:11:30 (22.3 KB/s) - `java-gnome.sourceforge.net/4.0/README.html' saved [16052]
$

Their web server sent an HTTP/1.1 response anyway! Talk about adding insult to injury. What a waste of time.

I’m sure this is just one of those transient conditions that’ll be gone by this time next week, but it sure as hell frustrated me. If you’re seeing something weird going on with PHP pages on one of your SourceForge sites, perhaps have a look and see if this workaround helps.

{shrug}

AfC

There’s one (and only one) legacy Windows application that we have. We used to run it in VMware, which worked, but that was a pain, looked terrible, and of course was required us to have our copy of Windows 2000 installed there. Decrepit. But far worse, meant we had to use Windows. Yuk. And worst of all to have to keep switching in and out of it. Bah.

Life would be far better if we could run it on Linux under Wine — then it’d just be another app on running on the GNOME desktop and under the control of the window manager. We’d tried seeing if we could get this application to run in Wine a few times over the years; no joy. But somewhere along the line it started noticing list posts from other people reporting positive experiences. So {shrug} I gave it another try this month, and to my intense pleasure it installed cleanly and is running fine. Terrific! The people who hack on Wine are amazing.

So I started working away, only to suddenly realize that the dates were in American mm/dd/yy format. Gah. More searching, but none of the workarounds suggested (mostly to do with manually changing things with Wine’s regedit) seemed to work. Quote a pain.

I had come across a number of references saying that Wine and in turn the program in question would pick up the locale settings provided by your environment. I’d checked that, and tried various combinations. Nothing seemed to work.

I mostly run in en_CA, ie:

$ echo $LANG
en_CA.UTF-8
$

the impact of which you can see by running the locale command:

$ locale
LANG=en_CA.UTF-8
LC_CTYPE="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME=en_DK.UTF-8
LC_COLLATE="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_PAPER=en_AU.UTF-8
LC_NAME="en_CA.UTF-8"
LC_ADDRESS="en_CA.UTF-8"
LC_TELEPHONE="en_CA.UTF-8"
LC_MEASUREMENT="en_CA.UTF-8"
LC_IDENTIFICATION="en_CA.UTF-8"
LC_ALL=
$

all as one would expect And yet it had no impact on running the program running under Wine [Setting LC_TIME to en_DK is an old trick for people in en locales to get proper 24-hour time formatting; yes, I tried unsetting LC_TIME; I tried dropping the UTF-8 settings; I tried LANG=en_AU. Nothing made any difference].

After a lot more frustration searching around, I finally came across a support article on CodeWeaver’s website that mentioned setting the locale environment variables. Yes yes, I thought, but then I looked more closely. They were setting LC_ALL. Wait a minute. LC_ALL is a special variable that “if set to a non-empty string value, override the values of all the other internationalization variables.” You’re not supposed to set that… certainly one doesn’t set that at login — that’s what LANG is for. I didn’t really expect anything to come of it, but I tried it anyway:

$ env LC_ALL=en_CA.UTF-8 WINEPREFIX="/home/andrew/.wine" wine "C:\Premier12\Myobp.exe"

and ta-da everything worked: I had dd/mm/yy date formatting just like we wanted. Yeay!

I immediately closed everything down and went out for a beer. Sometimes we forget to celebrate those brief moments when things actually work.

But talk about bitterness. I’m not sure if this was a Wine behaviour, a Windows behaviour, or just nonsense code in MYOB. I suspect the latter. But either way, I saw so many posts asking “how can I get the date format to behave under Wine” that I thought I should write about the workaround that did it for us.

AfC

I was pairing with a colleague the other day and they were impressed with how I had things set up — and more to the point my rationales for having done so — and they encouraged me to write about it.

I don’t particularly think I have anything original or novel to say, but in so far as some of us have built up configurations over the years that (for ourselves, at least) improve usability and which define our productivity, I think it’ll be a nice contribution to show others what we’re using and how we got things that way. If you think you know what you’re doing, then I encourage you to write about your customizations — and why you made them. People using commercial distros like Ubuntu and Fedora will justly be able to say “it already looks good” and so if you’re happy you can ignore this, but even in such environments we still often tighten things up to individual taste and so I’m sure you have something to offer.

Anyway, thus begins an occasional series about things I’ve done to beautify my GNOME Desktop on our Gentoo Linux systems.

A few years back I was enjoying a drink with Carl Worth and Keith Packard (free desktop graphics hackers extraordinaire, for those who haven’t met them). Later in the evening Keith leaned over, saw my laptop’s screen, and starting swearing at me: “You need to fix your fonts!” (I was using Luxi or something). So, ok, what do you want me to do? They told me to switch to the Bitstream Vera family as being well sculpted, highly professional, and optimized for screen display. Needless to say I take what these gentlemen say extremely seriously, so did what I was told :).

Like any font or appearances change, for the first day or so it felt a bit odd, but then something shifted and it felt very comfortable. The real give-away that I was on the right track was a few days later when I tried switching back to the old font for the hell of it and recoiled in horror “Gawd that’s ugly!”

Bitstream Vera had done me well for quite some time, but recently I started running into some subtle bugs. One showed up with the weather applet; for whatever reason a few versions ago they changed the string from ‘°’ + ‘C’ to unicode character 0x2103, the symbol for degrees Celsius. I was getting a weird fall back to some horrible looking old bit-mapped font. Yikes (if °C and ℃ don’t look about the same then you’ll see what I mean). No big deal, but it was annoying me.

So I was checking around again to see if there was a different font family I should be using, when I came across the DejaVu. It starts from Bitstream Vera (so the base ASCII characters are all the same and mostly untouched) but adds a ton of hinting corrections, bug fixes, and extensions up in the higher unicode sequences needed by European languages and scientific work. There are also a number of additional faces available. And regular releases (always a good sign). And a different name, per the Bitstream licence terms. When I asked about DejaVu in #gnome-hackers, the answer was “hell yes you should be using it.” Huh. So I switched!

The DejaVu fonts are available on Gentoo in the media-fonts/dejavu package.

The “right” way to change the font globally would be to change the “Sans” and “Serif” and “Monospace” aliases in the fontconfig setup in /etc/fonts. But I generally try to leave those files alone, so instead ran gnome-appearance-properties and from the Font tab changed the system fonts to be what I wanted.

Immediately it fixed my glitch with the weather applet (yeay), but I started to notice anything using the higher order mathematical symbols was likewise drastically improved. Hurrah!

With gnome-font-viewer,

Making screenshots of what they look like on my screen is a bit silly because they won’t necessarily look the same on your screen. But hey. We do our best.

Open Source is about choice, and so I cannot talk about fonts without also mentioning the Liberation font family. These were obtained by Red Hat and intended as a drop-in replacement for the Microsoft core fonts (Courier New, Arial, etc) which are widely used but not libre. For whatever reason the Liberation fonts don’t look quite right on my screen (there are red and green hinting glitches around the glyphs; something is obviously misconfigured somewhere) but it’s a different free font that has been generously made available to the community. They’re available on Gentoo in the media-fonts/liberation-fonts-ttf package.

AfC

Update:

1. So it turns out that DejaVu is set to be the default on Gentoo, but they’ve got a bug whereby they’ve got their ordering wrong: if you also have Bitstream Vera installed it gets picked up for Sans in preference. The folks in #gentoo-desktop say they’re going to fix that. (This is only relevant if you have the x11-base/xorg-x11 meta package installed; it pulls in media-fonts/ttf-bitstream-vera. If you only have the lower level x11-base/xorg-server package installed you can manually install media-fonts/dejavu [only] and then the default aliases will be to DejaVu). Anyway, the real point of this article was about being aware that you can have control of your fonts, so all good.

This blog post is an extract of the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.8 (15 Aug 2008)

Cleanups and fixups.

This release is mostly to push out bug fixes and internal improvements, setting the stage for some major new feature development. We’ve also taken the opportunity to introduce a major change to the way we connect handlers for signals.

New coverage and continuing improvement

With thanks to new contributors Stefan Prelle and Andreas Kuehntopf we have a number of small improvements to the TreeView/TreeModel APIs. As always, Widget and Window saw a bunch of work, with Window.ConfigureEvent now being available and a number of additional property setters and methods relating to window type. Widgets that scroll around a view of a broader underlying canvas have seen a fair bit of activity related to controlling that scrolling.

New features include refinements and new coverage of methods in a variety of lower level classes including that further support drawing operations. Bug fixes, debugging improvements, and defencive enhancements to our thread safety measures have also featured largely.

Signal naming change

The names of the inner interfaces used to specify the prototypes of the methods which receive signal callbacks have changed to the pattern Button.Clicked, this being more appropriate to Java type naming conventions and providing better consistency between the signal name, the method to be implemented to receive the callback, and the method that can be used to emit this signal.

API compatibility to previous releases in the 4.0 series has been preserved. The old signal interfaces and connect() methods are @deprecated. Developers are encouraged to migrate quickly; new coverage will of course all follow the new pattern. Making the transition is is easy, especially in an IDE. Most of the people we’re aware of using the library have already ported their code. Just turn assertions on to double check.

Build changes

java-gnome now defines C compiler flags like GTK_DISABLE_DEPRECATED to ensure we are not linking against code that will be unavailable in GTK 3.0. Many thanks are due to new contributor Kenneth Prugh for having done some terrific grunt work to prune deprecated classes and methods from our .defs data so that java-gnome compiles without using these APIs.

The build system internally now ensures that multiple runs don’t occur simultaneously, fixing a number of annoyances that cropped up when using IDEs which tend to like trying to frequently re-run the build even if a previous one hasn’t finished.

Documentation, examples, and testing

Our API documentation and the growing set of example code have all been updated to reflect the new signal interface names. Doing so forced us to review a wide swath of the documentation, and so along the way a huge number of minor improvements were made. Given how detailed our JavaDoc is, this sort of painstaking work really makes a genuine contribution to overall quality.

There has been steady growth in our test suite, which is great. When combined with the snapshots used to illustrate our documentation, the coverage level is substantial.

Error handling continues to improve. In the (hand written) public API wrapper layer we do our best to catch misuses of the library before they get sent to the native code. But that’s not always possible, and in 4.0.7 we introduced a mechanism whereby GLib error messages get translated into Java Exceptions and thrown. As of 4.0.8, in addition to ERROR and CRITICAL, we also throw WARNINGs as Exceptions. Getting a stack trace this way has proved very useful in helping developers track down where they are making mistakes in using the library.

Conclusion

You can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.7..tag:v4.0.8

Because of the API changes to signal handling this release touches just about every public class in the library and so isn’t quite as clean (as a summary) as in previous releases — but it does show you everything that changed. :)

Looking ahead

Most of the contributors to java-gnome are working on branches that didn’t reach sufficient maturity to be merged in time for 4.0.8; that’s the way it goes sometimes. Major effort continues on implementing coverage of GTK’s powerful TextView/TextBuffer APIs, along with further drawing capabilities in Cairo and Pango. There have also been a surprising level of interest on other areas of the GNOME stack, with new contributors working on adding support to java-gnome for Nautilus, GStreamer, and even WebKit. Exciting stuff!

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr checkout bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

AfC

Cody only got involved in this issue recently, but he took the time to really dig into it, examine different hypothesises, and test like crazy. This was remarkable: it’s not every day you see someone wade into a bug that’s been open for over half a decade and with over a hundred comments, and then persist for over a year to get it solved.

Way to go!

AfC

An Eclipse project has several metadata files:

.project
.classpath
.settings/org.eclipse.cdt.core.prefs
.settings/org.eclipse.jdt.core.prefs
.settings/org.eclipse.jdt.ui.prefs

Which seems a bit of a mess, but whatever. The .settings/ stuff gets written down when you decide to tell a Project to have settings custom that override the user’s defaults. It goes without saying that for a collaboratively developed project like the Java bindings we have the Eclipse project metadata for compiler warnings tuned, the code formatter settings, how to invoke the build system etc all stored in Bazaar along side the real sources.

Anyway, this’s all fine and nice for people working on my software in Eclipse, but what about people wishing to use a different editor?

For a few versions now, Eclipse has exposed the the ability to run its code formatter from the command line; the work was done by Ben Konrath, someone I met a few years ago at a java-gnome hackfest. Since I use Eclipse I hadn’t needed to bother to figure out how to use whatever it was that he’d done. Since becoming maintainer of java-gnome, though, I’ve noticed a bit of unnecessary formatting churn in the contributions we receive. We do have a style guide, but it is secondary to the fact that Eclipse just normalizes the code every time I save something (nice). But for people not using Eclipse I thought I might have a look.

A bit of searching around turned up this blog post by one Peter Friese. It included two important points that are non-trivial. Firstly, that configuration the code formatter from command line will use can be controlled by the pointing it at the .settings/org.eclipse.jdt.ui.prefs file which in turn can be created by the “Enable project specific settings” described above (we’d already done that, so no problem, but how on earth would anyone have guessed this?). Secondly, the command line that one has to incant to make it go is pretty voodoo, so I’m pretty thankful that he described what he’d done.

In our environment, this works out to:

$ /opt/local/eclipse/eclipse -nosplash
    -application org.eclipse.jdt.core.JavaCodeFormatter
    -verbose
    -config .settings/org.eclipse.jdt.core.prefs
    src/ tests/ doc/examples/

Gah!

For a brief moment I wondered if it would be easier just to run the Java class in question. So I interrupted the process and had a look in the process table.

/usr/lib/jvm/sun-jdk-1.6/bin/java
    -Dosgi.requiredJavaVersion=1.5
    -client
    -Xms40m
    -Xmx256m
    -jar /opt/local/eclipse/plugins org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
    -os linux
    -ws gtk
    -arch x86
    -launcher /opt/local/eclipse/eclipse
    -name Eclipse
    --launcher.library /opt/local/eclipse/plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.0.2.R331_v20071019/eclipse_1021.so
    -startup /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
    -exitdata 84001b
    -application org.eclipse.jdt.core.JavaCodeFormatter
    -verbose
    -config .settings/org.eclipse.jdt.core.prefs
    src/ tests/ doc/examples/
    -vm /usr/bin/java
    -vmargs
    -Dosgi.requiredJavaVersion=1.5
    -client
    -Xms40m
    -Xmx256m
    -jar /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar

Good Lord. What the hell is all that? I mean, I know Java environments tend to get a bit unwieldy because there is still no link-to-binary facility, and I know Eclipse is an OSGi framework and what not, but that is pathetic. I guess running the command as described at the top isn’t such a bad idea after all.

I’ve got an even better idea: how about a convenience target in our Makefile front end. Yup. So I’ve added a format target:

$ make format

(or

$ ECLIPSE=/opt/local/eclipse/eclipse make format

in my case). Running that before committing will do the trick quite nicely.

I agree with the bug mentioned by one Nicholas Irving in his post on the topic: it is somewhat asinine that this bit of Eclipse bumps the modification time of every file it formats; it would be nice if it only changed files in which it actually found something it needed to alter. Fortunately our internal build system is actually md5 hash (not Make) based, so it doesn’t hurt us.

AfC

One of the neat things about the 3rd generation distributed version control system projects (ie Bazaar, Git, or Mercurial) is their plugins which allow you to continue to work with a Subversion upstream project but use the modern tool as a front end locally. For those of us who have been very productively using DVCS for some time, this is a godsend.

Using one of these plugins to thence evaluate the usability or performance of one of a DVCS is not really the best approach one could take; none of them were built with using Subversion as a peer in mind, and lord knows Subversion is not built to store revisions with such complex relationships. Thus the idea has turned out to be a very challenging problem space for everyone but most have had their tools mature to the point where Subversion is the bottleneck. And despite my misgiving that starting out your experience with a 3rd generation DVCS by first using it to access some legacy project will not show your new shiny tool in its best light, the reality remains that most people need to get on with being productive, and doing a full blown conversion (or starting use with a brand new projects) are less likely to be your circumstance.

Having been using Bazaar (known best by the abbreviation bzr) for well over 18 months and overall being very happy with the experience, I would like to encourage other GNOME hackers to try it; likewise I have some patches I’d like to contribute to GTK so I thought I’d start by using bzr-svn to get a branch of GTK as a starting point (and to likewise give Bazaar’s Subversion interface a workout). So I created Bazaar branch of GTK.

Initial import was very slow

The first step was to create a Bazaar branch that represents the foreign Subversion repository. This was painful.

I originally attempted to do this while I was at the GTK Hackfest we had in Berlin; that turned out to be a disaster because (as one does at a conference or meeting) we were moving around every few hours connecting and disconnecting from the ‘net, and I was busy trying to do this from my laptop rather than a server somewhere. (Unfortunately bzr-svn relies on a memory fix that is not yet in the released version of Subversion, and not every distro has their Subversion patched with it). So that fell flat on its face.

I was able to restart the process a week or two later and leave it running overnight. Along the way I’d learned a technique for doing a few hundred revisions at a time (do bzr pull -r $i in a loop and increment the variable by a hundred or whatever each cycle) and so was able to cope with the need to disconnect periodically. Which is well, because it took 2 days to do the import.

While I was at first appalled by this, I really can’t blame Bazaar. The GTK codebase is, of course, rather large. First preserved commits were in 1997; there are over 15,000 revisions; a working tree (excluding VCS metadata) is 83 MB in size. The bigger problem however, is that Subversion is not fast at accessing historical data (the time taken to process revisions started at over 32 minutes per 100 but eventually dropped to about 9 minutes per 100 as it got to the most recent history; interesting).

Clearly, I could have picked an easier example for my experiments with bzr-svn, but it’s done now.

Usage is fine once branch created

I wasn’t entirely sure whether the DVCS property that branches are all peers would apply to one that had started life backed by a Subversion repository would hold, but things work just as they are supposed to. Creating a new branch to work on a feature with bzr branch, or use the technique of using a bzr checkout along with bzr switch to do change-in-place, both worked fine. So I’m all set.

The real point, though, was to encourage more GNOME hackers to give Bazaar a try. If everyone had to put up with a endless import process like mentioned above then no one would touch it. But now that the branch is created (and up to date as of yesterday or so), anyone can use it.

With Olav’s concurrence, I put my branch on in my GNOME directory, so you can branch from it as follows:

$ bzr init-repo --rich-root-pack gtk+
$ cd gtk+/
$ bzr branch http://www.gnome.org/~afcowie/bzr/gtk+/trunk/

You do the first step so that you can create a “shared repository” (in Bazaar parlance) which will allow that various branches under that directory will share all the revision data for the project. I need to warn you, though. Modern DVCS tools give you the full history of the project, but for a large project that can be pretty big. If you grab the above branch you’ll be downloading about 180 MB and it takes about 5 minutes. Yes 5 minutes is a long time, but no there’s no way around it* and in any case 180 MB is not unreasonable for such a mature project. Don’t be too worried if it seems like it is taking a bit to do the transfer. You only need to do it once, and look on the bright side. It took me 2 days. Just let it run.

Workin’

Personally, I always do my work in a working branch separate from my mirror of upstream, allowing me to easily compare against the last update of upstream that I have done. So:

$ bzr branch trunk working
$ cd working/
$ ./autogen.sh
$ make

and away we go. Branching takes like 3 seconds. Nice and fast, no problem.

I haven’t got a script running religiously updating my GTK branch; I do a bzr pull locally once every few days in my copy of 'trunk' and have been pushing that up to http://www.gnome.org/~afcowie/gtk+/trunk/. But now that you’ve got the branch, you don’t have to rely on me anymore; this whole distributed thing kicks in and you can do it yourself. Assuming you have a svn.gnome.org account, then you can update with:

$ cd ../trunk/
$ bzr pull --remember svn+ssh://username@svn.gnome.org/svn/gtk+/trunk

(use the --remember argument the first time to change where it updates from by default so you’re not relying on my branch anymore) and push with:

$ bzr push svn+ssh://username@svn.gnome.org/svn/gtk+/trunk

The very first time you update from Subversion bzr-svn will have to build some caches and whatnot locally, but it’ll be pretty fast from there on.

I’m ignoring the pushing and pulling of revisions between your mirror of ‘trunk’ and your local 'working' (or whatever) branch; I’m sure you can figure that part out yourself: but here’s a hint: do your work, test it, and commit it, all in 'working'; repeat; then use, say:

$ bzr diff -r ancestor:../trunk/

to consider what your current patch looks like, then shuttle it to 'trunk' with bzr pull or bzr merge (assuming you run it from 'trunk’ and whether you need to merge or not) and then bzr push to svn.gnome.org. Hope it works for you.

If you need help, poke your head into #bzr on FreeNode or write to their mailing list, or ping me and I’ll try and lend a hand. Make sure you’re using the latest releases; I doubt it will work otherwise and so recommend that you use bzr >= 1.4.0 and bzr-svn >= 0.4.10. Older versions do not contain critical bug fixes. I also encourage you to grab bzr-gtk >= 0.93.0 as the visualize command it adds:

$ bzr viz

is really amazing.

Good luck!

AfC

Update:
You need to have PycURL installed (it’s an optional dependency; used at runtime if detected). There is a bug that will prevent you doing the branch from me if you don’t. Debian and Ubuntu package python-pycurl; Gentoo USE=curl pulls in package dev-python/pycurl; Fedora package is also python-pycurl.

This very moment as the GNOME systems administration team is presently working through the consequences of the Debian SSH key vulnerability; they have for the moment very properly locked down all access to the services provided to GNOME hackers, and that knocks out access to the Subversion server meaning people can’t work with their source code properly. If there was ever a clearer demonstration on the inadequacy of old 1st generation centralized version control systems, I cannot think of it.

This blog post is an extract of the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.7 (30 Apr 2008)

Draw some.

In addition to the usual improvements to our coverage of the GNOME libraries, this release introduces preliminary coverage of the Cairo Graphics drawing library, along with the infrastructure to make it work within a GTK program.

Drawing with Cairo

The trusty Cairo context, traditionally declared as a variable named cr in code, is mapped as class Context. Various Cairo types such as different surfaces and patterns are mapped as an abstract base class (Surface, Pattern) along with various concrete subclasses (ImageSurface, XlibSurface, and SolidPattern, RadialPattern, etc). Error checking is implicit: the library status is checked internally after each operation and an Exception thrown if there is a failure.

Thanks in particular to Carl Worth for having reviewed our API and having helped test our implementation.

New coverage and continuing improvement

The single option choice buttons in GTK are called RadioButtons and have now been exposed. When using them you need to indicate the other buttons they are sharing a mutually exclusive relationship with, and this is expressed by adding them to a RadioButtonGroup.

The usual steady refinements to our coverage of the GtkTreeView API continue. There’s a new DataColumn type for Stock icons, and TreeModelSort is now implemented, along with minor changes to various other miscellaneous classes.

Considerable internal optimizations have been done, especially relating to ensuring proper memory management, with notable refinements to make use of “caller owns return” information available in the .defs data. This fixes a number of bugs. Thanks to Vreixo Formoso for having driven these improvements.

Error handling has been improved for GLib based libraries as well. If an ERROR or CRITICAL is emitted, our internals will trap this and throw an exception instead, allowing the developer to see a Java stack trace leading them to the point in their code where they caused the problem.

Internationalization support

java-gnome now has full support for the GNOME translation and localization infrastructure, including the standard _("Hello") idiom for marking strings for extraction and translation, but combined with some of the powerful support for positional parameters available from Java’s MessageFormat as well. There’s a fairly detailed explanation in the Internationalization utility class.

Build changes

Note that as was advertised as forthcoming some time ago, Java 1.5 is now the minimum language level required of your tool chain and Java virtual machine in order to build and use the java-gnome library.

Thanks to Colin Walters, Manu Mahajan, Thomas Girard, Rob Taylor, and Serkan Kaba for contributing improvements allowing the library to build in more environments and for their work on packages for their distributions.

The download page has updated instructions for getting either binary packages or checking out the source code.

Documentation, examples, and testing

Refinements to the API documentation continue across the board, notably improving consistency. A large number of javadoc warnings have also been cleaned up.

While not a full blown tutorial, the number of fully explained examples is growing. There are examples for box packing and signal connection, presenting tabular data, and basic drawing, among others. See the description page in the doc/examples/ section.

This code, together with the not inconsiderable number of unit tests and the code for generating snapshots of Widgets and Windows means that a large portion of the public API is tested within the library itself. The number of non-trivial applications making use of java-gnome is starting to grow, which are likewise providing for ongoing validation of the codebase.

Summary

You can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.6..tag:v4.0.7

Looking ahead

It’s probably unwise to predict what will be in future releases. The challenge for anyone contributing is that they need to understand what something does, when to use it (and more to the point, when not to!), and be able to explain it to others. This needs neither prior experience developing with GNOME or guru level Java knowledge, but a certain willingness to dig into details is necessary.

That said, I imagine we’ll likely see further Cairo improvements as people start to use it in anger. It shouldn’t take too long until the bulk of the functionality needed for most uses is present in java-gnome. In particular, forthcoming coverage of the Pango text drawing library will round things out nicely.

There are a number of other major feature improvements we’d like to see in java-gnome. Conceptual and design work is ongoing on for bindings of GConf, GStreamer, and even support for applets. Within GTK, there have been a number of requests made for various things to be exposed, for example, the powerful GtkTextView / GtkTextBuffer text display and editing capability. Some of these have preliminary implementations; whether or not any given piece of work is acceptable in time for any particular future release will remain to be seen and depends on the willingness of clients to fund us to review and test such work.

In the mean time, people are happily using the library to develop rich user interfaces, which is, of course, the whole point. We’re always pleased to welcome new faces to the community around the project. If you want to learn more, stop by #java-gnome and say hello!

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

AfC

The Java Servlet APIs are comprehensive, and there are a number of very good implementations (we use Resin in highly loaded environments and it scales like a dream; we recommend it heartily to our clients). Nevertheless, sometimes you’re working on something small or ad-hoc, and you just want a simple web server that doesn’t involve application descriptors, XML, configuring an entire service instance, and everything else. Somewhat to everyone’s long standing annoyance (not to mention the derision of people who work in other languages), there is nothing in the Java standard library to just get on with this.

In their release of Java 1.6, however, Sun quietly included such a server. Apparently it had been around for some time powering internals of various other projects, but in 1.6 they exposed it as a public library, in package com.sun.net.httpserver, and it ships alongside the rest of their Java VM and standard library.

It’s got a rather nice API. You fire up a server and add one or more handlers:

        addr  = new InetSocketAddress("localhost", 8000);
        server = HttpServer.create(addr, 10);
        server.createContext("/eg", new ExampleHandler());
        server.start();

the “contexts” are paths from root that are handled by different HttpHandler instances; implementing that interface is simply:

    public void handle(HttpExchange t) throws IOException {
        ...
    }

Once there, you’ve got a number of really straight forward methods on HttpHandler to get done what needs doing:

        t.getProtocol();
        t.getRequestMethod();
        t.getRequestURI();
        t.getRequestHeaders();
        t.sendResponseHeaders(code, length);

for the metadata, and an InputStream and an OutputStream from:

        t.getRequestBody();            
        t.getResponseBody();

respectively. You write the response to the later. Too easy. I did up a modified version of the example they have in the package description file as something that echos back the HTTP request as the response and it was quite straight forward. Java programmers can glance at EchoServer.java if they’re interested.

The catch

Nothing is ever simple. The bytecode for the com.sun.net.httpserver package is bundled in with the rt.jar that comprises the bulk of the standard library, but for whatever reason they didn’t include the JavaDoc for these classes in the same tree as the rest of the API documentation.¹ This is a huge pain, because once you’ve told Eclipse where to look for the docs for a given blob (in this case, for rt.jar) you can’t then tell it to look somewhere else for a sub part. And if you’re trying to develop something in a beautiful IDE like Eclipse, the idea of not having inline popup documentation and completion working properly is just a non-starter.

You can point Eclipse at either a JavaDoc tree, or at the raw sources and it’ll just extract the documentation on the fly. But since the src.zip shipping with Sun’s proprietary Java 1.6 didn’t include the classes for this web server code, that didn’t work either.

So that would have been the writing on the wall for Sun’s httpserver.

Open Java to the rescue

After chatting with some people in #gentoo-java and #classpath, we suddenly realized that the solution was at hand. Source code to Open Java is available.² And even though it’s still not yet a fully Libre replacement for Sun’s proprietary Java 1.6 (the size of the binary proprietary plugs you have to drop into it to make it work is staggering), it does contain most of the sources to the Java standard library. So we grabbed a copy of the openjdk 1.6 b08 code drop, and just pointed Eclipse at it as the sources for rt.jar for cases where it couldn’t find the JavaDoc when it was looking something up. And ta-da!

So, amidst considerable astonishment but great pleasure, here was an example of something someone had Open Sourced working in a synergistic way but in a way that they hadn’t quite expected. This wouldn’t have worked if Sun hadn’t Open Sourced (most of) Java; but because they have we were able to use those sources to solve a problem. Thanks to the heroes at Sun Microsystems who made this possible!

Eclipse users will have to adjust their filters so that com.sun.* isn’t excluded from Type completions and what not; While we’ve all loved our com.sun.* excludes, it blots out com.sun.net.httpserver. If you plan on using it, then I suggest switching to excluding com.sun.org.* and com.sun.xml.*, etc. A pain, but that’s what we get for Sun not having put this into the standard library.

Speaking of which, I doubt we’ll ever see this as a part of the “Java” (wouldn’t java.net.httpserver just do the trick?) because the marketing side of Sun has a fairly strong fixation on “Servlets are what you use for everything”. Oh well. It would be a really pleasant surprise, though. Here’s hoping.

But not really

We would never use this in something we were distributing to anyone outside; after all, they might be using some other VM and com.sun.net.httpserver isn’t available as a standalone library. And that’s not to mention scalability; if you need something bulletproof then use the Resin servlet runner. But if you need a quick-and-dirty but easy-to-use HTTP server and you happen to have Sun Java or the future tense Open Java available, then this should do the trick for you. And if you’re feeling like whipping up something of your own, this really does have a nice API (somewhat a rarity in the Java landscape) and so this might be a good starting point for you. Either way, have a look.

AfC

¹ As mentioned, the API docs for com.net.sun.httpserver are not in the usual location at docs/api/ along with the rest of the standard library but at docs/jre/api/net/httpserver/spec/ of all places. Who knows what they were smoking that day.

² I am very tired of apologizing to people who have just installed “Java” on their Linux boxes that they have to download something else. I’ve given up trying to explain the difference between a JRE and a JDK to people; even after I do people respond with “who cares; I just want a working Java that I can use to build your software.” Sun having attempted to brand their Libre implementation of Java as “Open JDK” just perpetuates this nonsense. Not including the compiler classes in something that is already 18 MB big is stupid, and it’s not like the desktop VM is what you use on a resource limited embedded device. Open Java works just fine as a name, thank you very much, and it is (almost) a Free Java. Almost.

Adding Cairo is a feature we’ve been working on for about 6 months now, and so I’m pretty pleased that during the 4.0.7 development cycle we’ve been able to land it. Cairo is a huge library, of course, but we’ve put enough coverage in place to ensure that things are working.

Cairo has lots of convenience functions and tons of obscure uses; no surprise (and no apology) that there’s still lots that will need doing. If you want to help make sure it has what you need, then grab ‘mainline’ and see org.freedesktop.cairo.Context and ExampleCairoDrawingInExposeEvent.

Huge thanks go out to Behdad Esfahbod and Carl Worth; Behdad was really critical in explaining some basic Cairo concepts to get me started when we were working together at the GNOME Summit back in October in Boston, and at the GTK hackfest in Berlin in March, Carl Worth checked our preliminary APIs and helped sort me out as we were working our way through to create some examples. Awesome!

java-gnome 4.0.7 should be out in the next week or two, depending on how long it takes to finish up testing, QA, and to do the release engineering.

AfC

#gnome-hackers		#gentoo-desktop

The upgrade to GNOME 2.22 went really smoothly. Well done to the Gentoo Linux desktop team for their heroic efforts bumping so many .ebuilds, and congratulations to all the GNOME hackers, testers and translators around the world on what continues to be superb software!

AfC

My wife asked me what I was reading. I showed her the title and when she saw it concerned computer memory, she looked at me with a totally straight face and summed up the 114 page paper by answering “Yes, you should get some”.

Like most of Ulrich’s writing, this paper is immense: long on detail and well presented. I must admit that I agreed with many others that after having read the work I was left wondering what action, in practical terms, I should be taking as a result. Clearly most of this knowledge is very important for Kernel hackers and for those doing low level systems programming, but since I do higher level work (ie advising people with large enterprise systems), it is hard to asses the choices I might be making in the terms of what Ulrich describes. I can only hope that the people who work on the parts of the low level libraries that run next to the metal (such as drivers, GLib, and especially the VMs like the Java and Python runtimes) are aware of these issues and making design choices based on these insights.

AfC

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.6 (12 Feb 2008)

Finding the missing methods.

Most of our effort recently has simply been fleshing out areas of the public API. The focus for this work as been getting the coverage needed to allow us to port some of our in-house applications to java-gnome 4.0. It’s not especially glamorous — if anything it has been tedious as hell — but the result has been a large body of improvements to java-gnome as a whole which we’re pleased to release as java-gnome 4.0.6

The bulk of this development took place on the ‘missing’ branch, so named because that’s where I was working on what was missing :).

Continuing Improvement

Notable public changes include coverage additions to enable key stroke and mouse button handling

Rather than exposing the int keyvals that bubble up out of the X server, we have wrapped these as constants of type Keyval (thereby being consistent with the rest of java-gnome in our working to the strengths of Java as a strongly-typed language; MouseButton was created for the same reason, helping developers understand just what on earth mouse button “1” is, anyway). Along with ModifierType, this gives enough to deal with the KEY_PRESS_EVENT and KEY_RELEASE_EVENT signals when the developer wishes to deal with key strokes.

We’ve finally gotten around to providing proper coverage of the box packing model which underlies every aspect of how GTK presents user interfaces. To understand the size-request/size-allocation process, you might start with Widget’s setSizeRequest().

We’ve also added coverage for SizeGroup, which, when used in concert with nested VBoxes and HBoxes, can work wonderful magic and is often far better than messing around with Table when doing complex layouts.

After dithering for several releases, we’ve settled on how we’re going to deal with ComboBox and family. The underlying GtkComboBox presents something of a nightmare as it is really two classes in one with more-or-less incompatible APIs. So, not surprisingly, we’ve presented it as two sets of classes, with the text-only convenience API spliced out of ComboBox and ComboBoxEntry into TextComboBox and TextComboBoxEntry respectively.

We’ve added a few new features in our coverage of GTK’s TreeView / TreeModel API, and many other classes involved have also seen improvements. The persistent reference to a row provided by TreeRowReference is now available as is model type TreeModelFilter.

Support for the actual filtering in TreeModelFilter is notable for having been quite tricky. The underlying C library use a function pointers rather than a GObject signal emission, and we don’t have any mechanism to handle that. We do, however, have a fantastic capability to marshal signals, so we dealt with the problem by creating a custom signal and then passing a function which emits it when the TreeModelFilter wants to ask the developer whether to include a row or not.

The new classes include support for TreeModel columns storing long data as well as setting properties of that type.

It should also be noted that most of the methods taking a TreeViewColumn have been converted to taking an argument of type CellLayout (an interface implemented by TreeViewColumn). This has no change to how you use our TreeView API, but was necessary to support ComboBox properly.

Finally lots and lots of minor additions to both public APIs and internals deeper down in the GDK part of the toolkit.

As ever, you can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.5..tag:v4.0.6

Documentation

We’ve always had HTML JavaDoc for the current stable release at doc/api/ on the java-gnome website. We’re going to change that a bit, though. As fixes to the explanatory documentation happen quite frequently to classes and methods all over the place, we’ve decided to generate the JavaDoc from ‘mainline’ periodically and upload that instead. This means that there will, of course, be descriptions of some methods which aren’t yet available in a released version of the library, but they will clearly identifiable by virtue of having a @since tag showing a version number greater than the most recent release. We’ll revert back to more traditional behaviour once we hit 4.2.0, but in the mean time we can give people access to the best information we can online.

The idea of having up-to-date illustrations of the various Widgets has proved popular, and we’ve continued to update the suite of snapshots. Doing that is also tedious, but it does provide a good opportunity to test APIs we are exposing especially where unit tests are less suitable.

Looking ahead

Almost as complex as the TreeView / TreeModel API are GTK’s powerful TextView / TextModel classes, collectively a Widget used to display and edit large text documents. Working out the java-gnome coverage for TextView will take a fair bit of consideration, but TreeView provides a road map, and, as with the coverage in 4.0.5 and 4.0.6 (which was driven largely by existing applications we were porting), we have some significant uses of GtkTextView which will guide us on our way.

The next release will also feature significant work outside of GTK; we should be in a position to merge our coverage of the excellent Cairo drawing library soon, and likewise we have tentative work in place letting people store configuration and settings data in GConf. Both the ‘cairo’ and ‘gconf’ branches need more QA and documentation work, but they’re looking good and will definitely be featured in java-gnome 4.0.7.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

Well, now that I have this out of the way, I can finally stop worrying about having so much unreleased code and get back to the business of doing what I do for a living.

On a personal note, I mentioned that much of this coverage was driven by one of our legacy apps. About half way though this cycle, however, we decided we would be abandoning that program. Needless to say, it was more than slightly depressing to be putting so much work into writing new coverage for java-gnome in order to port such a large codebase in the certain knowledge that the application was not going to be resurrected after all, especially as we have not yet managed to raise any revenue to offset our costs in taking on the Java bindings for GNOME.

Cognitive Dissonance.

Nevertheless, I kept at it largely because getting that app to run again represented a good test of the quality and comprehensiveness of the Java bindings. It needed doing, and that has ultimately been the motivation behind all our work re-engineering java-gnome. I gotta say it was wonderful seeing that program come back to life bit by bit as the errors slowly fell away as missing classes and methods were added to the library. That monster running again after almost two years represents the 4.0 version of java-gnome having surpassed the level of coverage that we used from the old 2.x version, and that’s something to be very positive about indeed.

Of course, now all I have to do is to continue to resist the temptation to go back to working on that beast. :) No, better things to be about.

AfC

#! interpreter

Anyone who has ever written a Shell script, Perl program, or Python program is familiar with #! lines:

    #! /bin/sh
    #
    # A program to do something very special.
    #

    echo "Hello World"

and

    #! /usr/bin/perl
    #
    # Another program to do something very special.
    #

    while (<>) {
        print "Hello World\n";
    }

etc. The program mentioned after the magic #! characters is the program that will interpret the script. There are many gotchas with that (notably portability concerns owing to the fact that some idiotic flavours of Unix don’t put Perl in /usr/bin, that sort of thing)

I always figured that the script file got piped by the OS into the interpreter on stdin. A reasonable guess given the way that most of the tools we use work, but it turns out it is nothing of the sort. Every time I tried to write an interpreter (in C) I got stuck.

What threw me off was that cat works as an “interpreter”:

    #! /bin/cat
    This is the script body
    which will happily be sent
    to stdout

If you put that in a file called script and run that from your terminal, then sure enough,

$ ./script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$

which of course is exactly what would happen if you did:

$ cat < script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$

and that’s what totally had me on the wrong track. I figured that the interpreter on the #! line was being fed the body of the executing file on stdin. Nope.

Seek and ye shall find, sort of

I thought that I might be able to find out what was going on by reading the code of an interpreter program. I started by looking at the sources for /sbin/runscript (which is on the #! line for all of Gentoo Linux’s RC scripts), expecting that to be quite simple. It was simple. Too simple. All it does is some environment filtering and then fires off bash to run /sbin/runscript.sh (in other words, it’s largely a workaround for the fact that you can’t actually make a shell script itself an interpreter). Nothing at all in there about reading stdin. So then I looked at the source code for Perl (Whoa, there’s a beast). Nothing obvious there either. Lots of stuff about reading from stdin but nothing about that being the origin of the script to be executed. A lot of messing around with argument signatures though.

#! is not exactly the easiest term to put into a search engine. I did, however, happen to know that one of the ways #! is pronounced is “hash bang” (being two common names for the respective characters, though lots of old suspender-snapping sandal-wearing bearded Unix freaks would, I’m sure, tell you with great passion that it has to be pronounced some other way). Searching on “hash bang” brought up lots of arcana, including something that lead me to an obscure article by one Andries Brouwer on the parameter signature at invocation wherein I discovered that there is a calling convention for how arguments are passed to the interpreter program being invoked.

It’s a bit complicated, since you can have command line arguments for both the interpreter and for the script being run. It goes something like this. Let’s say you have an script that begins with the following:

    #! /path/to/program -v -d

(with -v perhaps meaning “verbose” and -d perhaps meaning “debug”) and you have it in a file called ./script, then running it will actually cause program to execute. The trick is, with what arguments? Check this out. If you do:

$ ./script -p -r

(with -r and -p, for the sake of illustration, perhaps having the same meanings as cp, that is “preserve” and “recursive” respectively) then when our interpreter program is executed, it will be invoked with the following arguments:

/path/to/program -v -d ./script -p -r

the mapping is a bit obscure. It’s actually:

(to use the terminology in the above link). This all shed a little light on what I’d seen in runscript.c and perl.c, but still not a single mention of the script being fed in on stdin. So I pondered that for a while longer, until finally the light bulb went off.

Eureaka & Company

The reason I couldn’t find any mention of ./script being fed in on stdin is because is is not fed in on stdin. You don’t need it to be: you’ve got the name of the script file fed to you in your interpreter’s argument list (from the above example, it’s in argv[2], one happy looking string containing “./script”). So read it already!

    FILE* body;

    body = fopen(argv[2], "r");
    ...

and ta da, that’s where you get your script’s program body from. Now you can at last get on with parsing your script, and running it.

Most big programs spend lots of time munging the argument list, dealing with the fact that argv[1] could be full of all sorts of stuff jammed into, or nothing, etc. The whole thing goes from elegant to clumsy when you discover that if there are no arguments to the interpreter on the #! line then the script file will be in argv[1], and it goes to nightmare level when you look at the list of variations in behaviour across different operating systems, compiled by one Sven Mascheck. Nonetheless, the interpreter is your program, and presumably you can recognize, parse, and skip over zero, one or more arguments to yourself before deciding you’ve reached the name of the script. Judicious use of argv++; argc--; is your friend here, apparently.

Anyway, this all explains why my cat example was working but my own efforts were not. cat is not reading data being fed to it on stdin (which is cat’s behaviour if you run it without any arguments), it’s being executed with an argument, namely ./script as argv[1], ie exactly the same as:

$ cat ./script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$

But now that I know what’s going on, I can write my own interpreter.c:

    #include <stdio.h>
    #define LEN 128

    int main(int argc, char** argv) {
        char buf[LEN];
        FILE* body;

        body = fopen(argv[1], "r");
        while (fgets(buf, LEN, body) != NULL) {
            printf("%s", buf);
        }
        fclose(body);

        return 0;
    }

and if I compile that to interpreter, then I can write a domain specific language that is interpreted by this program, say:

    #! ./interpreter
    This is a test of the Emergency Broadcast System

in a file called script, then, at last,

$ ./script
This is a test of the Emergency Broadcast System
$

Yeay!

Ok, so that’s cat, but cat is the Hello World of input/output. :) The real point is that running script caused interpreter to be executed, and interpreter got at the body of the script that was “run”, and was able to do something with it. Onwards at last.

AfC

Comments

Julio Merino Vidal wrote in suggesting:

Take a look at NetBSD’s script(7) manual page for some more details
about how that is supposed to work and some things you must consider
for portability (such as being able to feed a single argument to the
interpreter through the #! line, or the maximum length of it).

Updates

Quite by accident, I just came across the related information for Linux; see the execve(2) man page for a succinct treatment of both exec()‘ing in general, and the topic of interpreting scripts.

“… but we have to charge money for our eggs. People who don’t acknowledge that just do not want to understand the term ‘Free Range’ for what it really means … better eggs, and changing the term will not help that.”

The fact that the discussion started because of maddog’s suggestion that maybe they should be called “open range eggs” to eliminate the confusion is not the point (“that’s silly” the owner said, “everyone calls them free range eggs”). The term we use, Free Software, has a bigger problem. Consider the difference between:

• Free Range Eggs
• Free Eggs

Clearly, the term is “Free Range”, and applies as an adjective to “Eggs”, whereas the latter really does mean “free eggs”. Now consider this:

• Free Software

There’s something missing, and so the term free gets connotated as having to do with price.

No, I’m not about to say that we should call it Free Range Software [and while “let it run free!” is a lovely metaphor, I don’t quite think we want to be associating our work with chicken farming :)]. Perhaps someone will come up with an intermediate word that will do the trick. To be honest, though I’ve pretty much given up on the term Free Software; I write Open Source software, and the cause I advocate is Software Freedom.

And when people still stare at you blankly, you can say “you know, like Linux” and watch as comprehension dawns. To be sure, they probably still don’t get it, but chances are you’ve got more important things to talk about, and getting on with it is going to do you — and logiciel libre — a lot more good than getting lost because of the insufficiencies of the English language.

AfC

Thanks to Atul Chitnis for having passed on the link.

The documentation doesn’t help you much though. First, it is not sufficient and second (and important) you do not learn much from the documentation.

Thankfully, you have the source code and I really appreciate the source code of Eclipse is open. That’s because only from source you can learn as much as necessary. And this fact leads me to think more and more that open source is not about “reusing software” (commercial companies do that as well) but about “reusing experience” which is hidden inside the source.

I have the strong belief that people who write the software are more important than the software itself and by looking to the source code you can gain at least part of experience of people who wrote it. That’s the power of open source, that’s “reusing experience” concept at work!

This observation was written a few years ago by one Alexander Dymo who was expressing his amazement at the whole Software Freedom thing, especially the hands-on side of it. And as for his “people are more important than software”, well, hey; I couldn’t agree more. Enlightened organizations that want to preserve their strategic advantage understand that the people who are capable of working with such code are their competitive advantage. They are the ones who can reuse experience and leverage the power of the most wide reaching and enabling social phenomenon we have ever seen: the global openness movement.

Towards a technical definition of Open Source

I’m on a bit of a kick at the moment working to elucidate a practical technical definition of Open Source (ie, complementary to the necessary legal foundation which enables Free Software and the requisite social interaction which is at the heart of global Open Source communities). Comments like the one above are helping refine my thoughts on the topic: yes the interaction of licence and copyright law gives us structure, and yes communication between people distributed the world over is the genesis of the sense of community, belonging, and accomplishment which is the rich social fabric within which our software development takes place, but there is also a pragmatic aspect: can you actually work with the source code, get right into it, experiment with it, break it, and do crazy things with it?

That the four GNU freedoms stipulate that this must be “permitted” doesn’t really change the fact that there are practical prerequisites. Can you easily get the sources under development? Do those sources actually build? Is there a clear mechanism for contributing source back to the project and are they actively facilitating such contributions?

The source tarball as primary release artifact

The biggest give-away is whether the primary release artifact is source or an opaque binary.¹ Especially in the Java world but elsewhere as well, there is a surprising amount of activity for which, despite the fact that it may legally be Free Software and its loud proclamations to being Open Source, it remains clear that they just don’t get it: there are a huge number of projects and products which only do their releases in binary form. Bad sign when you start calling it a “product”, I think. In a frustratingly large number of cases, if you try to actually work with their code you will discover that it doesn’t actually compile! In extreme, there are statements like:

We distribute source but never claimed that you can build it out of the box. We don’t have time for such things.

I didn’t believe it when I saw it, but one of the hackers of a supposedly Open Source project actually said that in response to a bug report. Astonishing.

Being able to duplicate the result is a rigour that goes far beyond software; indeed it has been the bedrock of science — and human progress — since the dawn of the age of reason. Back to the present day, it has suddenly become obvious to me that the fundamental technical difference between proprietary Unix from the commercial vendors (not to mention proprietary operating systems from Microsoft and Apple) and Linux is that in our world (taken to mean the entire continuum of FOSS communities) the primary release artifact for all upstream projects is a source release (these days it’s typically in a .tar.bz2, but whether it is that or .zip or whatever else isn’t terribly relevant) that you can build. Not a tarball full of already built .class files and .jar files. Not a “source .zip” for Eclipse. Not a self-extractable full of .exes and .dlls. No. A source release is source code that someone else can build, right out of the tarball. THAT, ladies and gentlemen, is the technical definition of Open Source.

I’ve started to realize that this area is a big stumbling block. People I collaborate with in (upstream) global projects like GNOME, Free Java, Bazaar and elsewhere take the efficacy and primacy of source releases entirely for granted. But a number of clients that my firm is working with to enable Open Source often just don’t get why they should be doing source releases — and resist it.

Whether you buy into Software Freedom or not is a different topic (and your decision, not mine to impose on you), but if you’ve got a software project for which you’ve decided to free the source, you want it to be successful, right? Success is a project that which inspires user enthusiasm, which the major distributions can package and ship without hassle, and around which a vibrant community of developers grows, ultimately fostering new contributions. There are a lot of steps along the way, but a buildable source release is, in our view, the technical bar you’ve got to reach. Otherwise you’re not making releases of the software, you’re abandoning it. And your users.

AfC

¹Note that this is different from a Linux or Unix distro shipping binary packages — no matter if it was Fedora, Debian, Gentoo, FreeBSD, OpenSolaris or anyone else, they should still have been able to build their package from source. It matters little whether that compilation took place in a build farm somewhere or on the user’s desktop (which is what happens in the “binary” distros when someone wants to work on the package or the project itself) — the key point remains that we can build the software if we want to, and are not forced into relying on someone else’s proprietary (and more to the point, unavailable) tooling. If you can’t build it, it’s not Open Source.

Despite the flame bait, the thread contained some surprisingly insightful replies. It’s always great to hear some of the top software developers in the world noting their motivations and why they believe what they do works.

From Roman Kennke:

Both approaches (closed and open) apparently tend to produce relatively high quality code (or really crappy code, happens in both camps), where with the closed approach the developers (or vendors) have to take over 100% responsibility (because the end user has no way to interact with the development), which usually makes things very formal and slow, where the open approach relies very much on the end users reporting problems. In most active projects these are fixed really quickly, giving both the developers and the end users a warm fuzzy feeling ;-)

From Andrew John Hughes

There’s a lot to be said for feedback and interaction with your users that’s often overlooked. All the ideas of complicated quality control processes in the world is not going to make a user feel as loved as seeing someone responding quickly to their bug and fixing it in a short space of time.

From Mark Wielaard, a remark on the complex administrative process used by the project to review contributed code:

We do have a flow chart that people have to follow when contributing… It is all very formal really: http://gnu.wildebeest.org/~mark/patch.png

and from Archie Cobbs, a reminder about the track record of a certain formerly proprietary process on solving bug desperately desired by their user community:

The number #1 voted bug in their bug database has been unfixed for over 5 YEARS!

The comments on that bug make for hilarious reading, but the bigger point is this: the identity of the people making the decisions about the relevance of the issue are hidden. That sort of thing doesn’t inspire much hope for people on the outside. It’s not like we’re talking about national security or the future of western democracy; it’s a bug report that turned into a feature request for a piece of software that many, many people depend on! No one likes to be fed the line that their problem is so Top Secret that they won’t be told when (or even if) the problem will be addressed. The cloak of anonymity strikes again.

Fascinating thread.

AfC

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.5 (26 Nov 2007)

TreeView is here!

It’s always a great feeling when you bag a milestone, and with this release we have reached a major goal on our way to having outstanding Java bindings for the GNOME platform: coverage of GTK’s powerful yet complex TreeView & TreeModel API.

TreeView

TreeViews are a central part of almost every application. GUIs use lists for all sorts of things, and so a significant goal was to make coding TreeViews and their backing TreeModels as straight forward as possible.

The most challenging and complex part was to design the Java side API, which was no small matter. As a native library, the GtkTreeView API is complex and very much written with programming in the C language in mind, and as such our algorithmic mapping of the underlying libraries into Java doesn’t entirely fit. Long experience with the TreeViews in the previous bindings had made it clear just how nasty to use the API could be, and so the hardest part of the work was to come up with a mapping and a usage pattern that would be both faithful to GTK and be sensible to use.

The other significant challenge was to document the work effectively. Our Java side API documentation is a major feature of java-gnome, and merely exposing classes and methods is not sufficient; they need to be clearly explained in our JavaDoc as well.

along with numerous test cases in our unit test suite, and several comprehensively worked examples.

This was a monster patch, and the culmination of not just three months direct effort, but also where we’ve been heading since we first started the re-engineering of Java bindings for GNOME. Although largely written by Andrew Cowie, a significant contribution was made by Srichand Pendyala who not only exhaustively evaluated the design but also threw in some serious chunks of code. The work benefited from comprehensive input from Peter Miller on the modelling and design, and the comments of Bryan Clark, Owen Taylor, and Hanna Wallach were all really positive and helped us know that we’d gone in the right direction. Finally, thanks to Behdad Esfahbod and the GNOME Foundation who made it possible for us to meet in Boston at the GNOME Summit and so accomplish much of the final pulling together of this branch.

Continuing Improvement

Meanwhile, steady work continues on to the fundamental base classes, with a whack of additional signals and methods on Widget and especially Window, along with expansion of coverage in numerous other classes.

We’ve begun to get the basics of image handling in place,

One nice piece of contributed work came from Vreixo Formoso and Thomas Schmitz with coverage of the Dialog Window functionality in GTK. It took a bit of doing to map the int response codes used by GTK into something suitably strongly-typed, but all good.

Finally, minor improvements to all sorts of stuff, notably from new contributor Mario Torre. Awesome!

For further details you can always grab a copy of the sources and run

$ bzr diff -r tag:v4.0.4..tag:v4.0.5

to see the complete code delta.

Screenshots

For fun we built in a capability to create demonstrations to be captured as screenshots to illustrate various things. It doesn’t get more basic than the example on the Window documentation page, but it’s a nice touch. :) We’ve screenshots for a number of Dialog classes and one for the TreeView page. I imagine we’ll build up a nice library of images in the next few months (yes, dear contributors, you can add snapshots to the list of things we’ll be expecting along with well written documentation and unit tests when submitting additions to the public API).

Building and requirements

The library now depends on GTK >= 2.12. Those packaging java-gnome for their distributions please take note.

Looking ahead

Continuing to expand the coverage levels in the classes already exposed will continue to dominate our attention; there’s still a long way to go but we’re pleased with the progress we’ve made so far; you can definitely build real applications with java-gnome now.

The next release also ought to include preliminary coverage of GConf and Cairo. Doing each justice will again take a serious amount of work, but will continue to grow the fun things you can do with java-gnome.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

This release marks a particularly satisfying milestone for me. I have three+ years of work that’s been waiting to be ported to java-gnome 4.0, but I’ve needed a new TreeView binding in place before I could do so. So this is a big deal for me. At last I can start thinking about getting back to actually hacking on the applications I actually want to be working on, and not spending so much of time on the underlying bindings themselves. There’s still a lot that needs doing on java-gnome, but it’s getting closer.

AfC