| Mon | Tue | Wed | Thu | Fri | Sat | Sun |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| 27 | 28 | 29 | 30 | 31 |
This section:
Blog postings by Andrew Cowie about Open Source and Software Development
The syndication links at top left will give you a feed for the blog as a whole.
If you'd like a feed specific to this sub-category, see bottom of page.
Sun, 05 Oct 2008
Wine your own business
The things that drive you crazy…
There’s one (and only one) legacy Windows application that we have. We used to run it in VMware, which worked, but that was a pain, looked terrible, and of course was required us to have our copy of Windows 2000 installed there. Decrepit. But far worse, meant we had to use Windows. Yuk. And worst of all to have to keep switching in and out of it. Bah.
Life would be far better if we could run it on Linux under Wine — then it’d just be another app on running on the GNOME desktop and under the control of the window manager. We’d tried seeing if we could get this application to run in Wine a few times over the years; no joy. But somewhere along the line it started noticing list posts from other people reporting positive experiences. So {shrug} I gave it another try this month, and to my intense pleasure it installed cleanly and is running fine. Terrific! The people who hack on Wine are amazing.
So I started working away, only to suddenly realize that the dates were in American mm/dd/yy format. Gah. More searching, but none of the workarounds suggested (mostly to do with manually changing things with Wine’s regedit) seemed to work. Quote a pain.
I had come across a number of references saying that Wine and in turn the program in question would pick up the locale settings provided by your environment. I’d checked that, and tried various combinations. Nothing seemed to work.
I mostly run in en_CA, ie:
$ echo $LANG en_CA.UTF-8 $
the impact of which you can see by running the locale command:
$ locale LANG=en_CA.UTF-8 LC_CTYPE="en_CA.UTF-8" LC_NUMERIC="en_CA.UTF-8" LC_TIME=en_DK.UTF-8 LC_COLLATE="en_CA.UTF-8" LC_MONETARY="en_CA.UTF-8" LC_MESSAGES="en_CA.UTF-8" LC_PAPER=en_AU.UTF-8 LC_NAME="en_CA.UTF-8" LC_ADDRESS="en_CA.UTF-8" LC_TELEPHONE="en_CA.UTF-8" LC_MEASUREMENT="en_CA.UTF-8" LC_IDENTIFICATION="en_CA.UTF-8" LC_ALL= $
all as one would expect And yet it had no impact on running the program running under Wine [Setting LC_TIME to en_DK is an old trick for people in en locales to get proper 24-hour time formatting; yes, I tried unsetting LC_TIME; I tried dropping the UTF-8 settings; I tried LANG=en_AU. Nothing made any difference].
After a lot more frustration searching around, I finally came across a support article on CodeWeaver’s website that mentioned setting the locale environment variables. Yes yes, I thought, but then I looked more closely. They were setting LC_ALL. Wait a minute. LC_ALL is a special variable that “if set to a non-empty string value, override the values of all the other internationalization variables.” You’re not supposed to set that… certainly one doesn’t set that at login — that’s what LANG is for. I didn’t really expect anything to come of it, but I tried it anyway:
$ env LC_ALL=en_CA.UTF-8 WINEPREFIX="/home/andrew/.wine" wine "C:\Premier12\Myobp.exe"
and ta-da everything worked: I had dd/mm/yy date formatting just like we wanted. Yeay!
I immediately closed everything down and went out for a beer. Sometimes we forget to celebrate those brief moments when things actually work.
But talk about bitterness. I’m not sure if this was a Wine behaviour, a Windows behaviour, or just nonsense code in MYOB. I suspect the latter. But either way, I saw so many posts asking “how can I get the date format to behave under Wine” that I thought I should write about the workaround that did it for us.
AfC
Thu, 11 Sep 2008
Desktop beautification: fix your fonts
One of the nice things about Gentoo Linux is that they tend to leave the upstream defaults alone. This applies to file system locations, build configurations, but also things like preferences across the GNOME Desktop. This is good, in that Gentoo users see the raw upstream, but not entirely optimal in cases where those defaults aren’t as good as they could be or are too generic.
I was pairing with a colleague the other day and they were impressed with how I had things set up — and more to the point my rationales for having done so — and they encouraged me to write about it.
I don’t particularly think I have anything original or novel to say, but in so far as some of us have built up configurations over the years that (for ourselves, at least) improve usability and which define our productivity, I think it’ll be a nice contribution to show others what we’re using and how we got things that way. If you think you know what you’re doing, then I encourage you to write about your customizations — and why you made them. People using commercial distros like Ubuntu and Fedora will justly be able to say “it already looks good” and so if you’re happy you can ignore this, but even in such environments we still often tighten things up to individual taste and so I’m sure you have something to offer.
Anyway, thus begins an occasional series about things I’ve done to beautify my GNOME Desktop on our Gentoo Linux systems.
A few years back I was enjoying a drink with Carl Worth and Keith Packard (free desktop graphics hackers extraordinaire, for those who haven’t met them). Later in the evening Keith leaned over, saw my laptop’s screen, and starting swearing at me: “You need to fix your fonts!” (I was using Luxi or something). So, ok, what do you want me to do? They told me to switch to the Bitstream Vera family as being well sculpted, highly professional, and optimized for screen display. Needless to say I take what these gentlemen say extremely seriously, so did what I was told :).
Like any font or appearances change, for the first day or so it felt a bit odd, but then something shifted and it felt very comfortable. The real give-away that I was on the right track was a few days later when I tried switching back to the old font for the hell of it and recoiled in horror “Gawd that’s ugly!”
Bitstream Vera had done me well for quite some time, but recently I started running into some subtle bugs. One showed up with the weather applet; for whatever reason a few versions ago they changed the string from ‘°’ + ‘C’ to unicode character 0x2103, the symbol for degrees Celsius. I was getting a weird fall back to some horrible looking old bit-mapped font. Yikes (if °C and ℃ don’t look about the same then you’ll see what I mean). No big deal, but it was annoying me.
So I was checking around again to see if there was a different font family I should be using, when I came across the DejaVu. It starts from Bitstream Vera (so the base ASCII characters are all the same and mostly untouched) but adds a ton of hinting corrections, bug fixes, and extensions up in the higher unicode sequences needed by European languages and scientific work. There are also a number of additional faces available. And regular releases (always a good sign). And a different name, per the Bitstream licence terms. When I asked about DejaVu in #gnome-hackers, the answer was “hell yes you should be using it.” Huh. So I switched!
The DejaVu fonts are available on Gentoo in the media-fonts/dejavu package.
The “right” way to change the font globally would be to change the “Sans” and “Serif” and “Monospace” aliases in the fontconfig setup in /etc/fonts. But I generally try to leave those files alone, so instead ran gnome-appearance-properties and from the Font tab changed the system fonts to be what I wanted.
Immediately it fixed my glitch with the weather applet (yeay), but I started to notice anything using the higher order mathematical symbols was likewise drastically improved. Hurrah!
With gnome-font-viewer,
Making screenshots of what they look like on my screen is a bit silly because they won’t necessarily look the same on your screen. But hey. We do our best.
Open Source is about choice, and so I cannot talk about fonts without also mentioning the Liberation font family. These were obtained by Red Hat and intended as a drop-in replacement for the Microsoft core fonts (Courier New, Arial, etc) which are widely used but not libre. For whatever reason the Liberation fonts don’t look quite right on my screen (there are red and green hinting glitches around the glyphs; something is obviously misconfigured somewhere) but it’s a different free font that has been generously made available to the community. They’re available on Gentoo in the media-fonts/liberation-fonts-ttf package.
AfC
Update:
- So it turns out that DejaVu is set to be the default on Gentoo, but they’ve got a bug whereby they’ve got their ordering wrong: if you also have Bitstream Vera installed it gets picked up for Sans in preference. The folks in
#gentoo-desktopsay they’re going to fix that. (This is only relevant if you have thex11-base/xorg-x11meta package installed; it pulls inmedia-fonts/ttf-bitstream-vera. If you only have the lower levelx11-base/xorg-serverpackage installed you can manually installmedia-fonts/dejavu[only] and then the default aliases will be to DejaVu). Anyway, the real point of this article was about being aware that you can have control of your fonts, so all good.
Mon, 18 Aug 2008
java-gnome 4.0.8 released!
This blog post is an extract of the release note from the NEWS file which you can read online … or in the
sources,
of course!
java-gnome 4.0.8 (15 Aug 2008)
Cleanups and fixups.
This release is mostly to push out bug fixes and internal improvements, setting the stage for some major new feature development. We’ve also taken the opportunity to introduce a major change to the way we connect handlers for signals.
New coverage and continuing improvement
With thanks to new contributors Stefan Prelle and Andreas Kuehntopf we have a
number of small improvements to the TreeView/TreeModel APIs. As always, Widget and Window saw a bunch of work, with Window.ConfigureEvent now being available and a number of additional property setters and methods relating to window type. Widgets that scroll around a view of a broader underlying canvas have seen a fair bit of activity related to controlling that scrolling.
New features include refinements and new coverage of methods in a variety of lower level classes including that further support drawing operations. Bug fixes, debugging improvements, and defencive enhancements to our thread safety measures have also featured largely.
Signal naming change
The names of the inner interfaces used to specify the prototypes of the
methods which receive signal callbacks have changed to the pattern
Button.Clicked, this being more appropriate to Java type naming conventions
and providing better consistency between the signal name, the method to be
implemented to receive the callback, and the method that can be used to emit
this signal.
API compatibility to previous releases in the 4.0 series has been preserved.
The old signal interfaces and connect() methods are @deprecated.
Developers are encouraged to migrate quickly; new coverage will of course all
follow the new pattern. Making the transition is is easy, especially in an
IDE. Most of the people we’re aware of using the library have already ported
their code. Just turn assertions on to double check.
Build changes
java-gnome now defines C compiler flags like GTK_DISABLE_DEPRECATED to
ensure we are not linking against code that will be unavailable in GTK 3.0.
Many thanks are due to new contributor Kenneth Prugh for having done some
terrific grunt work to prune deprecated classes and methods from our .defs
data so that java-gnome compiles without using these APIs.
The build system internally now ensures that multiple runs don’t occur simultaneously, fixing a number of annoyances that cropped up when using IDEs which tend to like trying to frequently re-run the build even if a previous one hasn’t finished.
Documentation, examples, and testing
Our API documentation and the growing set of example code have all been updated to reflect the new signal interface names. Doing so forced us to review a wide swath of the documentation, and so along the way a huge number of minor improvements were made. Given how detailed our JavaDoc is, this sort of painstaking work really makes a genuine contribution to overall quality.
There has been steady growth in our test suite, which is great. When combined with the snapshots used to illustrate our documentation, the coverage level is substantial.
Error handling continues to improve. In the (hand written) public API wrapper
layer we do our best to catch misuses of the library before they get sent to
the native code. But that’s not always possible, and in 4.0.7 we introduced a
mechanism whereby GLib error messages get translated into Java Exceptions and
thrown. As of 4.0.8, in addition to ERROR and CRITICAL, we also throw
WARNINGs as Exceptions. Getting a stack trace this way has proved very
useful in helping developers track down where they are making mistakes in
using the library.
Conclusion
You can see the full changes accompanying a release by grabbing a copy of the sources and running:
$ bzr diff -r tag:v4.0.7..tag:v4.0.8
Because of the API changes to signal handling this release touches just about
every public class in the library and so isn’t quite as clean (as a summary)
as in previous releases — but it does show you everything that changed. :)
Looking ahead
Most of the contributors to java-gnome are working on branches that didn’t reach sufficient maturity to be merged in time for 4.0.8; that’s the way it goes sometimes. Major effort continues on implementing coverage of GTK’s powerful TextView/TextBuffer APIs, along with further drawing capabilities in Cairo and Pango. There have also been a surprising level of interest on other areas of the GNOME stack, with new contributors working on adding support to java-gnome for Nautilus, GStreamer, and even WebKit. Exciting stuff!
You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:
$ bzr checkout bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome
AfC
Fri, 01 Aug 2008
Swat!
Huge kudos are due to Cody Russell for having fixed one of the longest-standing bugs in GTK. 11 June 2001!
Cody only got involved in this issue recently, but he took the time to really dig into it, examine different hypothesises, and test like crazy. This was remarkable: it’s not every day you see someone wade into a bug that’s been open for over half a decade and with over a hundred comments, and then persist for over a year to get it solved.
Way to go!
AfC
Sun, 06 Jul 2008
Using Eclipse’s source code formatter from the command line
One of the many wonderful capabilities built into the side of Eclipse is a powerful code formatter. Adjusting source code to match one’s particular preferences is nothing terribly new; what makes Eclipse’s so excellent is the degree of configurability and the preview of the effect each setting will have.
An Eclipse project has several metadata files:
.project .classpath .settings/org.eclipse.cdt.core.prefs .settings/org.eclipse.jdt.core.prefs .settings/org.eclipse.jdt.ui.prefs
Which seems a bit of a mess, but whatever. The .settings/ stuff gets written down when you decide to tell a Project to have settings custom that override the user’s defaults. It goes without saying that for a collaboratively developed project like the Java bindings we have the Eclipse project metadata for compiler warnings tuned, the code formatter settings, how to invoke the build system etc all stored in Bazaar along side the real sources.
Anyway, this’s all fine and nice for people working on my software in Eclipse, but what about people wishing to use a different editor?
For a few versions now, Eclipse has exposed the the ability to run its code formatter from the command line; the work was done by Ben Konrath, someone I met a few years ago at a java-gnome hackfest. Since I use Eclipse I hadn’t needed to bother to figure out how to use whatever it was that he’d done. Since becoming maintainer of java-gnome, though, I’ve noticed a bit of unnecessary formatting churn in the contributions we receive. We do have a style guide, but it is secondary to the fact that Eclipse just normalizes the code every time I save something (nice). But for people not using Eclipse I thought I might have a look.
A bit of searching around turned up this blog post by one Peter Friese. It included two important points that are non-trivial. Firstly, that configuration the code formatter from command line will use can be controlled by the pointing it at the .settings/org.eclipse.jdt.ui.prefs file which in turn can be created by the “Enable project specific settings” described above (we’d already done that, so no problem, but how on earth would anyone have guessed this?). Secondly, the command line that one has to incant to make it go is pretty voodoo, so I’m pretty thankful that he described what he’d done.
In our environment, this works out to:
$ /opt/local/eclipse/eclipse -nosplash
-application org.eclipse.jdt.core.JavaCodeFormatter
-verbose
-config .settings/org.eclipse.jdt.core.prefs
src/ tests/ doc/examples/
Gah!
For a brief moment I wondered if it would be easier just to run the Java class in question. So I interrupted the process and had a look in the process table.
/usr/lib/jvm/sun-jdk-1.6/bin/java
-Dosgi.requiredJavaVersion=1.5
-client
-Xms40m
-Xmx256m
-jar /opt/local/eclipse/plugins org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
-os linux
-ws gtk
-arch x86
-launcher /opt/local/eclipse/eclipse
-name Eclipse
--launcher.library /opt/local/eclipse/plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.0.2.R331_v20071019/eclipse_1021.so
-startup /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
-exitdata 84001b
-application org.eclipse.jdt.core.JavaCodeFormatter
-verbose
-config .settings/org.eclipse.jdt.core.prefs
src/ tests/ doc/examples/
-vm /usr/bin/java
-vmargs
-Dosgi.requiredJavaVersion=1.5
-client
-Xms40m
-Xmx256m
-jar /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
Good Lord. What the hell is all that? I mean, I know Java environments tend to get a bit unwieldy because there is still no link-to-binary facility, and I know Eclipse is an OSGi framework and what not, but that is pathetic. I guess running the command as described at the top isn’t such a bad idea after all.
I’ve got an even better idea: how about a convenience target in our Makefile front end. Yup. So I’ve added a format target:
$ make format
(or
$ ECLIPSE=/opt/local/eclipse/eclipse make format
in my case). Running that before committing will do the trick quite nicely.
I agree with the bug mentioned by one Nicholas Irving in his post on the topic: it is somewhat asinine that this bit of Eclipse bumps the modification time of every file it formats; it would be nice if it only changed files in which it actually found something it needed to alter. Fortunately our internal build system is actually md5 hash (not Make) based, so it doesn’t hurt us.
AfC
Thu, 15 May 2008
A Bazaar branch of GTK
Converting a project from one version control system to another is always painful. Doing so is a rather momentous change to contemplate. With organizational & community inertia being what it is, not something easily rushed into. Worse is when you attempt to use a modern tool as a better front end for an old one.
One of the neat things about the 3rd generation distributed version control system projects (ie Bazaar, Git, or Mercurial) is their plugins which allow you to continue to work with a Subversion upstream project but use the modern tool as a front end locally. For those of us who have been very productively using DVCS for some time, this is a godsend.
Using one of these plugins to thence evaluate the usability or performance of one of a DVCS is not really the best approach one could take; none of them were built with using Subversion as a peer in mind, and lord knows Subversion is not built to store revisions with such complex relationships. Thus the idea has turned out to be a very challenging problem space for everyone but most have had their tools mature to the point where Subversion is the bottleneck. And despite my misgiving that starting out your experience with a 3rd generation DVCS by first using it to access some legacy project will not show your new shiny tool in its best light, the reality remains that most people need to get on with being productive, and doing a full blown conversion (or starting use with a brand new projects) are less likely to be your circumstance.
Having been using Bazaar (known best by the abbreviation bzr) for well over 18 months and overall being very happy with the experience, I would like to encourage other GNOME hackers to try it; likewise I have some patches I’d like to contribute to GTK so I thought I’d start by using bzr-svn to get a branch of GTK as a starting point (and to likewise give Bazaar’s Subversion interface a workout). So I created Bazaar branch of GTK.
Initial import was very slow
The first step was to create a Bazaar branch that represents the foreign Subversion repository. This was painful.
I originally attempted to do this while I was at the GTK Hackfest we had in Berlin; that turned out to be a disaster because (as one does at a conference or meeting) we were moving around every few hours connecting and disconnecting from the ‘net, and I was busy trying to do this from my laptop rather than a server somewhere. (Unfortunately bzr-svn relies on a memory fix that is not yet in the released version of Subversion, and not every distro has their Subversion patched with it). So that fell flat on its face.
I was able to restart the process a week or two later and leave it running overnight. Along the way I’d learned a technique for doing a few hundred revisions at a time (do bzr pull -r $i in a loop and increment the variable by a hundred or whatever each cycle) and so was able to cope with the need to disconnect periodically. Which is well, because it took 2 days to do the import.
While I was at first appalled by this, I really can’t blame Bazaar. The GTK codebase is, of course, rather large. First preserved commits were in 1997; there are over 15,000 revisions; a working tree (excluding VCS metadata) is 83 MB in size. The bigger problem however, is that Subversion is not fast at accessing historical data (the time taken to process revisions started at over 32 minutes per 100 but eventually dropped to about 9 minutes per 100 as it got to the most recent history; interesting).
Clearly, I could have picked an easier example for my experiments with bzr-svn, but it’s done now.
Usage is fine once branch created
I wasn’t entirely sure whether the DVCS property that branches are all peers would apply to one that had started life backed by a Subversion repository would hold, but things work just as they are supposed to. Creating a new branch to work on a feature with bzr branch, or use the technique of using a bzr checkout along with bzr switch to do change-in-place, both worked fine. So I’m all set.
The real point, though, was to encourage more GNOME hackers to give Bazaar a try. If everyone had to put up with a endless import process like mentioned above then no one would touch it. But now that the branch is created (and up to date as of yesterday or so), anyone can use it.
With Olav’s concurrence, I put my branch on in my GNOME directory, so you can branch from it as follows:
$ bzr init-repo --rich-root-pack gtk+ $ cd gtk+/ $ bzr branch http://www.gnome.org/~afcowie/bzr/gtk+/trunk/
You do the first step so that you can create a “shared repository” (in Bazaar parlance) which will allow that various branches under that directory will share all the revision data for the project. I need to warn you, though. Modern DVCS tools give you the full history of the project, but for a large project that can be pretty big. If you grab the above branch you’ll be downloading about 180 MB and it takes about 5 minutes. Yes 5 minutes is a long time, but no there’s no way around it* and in any case 180 MB is not unreasonable for such a mature project. Don’t be too worried if it seems like it is taking a bit to do the transfer. You only need to do it once, and look on the bright side. It took me 2 days. Just let it run.
Workin’
Personally, I always do my work in a working branch separate from my mirror of upstream, allowing me to easily compare against the last update of upstream that I have done. So:
$ bzr branch trunk working $ cd working/ $ ./autogen.sh $ make
and away we go. Branching takes like 3 seconds. Nice and fast, no problem.
I haven’t got a script running religiously updating my GTK branch; I do a bzr pull locally once every few days in my copy of 'trunk' and have been pushing that up to http://www.gnome.org/~afcowie/gtk+/trunk/. But now that you’ve got the branch, you don’t have to rely on me anymore; this whole distributed thing kicks in and you can do it yourself. Assuming you have a svn.gnome.org account, then you can update with:
$ cd ../trunk/ $ bzr pull --remember svn+ssh://username@svn.gnome.org/svn/gtk+/trunk
(use the --remember argument the first time to change where it updates from by default so you’re not relying on my branch anymore) and push with:
$ bzr push svn+ssh://username@svn.gnome.org/svn/gtk+/trunk
The very first time you update from Subversion bzr-svn will have to build some caches and whatnot locally, but it’ll be pretty fast from there on.
I’m ignoring the pushing and pulling of revisions between your mirror of ‘trunk’ and your local 'working' (or whatever) branch; I’m sure you can figure that part out yourself: but here’s a hint: do your work, test it, and commit it, all in 'working'; repeat; then use, say:
$ bzr diff -r ancestor:../trunk/
to consider what your current patch looks like, then shuttle it to 'trunk' with bzr pull or bzr merge (assuming you run it from 'trunk’ and whether you need to merge or not) and then bzr push to svn.gnome.org. Hope it works for you.
If you need help, poke your head into #bzr on FreeNode or write to their mailing list, or ping me and I’ll try and lend a hand. Make sure you’re using the latest releases; I doubt it will work otherwise and so recommend that you use bzr >= 1.4.0 and bzr-svn >= 0.4.10. Older versions do not contain critical bug fixes. I also encourage you to grab bzr-gtk >= 0.93.0 as the visualize command it adds:
$ bzr viz
is really amazing.
Good luck!
AfC
Update:
You need to have PycURL installed (it’s an optional dependency; used at runtime if detected). There is a bug that will prevent you doing the branch from me if you don’t. Debian and Ubuntu package python-pycurl; Gentoo USE=curl pulls in package dev-python/pycurl; Fedora package is also python-pycurl.
This very moment as the GNOME systems administration team is presently working through the consequences of the Debian SSH key vulnerability; they have for the moment very properly locked down all access to the services provided to GNOME hackers, and that knocks out access to the Subversion server meaning people can’t work with their source code properly. If there was ever a clearer demonstration on the inadequacy of old 1st generation centralized version control systems, I cannot think of it.
Mon, 05 May 2008
java-gnome 4.0.7 released!
This blog post is an extract of the release note from the NEWS file which you can read online … or in the
sources,
of course!
java-gnome 4.0.7 (30 Apr 2008)
Draw some.
In addition to the usual improvements to our coverage of the GNOME libraries, this release introduces preliminary coverage of the Cairo Graphics drawing library, along with the infrastructure to make it work within a GTK program.
Drawing with Cairo
The trusty Cairo context, traditionally declared as a variable named cr in
code, is mapped as class
Context. Various Cairo types
such as different surfaces and patterns are mapped as an abstract base class
(Surface, Pattern) along with various concrete subclasses (ImageSurface,
XlibSurface, and SolidPattern, RadialPattern, etc). Error checking is
implicit: the library status is checked internally after each operation and an
Exception thrown if there is a failure.
Thanks in particular to Carl Worth for having reviewed our API and having helped test our implementation.
New coverage and continuing improvement
The single option choice buttons in GTK are called RadioButtons and have now
been exposed. When using them you need to indicate the other buttons they are
sharing a mutually exclusive relationship with, and this is expressed by
adding them to a RadioButtonGroup.
The usual steady refinements to our coverage of the GtkTreeView API continue.
There’s a new DataColumn type for Stock icons, and TreeModelSort is now
implemented, along with minor changes to various other miscellaneous classes.
Considerable internal optimizations have been done, especially relating to
ensuring proper memory management, with notable refinements to make use of
“caller owns return” information available in the .defs data. This fixes a
number of bugs. Thanks to Vreixo Formoso for having driven these improvements.
Error handling has been improved for GLib based libraries as well. If an
ERROR or CRITICAL is emitted, our internals will trap this and throw an
exception instead, allowing the developer to see a Java stack trace leading
them to the point in their code where they caused the problem.
Internationalization support
java-gnome now has full support for the GNOME translation and localization
infrastructure, including the standard _("Hello") idiom for marking strings
for extraction and translation, but combined with some of the powerful support for positional parameters available from Java’s MessageFormat as well. There’s a fairly detailed explanation in the
Internationalization
utility class.
Build changes
Note that as was advertised as forthcoming some time ago, Java 1.5 is now the minimum language level required of your tool chain and Java virtual machine in order to build and use the java-gnome library.
Thanks to Colin Walters, Manu Mahajan, Thomas Girard, Rob Taylor, and Serkan Kaba for contributing improvements allowing the library to build in more environments and for their work on packages for their distributions.
The download page has updated instructions for getting either binary packages or checking out the source code.
Documentation, examples, and testing
Refinements to the API documentation continue across the board, notably improving consistency. A large number of javadoc warnings have also been cleaned up.
While not a full blown tutorial, the number of fully explained examples is
growing. There are examples for box packing and signal connection, presenting
tabular data, and basic drawing, among others. See the description page in the
doc/examples/ section.
This code, together with the not inconsiderable number of unit tests and the
code for generating snapshots of Widgets and Windows means that a large
portion of the public API is tested within the library itself. The number of
non-trivial applications making use of java-gnome is starting to grow, which
are likewise providing for ongoing validation of the codebase.
Summary
You can see the full changes accompanying a release by grabbing a copy of the sources and running:
$ bzr diff -r tag:v4.0.6..tag:v4.0.7
Looking ahead
It’s probably unwise to predict what will be in future releases. The challenge for anyone contributing is that they need to understand what something does, when to use it (and more to the point, when not to!), and be able to explain it to others. This needs neither prior experience developing with GNOME or guru level Java knowledge, but a certain willingness to dig into details is necessary.
That said, I imagine we’ll likely see further Cairo improvements as people start to use it in anger. It shouldn’t take too long until the bulk of the functionality needed for most uses is present in java-gnome. In particular, forthcoming coverage of the Pango text drawing library will round things out nicely.
There are a number of other major feature improvements we’d like to see in java-gnome. Conceptual and design work is ongoing on for bindings of GConf, GStreamer, and even support for applets. Within GTK, there have been a number of requests made for various things to be exposed, for example, the powerful GtkTextView / GtkTextBuffer text display and editing capability. Some of these have preliminary implementations; whether or not any given piece of work is acceptable in time for any particular future release will remain to be seen and depends on the willingness of clients to fund us to review and test such work.
In the mean time, people are happily using the library to develop rich user
interfaces, which is, of course, the whole point. We’re always pleased to
welcome new faces to the community around the project. If you want to learn
more, stop by #java-gnome and say hello!
You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:
$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome
AfC
Thu, 10 Apr 2008
Sun’s secret web server
For a while now I’d been looking for something to just run up an HTTP server to answer simple queries.
The Java Servlet APIs are comprehensive, and there are a number of very good implementations (we use Resin in highly loaded environments and it scales like a dream; we recommend it heartily to our clients). Nevertheless, sometimes you’re working on something small or ad-hoc, and you just want a simple web server that doesn’t involve application descriptors, XML, configuring an entire service instance, and everything else. Somewhat to everyone’s long standing annoyance (not to mention the derision of people who work in other languages), there is nothing in the Java standard library to just get on with this.
In their release of Java 1.6, however, Sun quietly included such a server. Apparently it had been around for some time powering internals of various other projects, but in 1.6 they exposed it as a public library, in package com.sun.net.httpserver, and it ships alongside the rest of their Java VM and standard library.
It’s got a rather nice API. You fire up a server and add one or more handlers:
addr = new InetSocketAddress("localhost", 8000);
server = HttpServer.create(addr, 10);
server.createContext("/eg", new ExampleHandler());
server.start();
the “contexts” are paths from root that are handled by different HttpHandler instances; implementing that interface is simply:
public void handle(HttpExchange t) throws IOException {
...
}
Once there, you’ve got a number of really straight forward methods on HttpHandler to get done what needs doing:
t.getProtocol();
t.getRequestMethod();
t.getRequestURI();
t.getRequestHeaders();
t.sendResponseHeaders(code, length);
for the metadata, and an InputStream and an OutputStream from:
t.getRequestBody();
t.getResponseBody();
respectively. You write the response to the later. Too easy. I did up a modified version of the example they have in the package description file as something that echos back the HTTP request as the response and it was quite straight forward. Java programmers can glance at EchoServer.java if they’re interested.
The catch
Nothing is ever simple. The bytecode for the com.sun.net.httpserver package is bundled in with the rt.jar that comprises the bulk of the standard library, but for whatever reason they didn’t include the JavaDoc for these classes in the same tree as the rest of the API documentation.¹ This is a huge pain, because once you’ve told Eclipse where to look for the docs for a given blob (in this case, for rt.jar) you can’t then tell it to look somewhere else for a sub part. And if you’re trying to develop something in a beautiful IDE like Eclipse, the idea of not having inline popup documentation and completion working properly is just a non-starter.
You can point Eclipse at either a JavaDoc tree, or at the raw sources and it’ll just extract the documentation on the fly. But since the src.zip shipping with Sun’s proprietary Java 1.6 didn’t include the classes for this web server code, that didn’t work either.
So that would have been the writing on the wall for Sun’s httpserver.
Open Java to the rescue
After chatting with some people in #gentoo-java and #classpath, we suddenly realized that the solution was at hand. Source code to Open Java is available.² And even though it’s still not yet a fully Libre replacement for Sun’s proprietary Java 1.6 (the size of the binary proprietary plugs you have to drop into it to make it work is staggering), it does contain most of the sources to the Java standard library. So we grabbed a copy of the openjdk 1.6 b08 code drop, and just pointed Eclipse at it as the sources for rt.jar for cases where it couldn’t find the JavaDoc when it was looking something up. And ta-da!
So, amidst considerable astonishment but great pleasure, here was an example of something someone had Open Sourced working in a synergistic way but in a way that they hadn’t quite expected. This wouldn’t have worked if Sun hadn’t Open Sourced (most of) Java; but because they have we were able to use those sources to solve a problem. Thanks to the heroes at Sun Microsystems who made this possible!
Eclipse users will have to adjust their filters so that com.sun.* isn’t excluded from Type completions and what not; While we’ve all loved our com.sun.* excludes, it blots out com.sun.net.httpserver. If you plan on using it, then I suggest switching to excluding com.sun.org.* and com.sun.xml.*, etc. A pain, but that’s what we get for Sun not having put this into the standard library.
Speaking of which, I doubt we’ll ever see this as a part of the “Java” (wouldn’t java.net.httpserver just do the trick?) because the marketing side of Sun has a fairly strong fixation on “Servlets are what you use for everything”. Oh well. It would be a really pleasant surprise, though. Here’s hoping.
But not really
We would never use this in something we were distributing to anyone outside; after all, they might be using some other VM and com.sun.net.httpserver isn’t available as a standalone library. And that’s not to mention scalability; if you need something bulletproof then use the Resin servlet runner. But if you need a quick-and-dirty but easy-to-use HTTP server and you happen to have Sun Java or the future tense Open Java available, then this should do the trick for you. And if you’re feeling like whipping up something of your own, this really does have a nice API (somewhat a rarity in the Java landscape) and so this might be a good starting point for you. Either way, have a look.
AfC
¹ As mentioned, the API docs for com.net.sun.httpserver are not in the usual location at docs/api/ along with the rest of the standard library but at docs/jre/api/net/httpserver/spec/ of all places. Who knows what they were smoking that day.
² I am very tired of apologizing to people who have just installed “Java” on their Linux boxes that they have to download something else. I’ve given up trying to explain the difference between a JRE and a JDK to people; even after I do people respond with “who cares; I just want a working Java that I can use to build your software.” Sun having attempted to brand their Libre implementation of Java as “Open JDK” just perpetuates this nonsense. Not including the compiler classes in something that is already 18 MB big is stupid, and it’s not like the desktop VM is what you use on a resource limited embedded device. Open Java works just fine as a name, thank you very much, and it is (almost) a Free Java. Almost.
Sat, 05 Apr 2008
Cairo drawing from java-gnome
We’ve added support for the amazing Cairo Graphics library to the Java bindings for GTK and GNOME.
Adding Cairo is a feature we’ve been working on for about 6 months now, and so I’m pretty pleased that during the 4.0.7 development cycle we’ve been able to land it. Cairo is a huge library, of course, but we’ve put enough coverage in place to ensure that things are working.
Cairo has lots of convenience functions and tons of obscure uses; no surprise (and no apology) that there’s still lots that will need doing. If you want to help make sure it has what you need, then
grab ‘mainline’ and see org.freedesktop.cairo.Context and ExampleCairoDrawingInExposeEvent.
Huge thanks go out to Behdad Esfahbod and Carl Worth; Behdad was really critical in explaining some basic Cairo concepts to get me started when we were working together at the GNOME Summit back in October in Boston, and at the GTK hackfest in Berlin in March, Carl Worth checked our preliminary APIs and helped sort me out as we were working our way through to create some examples. Awesome!
java-gnome 4.0.7 should be out in the next week or two, depending on how long it takes to finish up testing, QA, and to do the release engineering.
AfC
Fri, 04 Apr 2008
Upgraded!
|
|
|
| #gnome-hackers | #gentoo-desktop |
The upgrade to GNOME 2.22 went really smoothly. Well done to the Gentoo Linux desktop team for their heroic efforts bumping so many .ebuilds, and congratulations to all the GNOME hackers, testers and translators around the world on what continues to be superb software!
AfC
Mon, 25 Feb 2008
Get Some
The other day I was wading through Ulrich Drepper’s paper about how memory works these days and what programmers should do as a result.
My wife asked me what I was reading. I showed her the title and when she saw it concerned computer memory, she looked at me with a totally straight face and summed up the 114 page paper by answering “Yes, you should get some”.
Like most of Ulrich’s writing, this paper is immense: long on detail and well presented. I must admit that I agreed with many others that after having read the work I was left wondering what action, in practical terms, I should be taking as a result. Clearly most of this knowledge is very important for Kernel hackers and for those doing low level systems programming, but since I do higher level work (ie advising people with large enterprise systems), it is hard to asses the choices I might be making in the terms of what Ulrich describes. I can only hope that the people who work on the parts of the low level libraries that run next to the metal (such as drivers, GLib, and especially the VMs like the Java and Python runtimes) are aware of these issues and making design choices based on these insights.
AfC
Tue, 12 Feb 2008
java-gnome 4.0.6 released!
This blog post is the release note from the NEWS file which you can read
online … or in the
sources,
of course!
java-gnome 4.0.6 (12 Feb 2008)
Finding the missing methods.
Most of our effort recently has simply been fleshing out areas of the public API. The focus for this work as been getting the coverage needed to allow us to port some of our in-house applications to java-gnome 4.0. It’s not especially glamorous — if anything it has been tedious as hell — but the result has been a large body of improvements to java-gnome as a whole which we’re pleased to release as java-gnome 4.0.6
The bulk of this development took place on the ‘missing’ branch, so named
because that’s where I was working on what was missing :).
Continuing Improvement
Notable public changes include coverage additions to enable key stroke and mouse button handling
Rather than exposing the int keyvals that bubble up out of the X server, we
have wrapped these as constants of type
Keyval (thereby being consistent with the
rest of java-gnome in our working to the strengths of Java as a strongly-typed
language; MouseButton was created for the same reason, helping developers
understand just what on earth mouse button
“1” is, anyway). Along with
ModifierType, this gives enough to
deal with the KEY_PRESS_EVENT and KEY_RELEASE_EVENT signals when the
developer wishes to deal with key strokes.
We’ve finally gotten around to providing proper coverage of the box packing
model which underlies every aspect of how GTK presents user interfaces. To
understand the size-request/size-allocation process, you might start with
Widget’s
setSizeRequest().
We’ve also added coverage for SizeGroup, which, when used in concert with nested VBoxes and HBoxes, can work wonderful magic and is often far better than messing around with Table when doing complex layouts.
After dithering for several releases, we’ve settled on how we’re going to deal with ComboBox and family. The underlying GtkComboBox presents something of a nightmare as it is really two classes in one with more-or-less incompatible APIs. So, not surprisingly, we’ve presented it as two sets of classes, with the text-only convenience API spliced out of ComboBox and ComboBoxEntry into TextComboBox and TextComboBoxEntry respectively.
We’ve added a few new features in our coverage of GTK’s TreeView / TreeModel API, and many other classes involved have also seen improvements. The persistent reference to a row provided by TreeRowReference is now available as is model type TreeModelFilter.
Support for the actual filtering in TreeModelFilter is notable for having been quite tricky. The underlying C library use a function pointers rather than a GObject signal emission, and we don’t have any mechanism to handle that. We do, however, have a fantastic capability to marshal signals, so we dealt with the problem by creating a custom signal and then passing a function which emits it when the TreeModelFilter wants to ask the developer whether to include a row or not.
The new classes include support for TreeModel columns storing long data as
well as setting properties of that type.
It should also be noted that most of the methods taking a TreeViewColumn have been converted to taking an argument of type CellLayout (an interface implemented by TreeViewColumn). This has no change to how you use our TreeView API, but was necessary to support ComboBox properly.
Finally lots and lots of minor additions to both public APIs and internals deeper down in the GDK part of the toolkit.
As ever, you can see the full changes accompanying a release by grabbing a copy of the sources and running:
$ bzr diff -r tag:v4.0.5..tag:v4.0.6
Documentation
We’ve always had HTML JavaDoc for the current stable release at
doc/api/ on the java-gnome website.
We’re going to change that a bit, though. As fixes to the explanatory
documentation happen quite frequently to classes and methods all over the
place, we’ve decided to generate the JavaDoc from ‘mainline’ periodically and
upload that instead. This means that there will, of course, be descriptions of
some methods which aren’t yet available in a released version of the library,
but they will clearly identifiable by virtue of having a @since tag showing
a version number greater than the most recent release. We’ll revert back to
more traditional behaviour once we hit 4.2.0, but in the mean time we can give
people access to the best information we can online.
The idea of having up-to-date illustrations of the various Widgets has proved popular, and we’ve continued to update the suite of snapshots. Doing that is also tedious, but it does provide a good opportunity to test APIs we are exposing especially where unit tests are less suitable.
Looking ahead
Almost as complex as the TreeView / TreeModel API are GTK’s powerful TextView / TextModel classes, collectively a Widget used to display and edit large text documents. Working out the java-gnome coverage for TextView will take a fair bit of consideration, but TreeView provides a road map, and, as with the coverage in 4.0.5 and 4.0.6 (which was driven largely by existing applications we were porting), we have some significant uses of GtkTextView which will guide us on our way.
The next release will also feature significant work outside of GTK; we should
be in a position to merge our coverage of the excellent Cairo drawing library
soon, and likewise we have tentative work in place letting people store
configuration and settings data in GConf. Both the ‘cairo’ and ‘gconf’
branches need more QA and documentation work, but they’re looking good and will
definitely be featured in java-gnome 4.0.7.
You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:
$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome
Well, now that I have this out of the way, I can finally stop worrying about having so much unreleased code and get back to the business of doing what I do for a living.
On a personal note, I mentioned that much of this coverage was driven by one of our legacy apps. About half way though this cycle, however, we decided we would be abandoning that program. Needless to say, it was more than slightly depressing to be putting so much work into writing new coverage for java-gnome in order to port such a large codebase in the certain knowledge that the application was not going to be resurrected after all, especially as we have not yet managed to raise any revenue to offset our costs in taking on the Java bindings for GNOME.
Cognitive Dissonance.
Nevertheless, I kept at it largely because getting that app to run again represented a good test of the quality and comprehensiveness of the Java bindings. It needed doing, and that has ultimately been the motivation behind all our work re-engineering java-gnome. I gotta say it was wonderful seeing that program come back to life bit by bit as the errors slowly fell away as missing classes and methods were added to the library. That monster running again after almost two years represents the 4.0 version of java-gnome having surpassed the level of coverage that we used from the old 2.x version, and that’s something to be very positive about indeed.
Of course, now all I have to do is to continue to resist the temptation to go back to working on that beast. :) No, better things to be about.
AfC
Fri, 25 Jan 2008
The arcane secrets of hash-bang
I’ve been working for a while now prototyping various different domain specific approaches to modelling software configuration information. Most of these involve putting the configuration data in the body of an executable script. To that end, I’ve been digging in to how interpreted scripts actually work on Linux and other Unix-like operating systems work.
#! interpreter
Anyone who has ever written a Shell script, Perl program, or Python program is familiar with #! lines:
#! /bin/sh
#
# A program to do something very special.
#
echo "Hello World"
and
#! /usr/bin/perl
#
# Another program to do something very special.
#
while (<>) {
print "Hello World\n";
}
etc. The program mentioned after the magic #! characters is the program that will interpret the script. There are many gotchas with that (notably portability concerns owing to the fact that some idiotic flavours of Unix don’t put Perl in /usr/bin, that sort of thing)
I always figured that the script file got piped by the OS into the interpreter on stdin. A reasonable guess given the way that most of the tools we use work, but it turns out it is nothing of the sort. Every time I tried to write an interpreter (in C) I got stuck.
What threw me off was that cat works as an “interpreter”:
#! /bin/cat
This is the script body
which will happily be sent
to stdout
If you put that in a file called script and run that from your terminal, then sure enough,
$ ./script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$
which of course is exactly what would happen if you did:
$ cat < script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$
and that’s what totally had me on the wrong track. I figured that the interpreter on the #! line was being fed the body of the executing file on stdin. Nope.
Seek and ye shall find, sort of
I thought that I might be able to find out what was going on by reading the code of an interpreter program. I started by looking at the sources for /sbin/runscript (which is on the #! line for all of Gentoo Linux’s RC scripts), expecting that to be quite simple. It was simple. Too simple. All it does is some environment filtering and then fires off bash to run /sbin/runscript.sh (in other words, it’s largely a workaround for the fact that you can’t actually make a shell script itself an interpreter). Nothing at all in there about reading stdin. So then I looked at the source code for Perl (Whoa, there’s a beast). Nothing obvious there either. Lots of stuff about reading from stdin but nothing about that being the origin of the script to be executed. A lot of messing around with argument signatures though.
#! is not exactly the easiest term to put into a search engine. I did, however, happen to know that one of the ways #! is pronounced is “hash bang” (being two common names for the respective characters, though lots of old suspender-snapping sandal-wearing bearded Unix freaks would, I’m sure, tell you with great passion that it has to be pronounced some other way). Searching on “hash bang” brought up lots of arcana, including something that lead me to an obscure article by one Andries Brouwer on the parameter signature at invocation wherein I discovered that there is a calling convention for how arguments are passed to the interpreter program being invoked.
It’s a bit complicated, since you can have command line arguments for both the interpreter and for the script being run. It goes something like this. Let’s say you have an script that begins with the following:
#! /path/to/program -v -d
(with -v perhaps meaning “verbose” and -d perhaps meaning “debug”) and you have it in a file called ./script, then running it will actually cause program to execute. The trick is, with what arguments? Check this out. If you do:
$ ./script -p -r
(with -r and -p, for the sake of illustration, perhaps having the same meanings as cp, that is “preserve” and “recursive” respectively) then when our interpreter program is executed, it will be invoked with the following arguments:
/path/to/program -v -d ./script -p -r
the mapping is a bit obscure. It’s actually:
| argv0 | argi | argn | args… | |
argv[0] | argv[1] | argv[2] | argv[3] | argv[4] |
/path/to/program | -v -d | ./script | -p | -r |
(to use the terminology in the above link). This all shed a little light on what I’d seen in runscript.c and perl.c, but still not a single mention of the script being fed in on stdin. So I pondered that for a while longer, until finally the light bulb went off.
Eureaka & Company
The reason I couldn’t find any mention of ./script being fed in on stdin is because is is not fed in on stdin. You don’t need it to be: you’ve got the name of the script file fed to you in your interpreter’s argument list (from the above example, it’s in argv[2], one happy looking string containing “./script”). So read it already!
FILE* body;
body = fopen(argv[2], "r");
...
and ta da, that’s where you get your script’s program body from. Now you can at last get on with parsing your script, and running it.
Most big programs spend lots of time munging the argument list, dealing with the fact that argv[1] could be full of all sorts of stuff jammed into, or nothing, etc. The whole thing goes from elegant to clumsy when you discover that if there are no arguments to the interpreter on the #! line then the script file will be in argv[1], and it goes to nightmare level when you look at the list of variations in behaviour across different operating systems, compiled by one Sven Mascheck. Nonetheless, the interpreter is your program, and presumably you can recognize, parse, and skip over zero, one or more arguments to yourself before deciding you’ve reached the name of the script. Judicious use of argv++; argc--; is your friend here, apparently.
Anyway, this all explains why my cat example was working but my own efforts were not. cat is not reading data being fed to it on stdin (which is cat’s behaviour if you run it without any arguments), it’s being executed with an argument, namely ./script as argv[1], ie exactly the same as:
$ cat ./script
#! /bin/cat
This is the script body
which will happily be sent
to stdout
$
But now that I know what’s going on, I can write my own interpreter.c:
#include <stdio.h>
#define LEN 128
int main(int argc, char** argv) {
char buf[LEN];
FILE* body;
body = fopen(argv[1], "r");
while (fgets(buf, LEN, body) != NULL) {
printf("%s", buf);
}
fclose(body);
return 0;
}
and if I compile that to interpreter, then I can write a domain specific language that is interpreted by this program, say:
#! ./interpreter
This is a test of the Emergency Broadcast System
in a file called script, then, at last,
$ ./script
This is a test of the Emergency Broadcast System
$
Yeay!
Ok, so that’s cat, but cat is the Hello World of input/output. :) The real point is that running script caused interpreter to be executed, and interpreter got at the body of the script that was “run”, and was able to do something with it. Onwards at last.
AfC
Comments
Julio Merino Vidal wrote in suggesting:
Take a look at NetBSD’s
script(7)manual page for some more details
about how that is supposed to work and some things you must consider
for portability (such as being able to feed a single argument to the
interpreter through the#!line, or the maximum length of it).
Updates
Quite by accident, I just came across the related information for Linux; see the execve(2) man page for
a succinct treatment of both exec()‘ing in general, and the topic of interpreting scripts.
Thu, 24 Jan 2008
Free Range Software
Jon Hall writes of his experience in a restaurant talking with its owner about “Free Range Eggs”:
“… but we have to charge money for our eggs. People who don’t acknowledge that just do not want to understand the term ‘Free Range’ for what it really means … better eggs, and changing the term will not help that.”
The fact that the discussion started because of maddog’s suggestion that maybe they should be called “open range eggs” to eliminate the confusion is not the point (“that’s silly” the owner said, “everyone calls them free range eggs”). The term we use, Free Software, has a bigger problem. Consider the difference between:
- Free Range Eggs
- Free Eggs
Clearly, the term is “Free Range”, and applies as an adjective to “Eggs”, whereas the latter really does mean “free eggs”. Now consider this:
- Free Software
There’s something missing, and so the term free gets connotated as having to do with price.
No, I’m not about to say that we should call it Free Range Software [and while “let it run free!” is a lovely metaphor, I don’t quite think we want to be associating our work with chicken farming :)]. Perhaps someone will come up with an intermediate word that will do the trick. To be honest, though I’ve pretty much given up on the term Free Software; I write Open Source software, and the cause I advocate is Software Freedom.
And when people still stare at you blankly, you can say “you know, like Linux” and watch as comprehension dawns. To be sure, they probably still don’t get it, but chances are you’ve got more important things to talk about, and getting on with it is going to do you — and logiciel libre — a lot more good than getting lost because of the insufficiencies of the English language.
AfC
Thanks to Atul Chitnis for having passed on the link.
Wed, 16 Jan 2008
Reusing Experience
I came across an interesting comment yesterday:
The documentation doesn’t help you much though. First, it is not sufficient and second (and important) you do not learn much from the documentation.
Thankfully, you have the source code and I really appreciate the source code of Eclipse is open. That’s because only from source you can learn as much as necessary. And this fact leads me to think more and more that open source is not about “reusing software” (commercial companies do that as well) but about “reusing experience” which is hidden inside the source.
I have the strong belief that people who write the software are more important than the software itself and by looking to the source code you can gain at least part of experience of people who wrote it. That’s the power of open source, that’s “reusing experience” concept at work!
This observation was written a few years ago by one Alexander Dymo who was expressing his amazement at the whole Software Freedom thing, especially the hands-on side of it. And as for his “people are more important than software”, well, hey; I couldn’t agree more. Enlightened organizations that want to preserve their strategic advantage understand that the people who are capable of working with such code are their competitive advantage. They are the ones who can reuse experience and leverage the power of the most wide reaching and enabling social phenomenon we have ever seen: the global openness movement.
Towards a technical definition of Open Source
I’m on a bit of a kick at the moment working to elucidate a practical technical definition of Open Source (ie, complementary to the necessary legal foundation which enables Free Software and the requisite social interaction which is at the heart of global Open Source communities). Comments like the one above are helping refine my thoughts on the topic: yes the interaction of licence and copyright law gives us structure, and yes communication between people distributed the world over is the genesis of the sense of community, belonging, and accomplishment which is the rich social fabric within which our software development takes place, but there is also a pragmatic aspect: can you actually work with the source code, get right into it, experiment with it, break it, and do crazy things with it?
That the four GNU freedoms stipulate that this must be “permitted” doesn’t really change the fact that there are practical prerequisites. Can you easily get the sources under development? Do those sources actually build? Is there a clear mechanism for contributing source back to the project and are they actively facilitating such contributions?
The source tarball as primary release artifact
The biggest give-away is whether the primary release artifact is source or an opaque binary.¹ Especially in the Java world but elsewhere as well, there is a surprising amount of activity for which, despite the fact that it may legally be Free Software and its loud proclamations to being Open Source, it remains clear that they just don’t get it: there are a huge number of projects and products which only do their releases in binary form. Bad sign when you start calling it a “product”, I think. In a frustratingly large number of cases, if you try to actually work with their code you will discover that it doesn’t actually compile! In extreme, there are statements like:
We distribute source but never claimed that you can build it out of the box. We don’t have time for such things.
I didn’t believe it when I saw it, but one of the hackers of a supposedly Open Source project actually said that in response to a bug report. Astonishing.
Being able to duplicate the result is a rigour that goes far beyond software; indeed it has been the bedrock of science — and human progress — since the dawn of the age of reason. Back to the present day, it has suddenly become obvious to me that the fundamental technical difference between proprietary Unix from the commercial vendors (not to mention proprietary operating systems from Microsoft and Apple) and Linux is that in our world (taken to mean the entire continuum of FOSS communities) the primary release artifact for all upstream projects is a source release (these days it’s typically in a .tar.bz2, but whether it is that or .zip or whatever else isn’t terribly relevant) that you can build. Not a tarball full of already built .class files and .jar files. Not a “source .zip” for Eclipse. Not a self-extractable full of .exes and .dlls. No. A source release is source code that someone else can build, right out of the tarball. THAT, ladies and gentlemen, is the technical definition of Open Source.
I’ve started to realize that this area is a big stumbling block. People I collaborate with in (upstream) global projects like GNOME, Free Java, Bazaar and elsewhere take the efficacy and primacy of source releases entirely for granted. But a number of clients that my firm is working with to enable Open Source often just don’t get why they should be doing source releases — and resist it.
Whether you buy into Software Freedom or not is a different topic (and your decision, not mine to impose on you), but if you’ve got a software project for which you’ve decided to free the source, you want it to be successful, right? Success is a project that which inspires user enthusiasm, which the major distributions can package and ship without hassle, and around which a vibrant community of developers grows, ultimately fostering new contributions. There are a lot of steps along the way, but a buildable source release is, in our view, the technical bar you’ve got to reach. Otherwise you’re not making releases of the software, you’re abandoning it. And your users.
AfC
¹Note that this is different from a Linux or Unix distro shipping binary packages — no matter if it was Fedora, Debian, Gentoo, FreeBSD, OpenSolaris or anyone else, they should still have been able to build their package from source. It matters little whether that compilation took place in a build farm somewhere or on the user’s desktop (which is what happens in the “binary” distros when someone wants to work on the package or the project itself) — the key point remains that we can build the software if we want to, and are not forced into relying on someone else’s proprietary (and more to the point, unavailable) tooling. If you can’t build it, it’s not Open Source.
Fri, 11 Jan 2008
Fascinating thread: FOSS Quality Control
A long-time critic of things Open Source showed up on the Classpath project’s mailing list and asked some rather provoking questions in a thread titled “Quality control and FOSS rant”. He at least ended with: “I suppose this is more of a troll than a criticism, sorry about that.”
Despite the flame bait, the thread contained some surprisingly insightful replies. It’s always great to hear some of the top software developers in the world noting their motivations and why they believe what they do works.
From Roman Kennke:
Both approaches (closed and open) apparently tend to produce relatively high quality code (or really crappy code, happens in both camps), where with the closed approach the developers (or vendors) have to take over 100% responsibility (because the end user has no way to interact with the development), which usually makes things very formal and slow, where the open approach relies very much on the end users reporting problems. In most active projects these are fixed really quickly, giving both the developers and the end users a warm fuzzy feeling ;-)
From Andrew John Hughes
There’s a lot to be said for feedback and interaction with your users that’s often overlooked. All the ideas of complicated quality control processes in the world is not going to make a user feel as loved as seeing someone responding quickly to their bug and fixing it in a short space of time.
From Mark Wielaard, a remark on the complex administrative process used by the project to review contributed code:
We do have a flow chart that people have to follow when contributing… It is all very formal really: http://gnu.wildebeest.org/~mark/patch.png
and from Archie Cobbs, a reminder about the track record of a certain formerly proprietary process on solving bug desperately desired by their user community:
The number #1 voted bug in their bug database has been unfixed for over 5 YEARS!
The comments on that bug make for hilarious reading, but the bigger point is this: the identity of the people making the decisions about the relevance of the issue are hidden. That sort of thing doesn’t inspire much hope for people on the outside. It’s not like we’re talking about national security or the future of western democracy; it’s a bug report that turned into a feature request for a piece of software that many, many people depend on! No one likes to be fed the line that their problem is so Top Secret that they won’t be told when (or even if) the problem will be addressed. The cloak of anonymity strikes again.
Fascinating thread.
AfC
Mon, 26 Nov 2007
java-gnome 4.0.5 released!
This blog post is the release note from the NEWS file which you can read
online … or in the
sources,
of course!
java-gnome 4.0.5 (26 Nov 2007)
TreeView is here!
It’s always a great feeling when you bag a milestone, and with this release we have reached a major goal on our way to having outstanding Java bindings for the GNOME platform: coverage of GTK’s powerful yet complex TreeView & TreeModel API.
TreeView
TreeViews are a central part of almost every application. GUIs use lists for all sorts of things, and so a significant goal was to make coding TreeViews and their backing TreeModels as straight forward as possible.
The most challenging and complex part was to design the Java side API, which was no small matter. As a native library, the GtkTreeView API is complex and very much written with programming in the C language in mind, and as such our algorithmic mapping of the underlying libraries into Java doesn’t entirely fit. Long experience with the TreeViews in the previous bindings had made it clear just how nasty to use the API could be, and so the hardest part of the work was to come up with a mapping and a usage pattern that would be both faithful to GTK and be sensible to use.
The other significant challenge was to document the work effectively. Our Java side API documentation is a major feature of java-gnome, and merely exposing classes and methods is not sufficient; they need to be clearly explained in our JavaDoc as well.
along with numerous test cases in our unit test suite, and several comprehensively worked examples.
This was a monster patch, and the culmination of not just three months direct effort, but also where we’ve been heading since we first started the re-engineering of Java bindings for GNOME. Although largely written by Andrew Cowie, a significant contribution was made by Srichand Pendyala who not only exhaustively evaluated the design but also threw in some serious chunks of code. The work benefited from comprehensive input from Peter Miller on the modelling and design, and the comments of Bryan Clark, Owen Taylor, and Hanna Wallach were all really positive and helped us know that we’d gone in the right direction. Finally, thanks to Behdad Esfahbod and the GNOME Foundation who made it possible for us to meet in Boston at the GNOME Summit and so accomplish much of the final pulling together of this branch.
Continuing Improvement
Meanwhile, steady work continues on to the fundamental base classes, with a whack of additional signals and methods on Widget and especially Window, along with expansion of coverage in numerous other classes.
We’ve begun to get the basics of image handling in place,
One nice piece of contributed work came from Vreixo Formoso and Thomas Schmitz
with coverage of the Dialog Window functionality in GTK. It took a bit of
doing to map the int response codes used by GTK into something suitably
strongly-typed, but all good.
Finally, minor improvements to all sorts of stuff, notably from new contributor Mario Torre. Awesome!
For further details you can always grab a copy of the sources and run
$ bzr diff -r tag:v4.0.4..tag:v4.0.5
to see the complete code delta.
Screenshots
For fun we built in a capability to
create demonstrations to be captured as screenshots to illustrate various
things. It doesn’t get more basic than the example on the
Window documentation page, but it’s a nice touch. :) We’ve
screenshots for a number of Dialog classes and one for the TreeView page. I
imagine we’ll build up a nice library of images in the next few months (yes,
dear contributors, you can add snapshots to the list of things we’ll be
expecting along with well written documentation and unit tests when submitting
additions to the public API).
Building and requirements
The library now depends on GTK >= 2.12. Those packaging java-gnome for their distributions please take note.
Looking ahead
Continuing to expand the coverage levels in the classes already exposed will continue to dominate our attention; there’s still a long way to go but we’re pleased with the progress we’ve made so far; you can definitely build real applications with java-gnome now.
The next release also ought to include preliminary coverage of GConf and Cairo. Doing each justice will again take a serious amount of work, but will continue to grow the fun things you can do with java-gnome.
You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:
$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome
This release marks a particularly satisfying milestone for me. I have three+ years of work that’s been waiting to be ported to java-gnome 4.0, but I’ve needed a new TreeView binding in place before I could do so. So this is a big deal for me. At last I can start thinking about getting back to actually hacking on the applications I actually want to be working on, and not spending so much of time on the underlying bindings themselves. There’s still a lot that needs doing on java-gnome, but it’s getting closer.
AfC
Wed, 21 Nov 2007
Automatically rendering and capturing screenshots
One of the things I’ve often noted in the API documentation for the various GNOME libraries is that the examle screenshots always seem out of date. I think this is mostly because the look and feel we all interpret as “modern” is just that which happens to be shipping with the current version of the GNOME Desktop we’re using — and that default changes fairly frequently.
As I’ve been working on the GTK coverage in java-gnome, I’ve had in the back of my mind that it would be really cool if we had illustrations of the various Widgets and whatnot under discussion. More than cool, actually; a lot of the people developing with java-gnome are experienced programmers but new to Linux and the GNOME Desktop, so it’s probably pretty important that we help them along in learning which controls on their desktop correspond to which Widget classes. Capturing such screenshots manually is a bit of a pain, though, and even if we did that we’d still end up with them looking out of date within a year or so.
So I had a bit of fun last week and knocked something together that would automatically take screenshots of individual Windows or fragments thereof such that we could use them in the java-gnome API documentation.
Snapshots
First I cooked up a scheme to make it easy to write the little programlets that draw the Windows and Widgets (I used the term Snapshot to refer to these; nothing as elegant as the JUnit API, but hopefully it’ll improve over time). Then I wrote a harness to instantiate the sequence of Snapshot programs and to capture a screenshot of each one in turn.
GNOME ships with a tool in gnome-utils called gnome-screenshot which is handy, but a bit too focused on being a user driven program and is not available as a library in any event. So we pulled out the parts that actually capture the image (my Lord, talk about ninja voodoo X programming — I remember working directly with Xlib back in the late 80s;I am so glad we don’t have to do that anymore) and made it accessible programmatically. So all Harness has to do is something like:
window = snapshot[i].getWindow();
window.showAll();
window.present();
pixbuf = Screenshot.capture();
and then save the Pixbuf to disk in the right place so that <img> tags in our JavaDoc can pick them up. I did up examples for Window, Button, a few MessageDialogs, and TreeView. They’ll be in java-gnome 4.0.5 which will be out within the next week or so.
Off screen
If you did this as a zillion Windows popping up in front of you, though, you’d probably end up capturing the wrong thing if you accidentally changed apps or something. So instead I tried to figure out how to do it off screen. The solution I came up with was using the virtual X server that ships with xorg, Xvfb. By doing:
$ Xvfb :1 -ac -screen 0 640x480x24 -wr
we have a new X server running on :1, and then by passing --display=:1 to the Gtk.init() call used to initialize GTK, the Windows rendered by the program will be sent to the virtual server, not the desktop running on :0, and then we can take our screenshots there.
I’ve had this in mind for a while as a way to do unit tests that absolutely need to have the Window mapped to the screen. Unlike the unit test scenario, though, there’s a catch: you want the Windows to look right, ie just as they would if they were on a user’s desktop. That doesn’t happen in a naked X server though; you need a window manager running and also the magic program that makes the GTK themes according to your current preferences:
$ metacity --display=:1
$ gnome-settings-daemon --display=:1
otherwise you get that rather ugly default built-in GTK fallback theme and no window decorations, which would defeat the whole point of the exercise. So hook that all up to be exec()d when the Harness program starts and everything works great!
I’ll have to remember to switch themes back to Clearlooks before generating the screenshots for our next release… :)
AfC
Wed, 26 Sep 2007
java-gnome 4.0.4 released!
This blog post is the release note from the NEWS file which you can read
online … or in the
sources,
of course!
java-gnome 4.0.4 (26 Sep 2007)
Coverage increasing!
Most of our work continues to be on infrastructure and architecture, improving the code generator that outputs the translation Java layer and JNI C layer which allow bindings hackers to reach the underlying native libraries. Nevertheless, there have been a number of publicly visible improvements across the board, so we wanted to push out a release highlighting these contributions!
Documentation improvements
Continuing our effort to have extensive developer friendly tutorial style documentation, there have been major additions to a number of existing classes. Of particular note is the Window class, containing the various utility methods used to ask the window manager to do things for you (we’ve also started exposing some of the deeper parts of the GTK toolkit, though only a few things that were immediately related to window management).
While the topic of thread safety was discussed at considerable length in the last release, we have added some of the more relevant information to the code documentation to reinforce its importance.
New coverage
Numerous people have been hard at work developing new coverage. The standards
for accepting patches which expose public API are high, so it’s awesome to see
bundles accepted for being merged to mainline from new contributors Thomas
Schmitz, Wouter Bolsterlee, and Nat Pryce.
The infrastructure for a number of areas important to supporting applications including Menus, Toolbars, and Actions has been put in place.
A number of Container related Widgets have been added, though coverage is preliminary. There have, of course, also been a number of minor improvements in other existing classes, miscellaneous constants and wrappers around the stock item identifiers.
Internals
Vreixo Formoso carried out an important refactoring to the type database and Generator family of classes in the code generator, with the result that more of the array passing and out-parameter cases are now being handled correctly. This kind of work is usually thankless and taken for granted, but it’s hugely appreciated!
The real gains are in internal quality. A number of serious bugs and
limitations have been overcome (Glade is working again, for example). The
generated code now guards against improper use (you can’t pass a null pointer
unless it’s allowed by the underlying library). Related to this is handling of
“GError” — Java side, bindings hackers will get GlibException which they
can then re-throw as an appropriate Java Exception, say FileNotFoundException
in the case of not being able to open a file.
This all goes along with numerous build system fixes by Srichand Pendyala to make for an increasingly robust project. Thanks guys!
Looking ahead
As mentioned above, we have mostly been focused on areas other than public API, but it is expanding steadily. The hard work on infrastructure, however, is starting to pay off, and the next release should include coverage of TreeView, GTK’s powerful but complex list Widget.
You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:
$ bzr branch bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome
Note switch to bzr:// protocol.
AfC
Bazaar 0.91 released!
Congratulations to the Bazaar team on the release of 0.91!
Stop using dumb protocols
Probably long overdue, but we set a bzr serve daemon running on our R&D site.
Hackers who had write access to create their own branches there used the bzr server automatically since they `bzr push bzr+ssh://... and can also use it for branching and pulling. Most other people, however, were doing an initial checkout via http:// because that’s the URL we published. Which wasn’t very nice of us: initial checkout time for java-gnome from using the dumb http:// protocol used to be 20 minutes (serious round-trip penalty when from Australia to Europe. Ouch).
Switching to bzr:// dropped the time to clone a branch to 1 minute 46 seconds… Sweet! And that’s even before the monster revision streaming and repository format performance improvements that are likely to land in the 0.92 - 0.93 time frame! Excuse me while I rush off to update our README and HACKING files :).
Yes, dumb protocol time was abysmal — But http is a convenience for when all else fails, and has the nice side effect of allowing people to surf your public branch’s code. The problem is that people don’t necessarily get around to putting the server in place (most people