←October→
Mon	Tue	Wed	Thu	Fri	Sat	Sun
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

This section:

Blog postings by Andrew Cowie about Open Source and Software Development

The syndication links at top left will give you a feed for the blog as a whole. If you'd like a feed specific to this sub-category, see bottom of page.

blogs > andrew > software > java-gnome

Mon, 18 Aug 2008

java-gnome 4.0.8 released!

This blog post is an extract of the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.8 (15 Aug 2008)

Cleanups and fixups.

This release is mostly to push out bug fixes and internal improvements, setting the stage for some major new feature development. We’ve also taken the opportunity to introduce a major change to the way we connect handlers for signals.

New coverage and continuing improvement

With thanks to new contributors Stefan Prelle and Andreas Kuehntopf we have a number of small improvements to the TreeView/TreeModel APIs. As always, Widget and Window saw a bunch of work, with Window.ConfigureEvent now being available and a number of additional property setters and methods relating to window type. Widgets that scroll around a view of a broader underlying canvas have seen a fair bit of activity related to controlling that scrolling.

New features include refinements and new coverage of methods in a variety of lower level classes including that further support drawing operations. Bug fixes, debugging improvements, and defencive enhancements to our thread safety measures have also featured largely.

Signal naming change

The names of the inner interfaces used to specify the prototypes of the methods which receive signal callbacks have changed to the pattern Button.Clicked, this being more appropriate to Java type naming conventions and providing better consistency between the signal name, the method to be implemented to receive the callback, and the method that can be used to emit this signal.

API compatibility to previous releases in the 4.0 series has been preserved. The old signal interfaces and connect() methods are @deprecated. Developers are encouraged to migrate quickly; new coverage will of course all follow the new pattern. Making the transition is is easy, especially in an IDE. Most of the people we’re aware of using the library have already ported their code. Just turn assertions on to double check.

Build changes

java-gnome now defines C compiler flags like GTK_DISABLE_DEPRECATED to ensure we are not linking against code that will be unavailable in GTK 3.0. Many thanks are due to new contributor Kenneth Prugh for having done some terrific grunt work to prune deprecated classes and methods from our .defs data so that java-gnome compiles without using these APIs.

The build system internally now ensures that multiple runs don’t occur simultaneously, fixing a number of annoyances that cropped up when using IDEs which tend to like trying to frequently re-run the build even if a previous one hasn’t finished.

Documentation, examples, and testing

Our API documentation and the growing set of example code have all been updated to reflect the new signal interface names. Doing so forced us to review a wide swath of the documentation, and so along the way a huge number of minor improvements were made. Given how detailed our JavaDoc is, this sort of painstaking work really makes a genuine contribution to overall quality.

There has been steady growth in our test suite, which is great. When combined with the snapshots used to illustrate our documentation, the coverage level is substantial.

Error handling continues to improve. In the (hand written) public API wrapper layer we do our best to catch misuses of the library before they get sent to the native code. But that’s not always possible, and in 4.0.7 we introduced a mechanism whereby GLib error messages get translated into Java Exceptions and thrown. As of 4.0.8, in addition to ERROR and CRITICAL, we also throw WARNINGs as Exceptions. Getting a stack trace this way has proved very useful in helping developers track down where they are making mistakes in using the library.

Conclusion

You can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.7..tag:v4.0.8

Because of the API changes to signal handling this release touches just about every public class in the library and so isn’t quite as clean (as a summary) as in previous releases — but it does show you everything that changed. :)

Looking ahead

Most of the contributors to java-gnome are working on branches that didn’t reach sufficient maturity to be merged in time for 4.0.8; that’s the way it goes sometimes. Major effort continues on implementing coverage of GTK’s powerful TextView/TextBuffer APIs, along with further drawing capabilities in Cairo and Pango. There have also been a surprising level of interest on other areas of the GNOME stack, with new contributors working on adding support to java-gnome for Nautilus, GStreamer, and even WebKit. Exciting stuff!

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr checkout bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

AfC

Sun, 06 Jul 2008

Using Eclipse’s source code formatter from the command line

One of the many wonderful capabilities built into the side of Eclipse is a powerful code formatter. Adjusting source code to match one’s particular preferences is nothing terribly new; what makes Eclipse’s so excellent is the degree of configurability and the preview of the effect each setting will have.

An Eclipse project has several metadata files:

.project
.classpath
.settings/org.eclipse.cdt.core.prefs
.settings/org.eclipse.jdt.core.prefs
.settings/org.eclipse.jdt.ui.prefs

Which seems a bit of a mess, but whatever. The .settings/ stuff gets written down when you decide to tell a Project to have settings custom that override the user’s defaults. It goes without saying that for a collaboratively developed project like the Java bindings we have the Eclipse project metadata for compiler warnings tuned, the code formatter settings, how to invoke the build system etc all stored in Bazaar along side the real sources.

Anyway, this’s all fine and nice for people working on my software in Eclipse, but what about people wishing to use a different editor?

For a few versions now, Eclipse has exposed the the ability to run its code formatter from the command line; the work was done by Ben Konrath, someone I met a few years ago at a java-gnome hackfest. Since I use Eclipse I hadn’t needed to bother to figure out how to use whatever it was that he’d done. Since becoming maintainer of java-gnome, though, I’ve noticed a bit of unnecessary formatting churn in the contributions we receive. We do have a style guide, but it is secondary to the fact that Eclipse just normalizes the code every time I save something (nice). But for people not using Eclipse I thought I might have a look.

A bit of searching around turned up this blog post by one Peter Friese. It included two important points that are non-trivial. Firstly, that configuration the code formatter from command line will use can be controlled by the pointing it at the .settings/org.eclipse.jdt.ui.prefs file which in turn can be created by the “Enable project specific settings” described above (we’d already done that, so no problem, but how on earth would anyone have guessed this?). Secondly, the command line that one has to incant to make it go is pretty voodoo, so I’m pretty thankful that he described what he’d done.

In our environment, this works out to:

$ /opt/local/eclipse/eclipse -nosplash
    -application org.eclipse.jdt.core.JavaCodeFormatter
    -verbose
    -config .settings/org.eclipse.jdt.core.prefs
    src/ tests/ doc/examples/

Gah!

For a brief moment I wondered if it would be easier just to run the Java class in question. So I interrupted the process and had a look in the process table.

/usr/lib/jvm/sun-jdk-1.6/bin/java
    -Dosgi.requiredJavaVersion=1.5
    -client
    -Xms40m
    -Xmx256m
    -jar /opt/local/eclipse/plugins org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
    -os linux
    -ws gtk
    -arch x86
    -launcher /opt/local/eclipse/eclipse
    -name Eclipse
    --launcher.library /opt/local/eclipse/plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.0.2.R331_v20071019/eclipse_1021.so
    -startup /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar
    -exitdata 84001b
    -application org.eclipse.jdt.core.JavaCodeFormatter
    -verbose
    -config .settings/org.eclipse.jdt.core.prefs
    src/ tests/ doc/examples/
    -vm /usr/bin/java
    -vmargs
    -Dosgi.requiredJavaVersion=1.5
    -client
    -Xms40m
    -Xmx256m
    -jar /opt/local/eclipse/plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20070828.jar

Good Lord. What the hell is all that? I mean, I know Java environments tend to get a bit unwieldy because there is still no link-to-binary facility, and I know Eclipse is an OSGi framework and what not, but that is pathetic. I guess running the command as described at the top isn’t such a bad idea after all.

I’ve got an even better idea: how about a convenience target in our Makefile front end. Yup. So I’ve added a format target:

$ make format

(or

$ ECLIPSE=/opt/local/eclipse/eclipse make format

in my case). Running that before committing will do the trick quite nicely.

I agree with the bug mentioned by one Nicholas Irving in his post on the topic: it is somewhat asinine that this bit of Eclipse bumps the modification time of every file it formats; it would be nice if it only changed files in which it actually found something it needed to alter. Fortunately our internal build system is actually md5 hash (not Make) based, so it doesn’t hurt us.

AfC

Mon, 05 May 2008

java-gnome 4.0.7 released!

This blog post is an extract of the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.7 (30 Apr 2008)

Draw some.

In addition to the usual improvements to our coverage of the GNOME libraries, this release introduces preliminary coverage of the Cairo Graphics drawing library, along with the infrastructure to make it work within a GTK program.

Drawing with Cairo

Example

The trusty Cairo context, traditionally declared as a variable named cr in code, is mapped as class Context. Various Cairo types such as different surfaces and patterns are mapped as an abstract base class (Surface, Pattern) along with various concrete subclasses (ImageSurface, XlibSurface, and SolidPattern, RadialPattern, etc). Error checking is implicit: the library status is checked internally after each operation and an Exception thrown if there is a failure.

Thanks in particular to Carl Worth for having reviewed our API and having helped test our implementation.

New coverage and continuing improvement

The single option choice buttons in GTK are called RadioButtons and have now been exposed. When using them you need to indicate the other buttons they are sharing a mutually exclusive relationship with, and this is expressed by adding them to a RadioButtonGroup.

RadioButton

The usual steady refinements to our coverage of the GtkTreeView API continue. There’s a new DataColumn type for Stock icons, and TreeModelSort is now implemented, along with minor changes to various other miscellaneous classes.

Considerable internal optimizations have been done, especially relating to ensuring proper memory management, with notable refinements to make use of “caller owns return” information available in the .defs data. This fixes a number of bugs. Thanks to Vreixo Formoso for having driven these improvements.

Error handling has been improved for GLib based libraries as well. If an ERROR or CRITICAL is emitted, our internals will trap this and throw an exception instead, allowing the developer to see a Java stack trace leading them to the point in their code where they caused the problem.

Internationalization support

java-gnome now has full support for the GNOME translation and localization infrastructure, including the standard _("Hello") idiom for marking strings for extraction and translation, but combined with some of the powerful support for positional parameters available from Java’s MessageFormat as well. There’s a fairly detailed explanation in the Internationalization utility class.

Build changes

Note that as was advertised as forthcoming some time ago, Java 1.5 is now the minimum language level required of your tool chain and Java virtual machine in order to build and use the java-gnome library.

Thanks to Colin Walters, Manu Mahajan, Thomas Girard, Rob Taylor, and Serkan Kaba for contributing improvements allowing the library to build in more environments and for their work on packages for their distributions.

The download page has updated instructions for getting either binary packages or checking out the source code.

Documentation, examples, and testing

Refinements to the API documentation continue across the board, notably improving consistency. A large number of javadoc warnings have also been cleaned up.

While not a full blown tutorial, the number of fully explained examples is growing. There are examples for box packing and signal connection, presenting tabular data, and basic drawing, among others. See the description page in the doc/examples/ section.

This code, together with the not inconsiderable number of unit tests and the code for generating snapshots of Widgets and Windows means that a large portion of the public API is tested within the library itself. The number of non-trivial applications making use of java-gnome is starting to grow, which are likewise providing for ongoing validation of the codebase.

Summary

You can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.6..tag:v4.0.7

Looking ahead

It’s probably unwise to predict what will be in future releases. The challenge for anyone contributing is that they need to understand what something does, when to use it (and more to the point, when not to!), and be able to explain it to others. This needs neither prior experience developing with GNOME or guru level Java knowledge, but a certain willingness to dig into details is necessary.

That said, I imagine we’ll likely see further Cairo improvements as people start to use it in anger. It shouldn’t take too long until the bulk of the functionality needed for most uses is present in java-gnome. In particular, forthcoming coverage of the Pango text drawing library will round things out nicely.

There are a number of other major feature improvements we’d like to see in java-gnome. Conceptual and design work is ongoing on for bindings of GConf, GStreamer, and even support for applets. Within GTK, there have been a number of requests made for various things to be exposed, for example, the powerful GtkTextView / GtkTextBuffer text display and editing capability. Some of these have preliminary implementations; whether or not any given piece of work is acceptable in time for any particular future release will remain to be seen and depends on the willingness of clients to fund us to review and test such work.

In the mean time, people are happily using the library to develop rich user interfaces, which is, of course, the whole point. We’re always pleased to welcome new faces to the community around the project. If you want to learn more, stop by #java-gnome and say hello!

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

AfC

Sat, 05 Apr 2008

Cairo drawing from java-gnome

We’ve added support for the amazing Cairo Graphics library to the Java bindings for GTK and GNOME.

Adding Cairo is a feature we’ve been working on for about 6 months now, and so I’m pretty pleased that during the 4.0.7 development cycle we’ve been able to land it. Cairo is a huge library, of course, but we’ve put enough coverage in place to ensure that things are working.

Cairo has lots of convenience functions and tons of obscure uses; no surprise (and no apology) that there’s still lots that will need doing. If you want to help make sure it has what you need, then grab ‘mainline’ and see org.freedesktop.cairo.Context and ExampleCairoDrawingInExposeEvent.

Huge thanks go out to Behdad Esfahbod and Carl Worth; Behdad was really critical in explaining some basic Cairo concepts to get me started when we were working together at the GNOME Summit back in October in Boston, and at the GTK hackfest in Berlin in March, Carl Worth checked our preliminary APIs and helped sort me out as we were working our way through to create some examples. Awesome!

example

java-gnome 4.0.7 should be out in the next week or two, depending on how long it takes to finish up testing, QA, and to do the release engineering.

AfC

Tue, 12 Feb 2008

java-gnome 4.0.6 released!

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.6 (12 Feb 2008)

Finding the missing methods.

Most of our effort recently has simply been fleshing out areas of the public API. The focus for this work as been getting the coverage needed to allow us to port some of our in-house applications to java-gnome 4.0. It’s not especially glamorous — if anything it has been tedious as hell — but the result has been a large body of improvements to java-gnome as a whole which we’re pleased to release as java-gnome 4.0.6

The bulk of this development took place on the ‘missing’ branch, so named because that’s where I was working on what was missing :).

Continuing Improvement

Notable public changes include coverage additions to enable key stroke and mouse button handling

Rather than exposing the int keyvals that bubble up out of the X server, we have wrapped these as constants of type Keyval (thereby being consistent with the rest of java-gnome in our working to the strengths of Java as a strongly-typed language; MouseButton was created for the same reason, helping developers understand just what on earth mouse button “1” is, anyway). Along with ModifierType, this gives enough to deal with the KEY_PRESS_EVENT and KEY_RELEASE_EVENT signals when the developer wishes to deal with key strokes.

We’ve finally gotten around to providing proper coverage of the box packing model which underlies every aspect of how GTK presents user interfaces. To understand the size-request/size-allocation process, you might start with Widget’s setSizeRequest().

We’ve also added coverage for SizeGroup, which, when used in concert with nested VBoxes and HBoxes, can work wonderful magic and is often far better than messing around with Table when doing complex layouts.

After dithering for several releases, we’ve settled on how we’re going to deal with ComboBox and family. The underlying GtkComboBox presents something of a nightmare as it is really two classes in one with more-or-less incompatible APIs. So, not surprisingly, we’ve presented it as two sets of classes, with the text-only convenience API spliced out of ComboBox and ComboBoxEntry into TextComboBox and TextComboBoxEntry respectively.

We’ve added a few new features in our coverage of GTK’s TreeView / TreeModel API, and many other classes involved have also seen improvements. The persistent reference to a row provided by TreeRowReference is now available as is model type TreeModelFilter.

Support for the actual filtering in TreeModelFilter is notable for having been quite tricky. The underlying C library use a function pointers rather than a GObject signal emission, and we don’t have any mechanism to handle that. We do, however, have a fantastic capability to marshal signals, so we dealt with the problem by creating a custom signal and then passing a function which emits it when the TreeModelFilter wants to ask the developer whether to include a row or not.

The new classes include support for TreeModel columns storing long data as well as setting properties of that type.

It should also be noted that most of the methods taking a TreeViewColumn have been converted to taking an argument of type CellLayout (an interface implemented by TreeViewColumn). This has no change to how you use our TreeView API, but was necessary to support ComboBox properly.

Finally lots and lots of minor additions to both public APIs and internals deeper down in the GDK part of the toolkit.

As ever, you can see the full changes accompanying a release by grabbing a copy of the sources and running:

$ bzr diff -r tag:v4.0.5..tag:v4.0.6

Documentation

We’ve always had HTML JavaDoc for the current stable release at doc/api/ on the java-gnome website. We’re going to change that a bit, though. As fixes to the explanatory documentation happen quite frequently to classes and methods all over the place, we’ve decided to generate the JavaDoc from ‘mainline’ periodically and upload that instead. This means that there will, of course, be descriptions of some methods which aren’t yet available in a released version of the library, but they will clearly identifiable by virtue of having a @since tag showing a version number greater than the most recent release. We’ll revert back to more traditional behaviour once we hit 4.2.0, but in the mean time we can give people access to the best information we can online.

The idea of having up-to-date illustrations of the various Widgets has proved popular, and we’ve continued to update the suite of snapshots. Doing that is also tedious, but it does provide a good opportunity to test APIs we are exposing especially where unit tests are less suitable.

Statusbar

Looking ahead

Almost as complex as the TreeView / TreeModel API are GTK’s powerful TextView / TextModel classes, collectively a Widget used to display and edit large text documents. Working out the java-gnome coverage for TextView will take a fair bit of consideration, but TreeView provides a road map, and, as with the coverage in 4.0.5 and 4.0.6 (which was driven largely by existing applications we were porting), we have some significant uses of GtkTextView which will guide us on our way.

The next release will also feature significant work outside of GTK; we should be in a position to merge our coverage of the excellent Cairo drawing library soon, and likewise we have tentative work in place letting people store configuration and settings data in GConf. Both the ‘cairo’ and ‘gconf’ branches need more QA and documentation work, but they’re looking good and will definitely be featured in java-gnome 4.0.7.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from ‘mainline’ using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

Well, now that I have this out of the way, I can finally stop worrying about having so much unreleased code and get back to the business of doing what I do for a living.

On a personal note, I mentioned that much of this coverage was driven by one of our legacy apps. About half way though this cycle, however, we decided we would be abandoning that program. Needless to say, it was more than slightly depressing to be putting so much work into writing new coverage for java-gnome in order to port such a large codebase in the certain knowledge that the application was not going to be resurrected after all, especially as we have not yet managed to raise any revenue to offset our costs in taking on the Java bindings for GNOME.

Cognitive Dissonance.

Nevertheless, I kept at it largely because getting that app to run again represented a good test of the quality and comprehensiveness of the Java bindings. It needed doing, and that has ultimately been the motivation behind all our work re-engineering java-gnome. I gotta say it was wonderful seeing that program come back to life bit by bit as the errors slowly fell away as missing classes and methods were added to the library. That monster running again after almost two years represents the 4.0 version of java-gnome having surpassed the level of coverage that we used from the old 2.x version, and that’s something to be very positive about indeed.

Of course, now all I have to do is to continue to resist the temptation to go back to working on that beast. :) No, better things to be about.

AfC

Mon, 26 Nov 2007

java-gnome 4.0.5 released!

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.5 (26 Nov 2007)

TreeView is here!

It’s always a great feeling when you bag a milestone, and with this release we have reached a major goal on our way to having outstanding Java bindings for the GNOME platform: coverage of GTK’s powerful yet complex TreeView & TreeModel API.

TreeView

TreeViews are a central part of almost every application. GUIs use lists for all sorts of things, and so a significant goal was to make coding TreeViews and their backing TreeModels as straight forward as possible.

The most challenging and complex part was to design the Java side API, which was no small matter. As a native library, the GtkTreeView API is complex and very much written with programming in the C language in mind, and as such our algorithmic mapping of the underlying libraries into Java doesn’t entirely fit. Long experience with the TreeViews in the previous bindings had made it clear just how nasty to use the API could be, and so the hardest part of the work was to come up with a mapping and a usage pattern that would be both faithful to GTK and be sensible to use.

The other significant challenge was to document the work effectively. Our Java side API documentation is a major feature of java-gnome, and merely exposing classes and methods is not sufficient; they need to be clearly explained in our JavaDoc as well.

along with numerous test cases in our unit test suite, and several comprehensively worked examples.

This was a monster patch, and the culmination of not just three months direct effort, but also where we’ve been heading since we first started the re-engineering of Java bindings for GNOME. Although largely written by Andrew Cowie, a significant contribution was made by Srichand Pendyala who not only exhaustively evaluated the design but also threw in some serious chunks of code. The work benefited from comprehensive input from Peter Miller on the modelling and design, and the comments of Bryan Clark, Owen Taylor, and Hanna Wallach were all really positive and helped us know that we’d gone in the right direction. Finally, thanks to Behdad Esfahbod and the GNOME Foundation who made it possible for us to meet in Boston at the GNOME Summit and so accomplish much of the final pulling together of this branch.

Continuing Improvement

Meanwhile, steady work continues on to the fundamental base classes, with a whack of additional signals and methods on Widget and especially Window, along with expansion of coverage in numerous other classes.

We’ve begun to get the basics of image handling in place,

One nice piece of contributed work came from Vreixo Formoso and Thomas Schmitz with coverage of the Dialog Window functionality in GTK. It took a bit of doing to map the int response codes used by GTK into something suitably strongly-typed, but all good.

Finally, minor improvements to all sorts of stuff, notably from new contributor Mario Torre. Awesome!

For further details you can always grab a copy of the sources and run

$ bzr diff -r tag:v4.0.4..tag:v4.0.5

to see the complete code delta.

Screenshots

For fun we built in a capability to create demonstrations to be captured as screenshots to illustrate various things. It doesn’t get more basic than the example on the Window documentation page, but it’s a nice touch. :) We’ve screenshots for a number of Dialog classes and one for the TreeView page. I imagine we’ll build up a nice library of images in the next few months (yes, dear contributors, you can add snapshots to the list of things we’ll be expecting along with well written documentation and unit tests when submitting additions to the public API).

Building and requirements

The library now depends on GTK >= 2.12. Those packaging java-gnome for their distributions please take note.

Looking ahead

Continuing to expand the coverage levels in the classes already exposed will continue to dominate our attention; there’s still a long way to go but we’re pleased with the progress we’ve made so far; you can definitely build real applications with java-gnome now.

The next release also ought to include preliminary coverage of GConf and Cairo. Doing each justice will again take a serious amount of work, but will continue to grow the fun things you can do with java-gnome.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:

$ bzr clone bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

This release marks a particularly satisfying milestone for me. I have three+ years of work that’s been waiting to be ported to java-gnome 4.0, but I’ve needed a new TreeView binding in place before I could do so. So this is a big deal for me. At last I can start thinking about getting back to actually hacking on the applications I actually want to be working on, and not spending so much of time on the underlying bindings themselves. There’s still a lot that needs doing on java-gnome, but it’s getting closer.

AfC

Wed, 21 Nov 2007

Automatically rendering and capturing screenshots

One of the things I’ve often noted in the API documentation for the various GNOME libraries is that the examle screenshots always seem out of date. I think this is mostly because the look and feel we all interpret as “modern” is just that which happens to be shipping with the current version of the GNOME Desktop we’re using — and that default changes fairly frequently.

As I’ve been working on the GTK coverage in java-gnome, I’ve had in the back of my mind that it would be really cool if we had illustrations of the various Widgets and whatnot under discussion. More than cool, actually; a lot of the people developing with java-gnome are experienced programmers but new to Linux and the GNOME Desktop, so it’s probably pretty important that we help them along in learning which controls on their desktop correspond to which Widget classes. Capturing such screenshots manually is a bit of a pain, though, and even if we did that we’d still end up with them looking out of date within a year or so.

So I had a bit of fun last week and knocked something together that would automatically take screenshots of individual Windows or fragments thereof such that we could use them in the java-gnome API documentation.

Snapshots

First I cooked up a scheme to make it easy to write the little programlets that draw the Windows and Widgets (I used the term Snapshot to refer to these; nothing as elegant as the JUnit API, but hopefully it’ll improve over time). Then I wrote a harness to instantiate the sequence of Snapshot programs and to capture a screenshot of each one in turn.

GNOME ships with a tool in gnome-utils called gnome-screenshot which is handy, but a bit too focused on being a user driven program and is not available as a library in any event. So we pulled out the parts that actually capture the image (my Lord, talk about ninja voodoo X programming — I remember working directly with Xlib back in the late 80s;I am so glad we don’t have to do that anymore) and made it accessible programmatically. So all Harness has to do is something like:

window = snapshot[i].getWindow();
window.showAll();
window.present();

pixbuf = Screenshot.capture();

and then save the Pixbuf to disk in the right place so that <img> tags in our JavaDoc can pick them up. I did up examples for Window, Button, a few MessageDialogs, and TreeView. They’ll be in java-gnome 4.0.5 which will be out within the next week or so.

Off screen

If you did this as a zillion Windows popping up in front of you, though, you’d probably end up capturing the wrong thing if you accidentally changed apps or something. So instead I tried to figure out how to do it off screen. The solution I came up with was using the virtual X server that ships with xorg, Xvfb. By doing:

$ Xvfb :1 -ac -screen 0 640x480x24 -wr

we have a new X server running on :1, and then by passing --display=:1 to the Gtk.init() call used to initialize GTK, the Windows rendered by the program will be sent to the virtual server, not the desktop running on :0, and then we can take our screenshots there.

I’ve had this in mind for a while as a way to do unit tests that absolutely need to have the Window mapped to the screen. Unlike the unit test scenario, though, there’s a catch: you want the Windows to look right, ie just as they would if they were on a user’s desktop. That doesn’t happen in a naked X server though; you need a window manager running and also the magic program that makes the GTK themes according to your current preferences:

$ metacity --display=:1
$ gnome-settings-daemon --display=:1

otherwise you get that rather ugly default built-in GTK fallback theme and no window decorations, which would defeat the whole point of the exercise. So hook that all up to be exec()d when the Harness program starts and everything works great!

I’ll have to remember to switch themes back to Clearlooks before generating the screenshots for our next release… :)

AfC

Wed, 26 Sep 2007

java-gnome 4.0.4 released!

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.4 (26 Sep 2007)

Coverage increasing!

Most of our work continues to be on infrastructure and architecture, improving the code generator that outputs the translation Java layer and JNI C layer which allow bindings hackers to reach the underlying native libraries. Nevertheless, there have been a number of publicly visible improvements across the board, so we wanted to push out a release highlighting these contributions!

Documentation improvements

Continuing our effort to have extensive developer friendly tutorial style documentation, there have been major additions to a number of existing classes. Of particular note is the Window class, containing the various utility methods used to ask the window manager to do things for you (we’ve also started exposing some of the deeper parts of the GTK toolkit, though only a few things that were immediately related to window management).

While the topic of thread safety was discussed at considerable length in the last release, we have added some of the more relevant information to the code documentation to reinforce its importance.

New coverage

Numerous people have been hard at work developing new coverage. The standards for accepting patches which expose public API are high, so it’s awesome to see bundles accepted for being merged to mainline from new contributors Thomas Schmitz, Wouter Bolsterlee, and Nat Pryce.

The infrastructure for a number of areas important to supporting applications including Menus, Toolbars, and Actions has been put in place.

A number of Container related Widgets have been added, though coverage is preliminary. There have, of course, also been a number of minor improvements in other existing classes, miscellaneous constants and wrappers around the stock item identifiers.

Internals

Vreixo Formoso carried out an important refactoring to the type database and Generator family of classes in the code generator, with the result that more of the array passing and out-parameter cases are now being handled correctly. This kind of work is usually thankless and taken for granted, but it’s hugely appreciated!

The real gains are in internal quality. A number of serious bugs and limitations have been overcome (Glade is working again, for example). The generated code now guards against improper use (you can’t pass a null pointer unless it’s allowed by the underlying library). Related to this is handling of “GError” — Java side, bindings hackers will get GlibException which they can then re-throw as an appropriate Java Exception, say FileNotFoundException in the case of not being able to open a file.

This all goes along with numerous build system fixes by Srichand Pendyala to make for an increasingly robust project. Thanks guys!

Looking ahead

As mentioned above, we have mostly been focused on areas other than public API, but it is expanding steadily. The hard work on infrastructure, however, is starting to pay off, and the next release should include coverage of TreeView, GTK’s powerful but complex list Widget.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:

$ bzr branch bzr://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

Note switch to bzr:// protocol.

AfC

Tue, 31 Jul 2007

java-gnome 4.0.3 released!

As promised at GUADEC, the landmark release of Java bindings for GTK and GNOME based on generated internals has indeed been made in July! (with like five and half minutes to spare) As is traditional now, this blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.3 (31 Jul 2007)

The code generator has landed!

Work has been underway for several months to develop the next stage of the new java-gnome: the code generator that will output the tedious translation and native layers that allow us to glue our public API to the native GNOME libraries. With this release we’re pleased to announce that the code generator is a reality!

Generated translation and JNI layers

The primary goal of the java-gnome 4.0 re-engineering effort has been to switch to an architecture whereby we could generate the bulk of the machinery necessary to take make native calls into the GNOME libraries from Java.

Extensive prototyping was done to establish the detailed design and to validate the architecture we had developed. Releases 4.0.0 through 4.0.2 contained this work along with mockups of the “translation” layer (the Java code that downshifts from our Proxy objects to primitives suitable to pass over the JNI boundary, along with the native declarations necessary in order to call methods actually written in C) and of the “JNI layer” (the C code that implements the methods declared in the translation layer which in turn coverts parameters into GLib terms and then makes the actual function call into the appropriate GNOME library).

With a solid foundation proving that our design was sound, we subsequently began the long effort to implement a code generator which would output these Java and C layers, allowing us to replace the mockup/ directory and at last leave behind the shackles of entirely hand written bindings. Over the past five months, the java-gnome hackers have been steadily working on the ‘codegen’ branch. The nature of the challenge meant that we had to have most of the code in place before any of it would be useful — never an enviable task to be working on. Thanks to the hard work of Andrew Cowie, Vreixo Formoso Lopes, and Srichand Pendyala, we reached the point where the output Java code compiled in May, and the output C code successfully compiled in by the end of June. Tremendous.

We’ve been bug hunting and refining since then, pushing towards the point where we could merge back to ‘mainline’, at last replacing the hand written mockup code. We are today pleased to announce the culmination of that work with the release of java-gnome 4.0.3.

This post on the development of the java-gnome code generator contains further details should you be interested; the file 5a-Architecture.txt in the doc/design/ directory of the source code explains the rationale and origin of the engineering design.

New coverage

Although our focus has evidently been on getting the generator into working order, there have nevertheless been a few minor coverage additions along the way which are largely to the credit of Sebastian Mancke for having submitted them and Srichand Pendyala for having fixed up their JavaDoc. There have also been steady improvements to a number of other classes; notably further signals and utility methods along with some preliminary coverage of the lower level GDK event machinery.

As is reasonable given our focus on writing the actual translation and JNI layer generators, most of these present only one or two methods from the underlying native class. Coverage will steadily improve as people contribute their knowledge and experience in documentation form.

People upgrading from 4.0.2 will actually notice that there are stubs for all of the public API classes; this was necessary to make the generated code compile. Most of these are empty as yet.

GList and friends

GList and GSList are the native GLib data structures used to represent lists and are the return type from quite a few methods across GTK. Vreixo Formoso Lopes worked out how to handle and working our wrap/unwrap functions for us to use in the JNI layer made a significant contribution to reducing our blacklisted method count.

Flags

We’ve also worked out handling of native types that, while type defined as enums, are actually bit fields.

Flags subclasses present an or() function allowing you to combine individual Flags constants into a new composite Flags object. Yes it’s a lot of machinery to do a logical |, but being strongly typed is a hallmark of java-gnome.

Build improvements

No release would be complete without mentioning that the code builds on more systems than it did before! Thanks to Maciej Piechotka and Srichand Pendyala for fixing problems resulting from Debian and Ubuntu strangeness.

The build internally now uses an optimized script that takes into account that even though the code generator may have run the translation and jni files may not actually be different. This was causing problems as Make only looks at file modification time. Instead, build/faster (great name, huh?) will only rebuild a target when source contents have changed. This was necessary for bindings hackers working in Eclipse; every time a file was saved Eclipse would merrily spawn off an auto-build which sooner or later would block the IDE UI. Yuk. Addressing this has also resulted in a faster build for everyone; all good.

The internal build script should be transparent; you still run ./configure and make as before. If you experience problems let us know.

Thread safety

In addition to the code generator, java-gnome 4.0.3 incorporates a comprehensive thread safety strategy.

None of the major Java graphical toolkits out there let you make GUI calls from threads other than the “main” one; they’re all single threaded. Even if all you want to do is a quick worker thread to carry out some input validation in the background after the user presses “OK”, you have to jump through horrific contortions to do so safely, resulting in cumbersome, clunky code.

By contrast, the new Java bindings of GTK presented in java-gnome are transparently thread safe, the first and only graphical user interface widget toolkit for Java to be so! We integrate properly with the underlying GDK thread lock and as a result you can safely make calls to various GTK methods from worker threads! This has been a long sought after goal and we hope a significant contribution to helping developers write elegant code.

Every call made to the native libraries is protected by entering the “GDK lock” [that’s gdk_threads_enter/leave() for those familiar with the C side of things]. The lock used is actually a Java side synchronized monitor and therefore reentrant; nested calls all behave properly. When in a signal handler callback the GDK lock is already held (you’re “in” the main loop when a callback happens), but since it just works transparently you don’t need to worry about it. If you do find a need to take the lock into account explicitly in your own code, see Gdk.lock in org.gnome.gdk.

It is worth noting that we have been warned that there are certain to be places in the underlying libraries that do not yet live up to the requirements of the GDK threads model — thus we will likely end up tripping over such things as we slowly add API coverage. We regard such inevitable instances as an opportunity to help contribute to improving the stability of the underlying libraries and will actively work with their maintainers to identify and resolve such issues. Nevertheless, in testing thus far our multi-threaded use of GTK has been rock solid. See these posts on GTK thread “awareness” and java-gnome’s thread strategy if you wish further details on our approach to the thread safety question.

Thanks in particular to Owen Taylor for having helped us navigate these waters!

Looking ahead

Taken together, these innovations represent the culmination of an immense amount of work towards realizing java-gnome as a viable platform for GTK and GNOME development.

Obviously with the generated translation layer in place the opportunity at last exists to start dramatically improving our coverage level, and we welcome contributions to this end. Prospective hackers are cautioned, however, that simply wrapping generated methods is insufficient — public API will only be added when it is clearly documented and meets the approachability criterion.

There are still areas where the code generator needs to be improved; we need to improve our handling for arrays, lists, and out-parameters — there are numerous permutations with all sorts of ugly corner cases.

Now that Free Java with support for generics is becoming widely available, 4.0.3 will be the last release holding the language level to Java 1.4; starting the next cycle 1.5 will be the minimum language requirement and we will be leveraging generics and other 1.5 features from here on.

Most importantly, the primary focus of the next few months will be developing a quality binding for the backbone of many applications: the TreeView Widget and the underlying TreeModel which powers it. The APIs in the native library is hideously complicated and has long been the source of confusion and pain for developers in C; it has long been a major goal amongst the java-gnome hackers to present a public API with as friendly and usable an interface as possible. It’ll be a good challenge.

You can download java-gnome from ftp.gnome.org or easily checkout a branch from mainline using Bazaar:

$ bzr branch http://research.operationaldynamics.com/bzr/java-gnome/mainline java-gnome

Back to work!

AfC

Fri, 08 Jun 2007

The code generator cometh

Calling C libraries from Java is somewhat arcane. It’s not so much that it’s “hard” as it is laborious. Once you’ve got your mind around it, however, and put the build infrastructure in place into your project (admittedly nasty learning curves), it’s not such a big deal to add another call or three on the margin. But for a library like java-gnome, which aspires to present a vast native libraries like GTK, each incremental function means an enormous amount of mundane and repetitive work.

Old way

In the old java-gnome, this Java and C code was hand written for every single method. Needless to day half the bugs that were in there were the kinds of errors that arise from madly doing so much cutting and pasting that your brain shuts off. Adding new coverage was insanely painful. No kidding it was abandoned.

New way

The primary motivation to re-engineer the Java bindings for GTK and GNOME was to switch from having to hand write all this glue code to an architecture whereby we could generate these complicated middle layers.

I released original prototype in November after 6 months of design work, and have been solidifying it in the java-gnome 4.0 releases so far. Obviously we had to mock up the parts that will be generated to make sure that we had the architecture right, which was no fun, because that’s right back to hand writing all the hideous boring shit in the middle. I’ve been pretty happy with it, though. The overall design has done a good job of handling unexpected corner cases as they’ve come up, and that’s always a good sign. Best of all has been the positive feedback from people who have looked at the code and complimented the design (we even had one guy come into #java-gnome the other day and say “java-gnome is really sexy”. Nice!).

But hand writing the glue code necessary to expand coverage sucks, especially when the whole point was to be generating all the boring bits. As you can see from this overview slide of the java-gnome 4.0 architecture,

it’s up to the code generator to do the bulk of the work. We needed to figure out the detailed design first, obviously, but clearly the next job is to get the machinery to output the necessary Java (code to translate from our public API into types we can safely ship across the Java/C boundary, and the native method declarations) and the corresponding C (the JNI code necessary to cast parameters and then actual call to the underlying library) … and of course the reverse direction for return values and callbacks like signals.

About 3 months ago we got started on it.

The codegen branch

The code generator has been evolving slowly but steadily. It started life as a little spike I wrote last September in Perl. That wasn’t going to scale, so Srichand Pendyala ported it to Python. He and I worked out some design issues there, but then we realized that that wasn’t going to scale either, so I ported it to Java. It’s been going gang busters since then, and we’ve since had lots of help from Vreixo Formoso and new contributor Sebastian Mancke. It’s getting to be quite the beasty.

The full stream of .defs data is massive; obviously the goal we’re pushing towards is to reach the point where the code generator can parse all the (define...) blocks therein and then output the appropriate Java and C code to implement each block.

So many corner cases. First time I tried it on the full data set it was not pretty :|. But we’ve been working away at it, and making painful but steady progress. Eventually I adopted the expediency of just blacklisting types that we didn’t know what to do with (an out-parameter that returns an array of arrays of GObject pointers? Fuck off!) — and finally last week I got a run that completed generation over the set of .defs files that represent GTK. That was an outstanding moment.

As of this writing, the java-gnome code generator outputs a translation layer of 475 classes containing 5454 methods (with 405 blacklisted due to requiring types that we haven’t figured out what to do with yet) and 733 callbacks (signal handler prototypes, mostly). There are still 162 odd compiler errors to go (some trivial, some really nasty), but we’re getting there!

Public API

This, of course, is just the glue in the middle. But this success means we are getting closer to the point where humans can get down to the serious work of writing the wrapper code and JavaDoc necessary to present those methods and signals we wish to expose in our public API.

One of the less appealing things about GTK in C is that numerous internals are unavoidably visible. In a language binding like java-gnome we don’t have to expose any of that — indeed, one of our primary design criteria has been zero such leakage. Thus there are a not insignificant number of entities in the .defs data that we don’t need to expose. GTK is huge, however, and it will take a long time to get our coverage up to a respectable level.

Where are we now? 58 public methods, 0.01%. That’d be radically embarrassing to admit, except that we gotta start somewhere. But it’s onwards and upwards from here — and although there are still quite a number of engineering design issues that have yet to be tackled, I don’t have to hand write JNI code any more, and neither do you.

The code generator cometh.

AfC

Tue, 29 May 2007

Multi-threaded GTK applications - Part 2: java-gnome

I have long cherished a dream to make the Java bindings for GNOME thread safe. I didn’t know if it could be done, but I sure wanted to try.

The workaround you had to do to be able to make GTK calls from other threads in the old java-gnome was terrible. It’s terrible in all the other major Java widget toolkits too. This is rather surprising, actually: one of Java’s strengths is its support for using threads. Now don’t get me wrong: multiple threads are not a panacea and I know as well as anyone that you can shoot yourself in the foot quite easily when doing threaded programming. But it is easy to get on with writing concurrent programs in Java, and threads are a fundamental feature of the runtime environment. Thus it has always struck me as a big let down that the GUI toolkits aren’t thread-safe (worse in fact: with the “GUI use must be single threaded” requirement, they are thread-hostile).

So I’ve been studying this for a while now. Could we do better in java-gnome 4.0?

Java’s synchronization primitives

Anyone who has done Java programming will be aware of the synchronized keyword. You can use it as a qualifier to a method:

public synchronized void doUsefulThings() {
    // code protected by object lock
}

but doing so is just shorthand for:

public void doUsefulThings() {
    synchronized (this) {
        // code protected by object lock
    }
}

The key point is that the object being used as the lock doesn’t have to be this; it can be any object, ie

public void doUsefulThings() {
    synchronized (obj) {
        // code protected by object lock
    }
}

Java synchronized blocks have a useful property: when you exit the block the lock is automatically released. This is good to know because if an Exception is thrown from some code called within the block, the lock won’t remain held. [By comparison, if you use lock functions explicitly (such as the J2SE 1.5 Lock interface) then you have to manually deal with the possibility of Exceptions being thrown by always using a try {} block as follows:

Lock obj;
...
    obj = new ReentrantLock();
...

public void doUsefulThings() {
    try {
        obj.lock();
        // code protected by object lock
    } finally {
        obj.unlock();
    }
}

which accomplishes the same thing, trading greater flexibility (you can have locks span methods or even classes) for slightly more cumbersome code]

Now, getting back to creating a Java binding around the GTK and GNOME libraries:

What others wouldn’t dream of considering

When you enable threading in GTK, you have to obey the injunction that all of your GDK and GTK calls must be done within the main GDK lock. As I discussed in my post about GTK thread awareness last week, this is fairly easy to ignore in C code largely because callbacks to signal handlers are done within the main loop running in a single (“main”) thread (since threads are only used infrequently in GNOME programs written in C), all within the GDK lock. But if you want to work from other threads, you have to surround that code which uses GTK with gdk_threads_enter() and gdk_threads_leave() and also do things like protect GIdle calls, etc.

As I said, I want to create java-gnome as a thread-safe API — no more of the ridiculous contortions imposed on the developer just to do GUI calls from a worker thread. A few ways to achieve this come to mind. Before I learned the above it seemed that the only way to do it would be to marshal every single call through some producer-consumer queue setup to convey the call across to the main thread. Yuk! [and more on this later]. Meanwhile, I learned about the actual thread awareness mechanism in GTK, and discovered that threaded use of GTK does not have to be done from the main thread only but from within the main lock. So why not just use that?

Instead of the default implementation (which is just is a simplistic GMutex), we:

use gdk_threads_set_lock_functions() to call custom functions to lock and unlock.
have these custom functions take and release a lock on a Java object.

You could certainly do (2) by calling from C back to Java via a JNI invoke and having custom lock() and unlock() methods written in Java. But rather than having to make a cross boundary call, (which would be a pain to write, is much more complicated, requires you to write your own Lock implementation and probably all be rather slow), JNI enables you to directly take the lock on an object!

    (*env)->MonitorEnter(env, obj);
    // code protected by object lock
    (*env)->MonitorExit(env, obj);

which is precisely the C equivalent of doing:

    synchronized (obj) {
        // code protected by object lock
    }

in Java!

(Yes, yes, JNI is ugly to use. Why do you think we’re rewriting java-gnome? So we can generate that layer!)

Anyway, needless to say, the custom lock functions to pass to gdk_threads_set_lock_functions() are pretty straight forward. All we have to do is get a C side reference to the Java object, and then call MonitorEnter() and MonitorLeave() in our lock() and unlock() functions, respectively. Too easy.

But why all this trouble to change the GDK lock to a synchronization monitor on a Java object? Simple: now we can use the same lock from both the GDK side and from the Java side.

This is brilliant, because it means we can transparently co-operate with the Java VMs thread co-ordination mechanisms, rather than fighting against them or worse ignoring them. And that suddenly means just get on with making the library thread-safe. Here’s why: Java monitors are reentrant. If the thread already holds the lock on an object and it encounters another synchronized block requesting the same lock, it just carries on [nesting the monitors, as you might imagine].

So the combination of:

Replacing the default GDK lock implementation with something that enters and exits a Java object monitor (thus protecting all internal GTK usage with that lock), plus
A synchronized wrapper around each GTK call (thus protecting all Java side usage of GTK with the same lock), especially Gtk.main() (thus creating the condition that the main loop is running within the lock), plus
The (largely unwritten) fact that the GTK main loop releases the lock by calling gdk_threads_leave() when it isn’t doing anything

means that other Java threads can safely make calls into GTK. Any thread that comes along while the main loop is busy will simply find the lock held by another thread (the “main” thread, as it happens) and will wait.

Point 3 bears closer examination. One of the other interesting details about Java synchronized blocks is that if a call is made to Object.wait() within it, the lock is silently released until the condition arises and then the thread tries to reacquire the lock so that the next instruction takes place inside the monitor as expected. Given that our wrapper code around the native call to start the main loop is this:

static void main() {
    synchronized (Gdk.lock) {
        gtk_main();
    }
}

we need that gtk_main() behaved as if a Java wait() call was made and thus releases the lock so that other threads can run. Interestingly, this is exactly the effect we end up with because although this code is marked as being within the section locking the Gdk.lock object, as the GTK main loop cycles it releases the lock via gdk_threads_leave() and then reestablishes it with gdk_threads_enter(). The effect is that the monitor on Gdk.lock is frequently relinquished, which is the behaviour that is expected if a piece of Java code object executes wait() within a monitor block. Which is exactly what we need!

As for point 2, while no C developer working with GTK would ever have contemplated protecting every single function call, for us wrapping a synchronize (lock) { ... } block around each call is easy (especially given that we’re generating that layer — two extra lines in the code generator, and BAMN! Done!) And because the lock is reentrant (recursive if you will), the normal C reflex to avoiding nesting the GDK lock discussed in my last post is no longer applicable. The thread making the calls already holds the lock, and just proceeds merrily.

But isn’t that going to be really slow?

No, actually. Sun’s “Hotspot” Java VM is highly optimized for two particular cases:

acquiring an uncontested lock, and
entering a block if the current thread already holds the lock in question.

You see, threads needing to take locks is very common, and early on in Java it was a bottleneck. So they tuned the hell out of it and both are really fast. More to the point we’ve tried it, and there is no user visible performance impact. Things run really, really nicely.

Remember I dismissed arbitrary function marshaling? There are thousands of functions in GTK alone; to arbitrary wrap and pass a representation of each native function would mean creating a message object to convey every single call, with another object to wrap each parameter. Ouch. One of the great achievements of java-gnome 4.0 was to reduce the object pressure. And while we might have been able to do it this way and so have only one GUI thread, we would have completely wrecked the type-safe and simple design we have now which takes advantage of the fact that passing primitive types across JNI boundary is cheap (The very reason that primitives in Java are not objects is because sooner or later the bytes they represent will have to be passed across to native code, and so by design they optimized things to that this would be cheap. In our case, we take advantage of this by simply using JNI to make a function call with arguments on the stack. Too easy. Otherwise you have to do complex bi-directional type lookups (reflection on the Java side and the JNI equivalents in C), and doing this on every method and parameter really would be slow.

Against all that, the cost of grabbing a lock is trivial, and the elegance of just simply using Java’s synchronized mechanism is really encouraging. That’s ultimately why I’m excited about this design for giving java-gnome thread safety.

java-gnome thread safety in action

Below is a tread dump from a small sample demo program that I wrote. All it does is fire off a worker thread when a Button is clicked. The worker is just a tight loop to make repeated calls to update a Label with an (incrementing) number. If you’re not a Java developer you won’t be used to reading these, but even if you aren’t you will probably get the idea pretty quickly: all the worker threads are waiting on the GTK lock except B which has the lock and is the thread currently able to update the Label!

(For clarity, I’ve cut out the VM’s system threads)

Button pressed. Launching A.
Button pressed. Launching B.
Button pressed. Launching C.
Button pressed. Launching D.

Full thread dump Java HotSpot(TM) Client VM (1.5.0_08-b03 mixed mode):

"D" prio=1 tid=0x0823a3c8 nid=0x2dba waiting for monitor entry [0xa969a000..0xa969aeb0]
        at org.gnome.gtk.GtkLabel.setLabel(GtkLabel.java:52)
        - waiting to lock <0xaaaf1440> (a org.gnome.gdk.Gdk$Lock)
        at org.gnome.gtk.Label.setLabel(Label.java:67)
        at WorkerTiming.run(WorkerTiming.java:112)
        at java.lang.Thread.run(Thread.java:595)

"B" prio=1 tid=0x0822e800 nid=0x2db8 runnable [0xa9598000..0xa9599030]
        at org.gnome.gtk.GtkLabel.gtk_label_set_label(Native Method)
        at org.gnome.gtk.GtkLabel.setLabel(GtkLabel.java:53)
        - locked <0xaaaf1440> (a org.gnome.gdk.Gdk$Lock)
        at org.gnome.gtk.Label.setLabel(Label.java:67)
        at WorkerTiming.run(WorkerTiming.java:112)
        at java.lang.Thread.run(Thread.java:595)

"C" prio=1 tid=0x0823f0b8 nid=0x2db9 waiting for monitor entry [0xa9517000..0xa9517fb0]
        at org.gnome.gtk.GtkLabel.setLabel(GtkLabel.java:52)
        - waiting to lock <0xaaaf1440> (a org.gnome.gdk.Gdk$Lock)
        at org.gnome.gtk.Label.setLabel(Label.java:67)
        at WorkerTiming.run(WorkerTiming.java:112)
        at java.lang.Thread.run(Thread.java:595)

"A" prio=1 tid=0x08085fc0 nid=0x2db7 waiting for monitor entry [0xa9496000..0xa9497130]
        at org.gnome.gtk.GtkLabel.setLabel(GtkLabel.java:52)
        - waiting to lock <0xaaaf1440> (a org.gnome.gdk.Gdk$Lock)
        at org.gnome.gtk.Label.setLabel(Label.java:67)
        at WorkerTiming.run(WorkerTiming.java:112)
        at java.lang.Thread.run(Thread.java:595)

"main" prio=1 tid=0x0805c880 nid=0x2d95 waiting for monitor entry [0xbfdab000..0xbfdac138]
        at org.gnome.gtk.Gtk.gtk_main(Native Method)
        at org.gnome.gtk.Gtk.main(Gtk.java:124)
        - locked <0xaaaf1440> (a org.gnome.gdk.Gdk$Lock)
        at WorkerTiming.main(WorkerTiming.java:92)

A done.
C done.
D done.
B done.

Note the cunningly selected class name of the lock instance :). Normally people just do Object lock = new Object(); and synchronize on that. But “waiting to lock a java.lang.Object” isn’t as helpful as it might be, and “waiting on the Gdk lock”, while cosmetic, seems a nice touch.

One slightly weird thing is that the VM thread dump for “main” notes a marker on the stack saying where a synchronized block was entered which does make it look an awful lot like it’s also holding the lock. That doesn’t make any sense because JNI side MonitorExit() must have been called and the main loop not running else another thread couldn’t be running. I worried about this until I read it a bit more closely and realized it is correct: the thread itself says it is waiting to be able to enter a monitor, and the only runnable thread is indeed B. It’s still a bit confusing, though; I think we should see about contributing a fixing for that in the VM — something that is now possible thanks to Sun having freed Java.

Conclusion

Thread safety is ultimately about protecting data from being left in an inconsistent state because of a context switch. That GTK imposes the restriction that calls must all be made from within a defined lock is not unreasonable at all and a standard approach thread safety. That we have the ability to supplant that mutex with a one that takes care of additional application requirements is a really wonderful feature of GDK.

Code to implement this strategy in java-gnome has been implemented on a quick temporary branch (go bzr!) so we can kill off this idea if we have to, but so far it has handled every tricky situation we can come with to throw at it: firing new Windows, cascading signal callbacks, nested main loops, and lock starvation. While throwing a synchronized block around every call at first seems extreme, it doesn’t cost us much, and actually follows good practice of keeping the locked sections as small as possible.

It still needs more testing, but thanks very much to Srichand Pendyala (who worked with me to figure out that a message queue architecture for passing arbitrary calls to a single GUI thread across the JNI boundary would be a nightmare), Vreixo Formoso and Karsten Bräckelmann (who have traced the code paths of this implementation of thread safety, listened to my arguments, and have reached the point of telling me why it will work and to stop worrying), Roman Kennke (who discussed with me his hypothesis that a recursive mutex would be a sound default for the GDK lock), and as ever to our guardian angel, Owen Taylor (who never ceases to amaze me with the quality of his contributions).

I have little doubt that if we follow this course of action, we will end up finding races in the underlying GTK libraries where the required GDK_THREADS_ENTER()/GDK_THREADS_LEAVE() macros were not used to protect code in idle callbacks and what not. It’s inevitable; the code paths that we are exercising are not as widely used as the more usual “all in the lock all in a single-thread” pattern that most GNOME applications written in C follow. Given the requirements of the GDK threads page, however, this would amount to a bug, and if tripping over such things occasionally (and contributing to rooting them out) is the price of a thread safe java-gnome, then it seems well worth it. Ultimately it will make the code better for everyone, and that’s what open source is about, after all.

AfC

Mon, 12 Feb 2007

java-gnome 4.0.2 released!

This blog post is the release note from the NEWS file which you can read online … or in the sources, of course!

java-gnome 4.0.2 (12 Feb 2007)

The End of the Beginning!

Major bugfixes and refactorings

Setting and getting properties on GObjects requires some tricky manoeuvring. We implemented the code to do this early on, and it looked like our general mechanism for getting Proxy instances for arbitrary pointers was working fine for properties. It turns out, however, that when you call g_type_name() on a GValue containing a GObject, it returns the name of the type that was listed when the property specification was registered, rather than saying it is a GValue (as you might expect) or what the object actually is (that you might also reasonably expect).

This led to all kinds of nastiness since the type name was what we were using in our instanceFor() mechanism to discriminate (on the Java side) what kind of Proxy subclass to create. The example we tripped over was asking for the parent property of a Button packed into a VBox. What g_type_name() told us was “GtkContainer”, not “GtkVBox”! And that was a big problem, because Container is abstract, and besides, we want to instantiate a concrete VBox Proxy, not a Container one!

The solution basically boiled down to having two separate code paths: one named objectFor() [a greatly simplified version of the previous instanceFor()] which returns normal Proxy objects for GObject subclasses (Buttons and Labels and whatnot), and a new code path available via valueFor() to specifically return our GValue Proxy for the cases where we know we’re getting a GValue back. Since that occurs in limited and known circumstances only (ie, when we’re getting properties) it’s no problem to know which to use when.

Thanks to Davyd Madeley for extensive debugging assistance, and credit to Manish Singh, James Henstridge, and Malcolm Tredinnick for having analyzed the root cause issue and having clarified that two code paths would indeed be necessary.

As often happens when you kick a stone loose, we were able to do a number of refactorings to clean things up. This eventually led to the realization (ok, epiphany) that our treatment of the GValue mechanism was needlessly complex. Toss. We no longer have individual Value subclasses for each different fundamental type, but rather just leave them as opaque references.

This allowed a further simplification of the valueFor() mechanism and even more smashing about in Plumbing with a chainsaw. Net result was a reduction by several hundred lines of code. Yeay!

All of these changes were confined to the internals of the binding machinery and are not user visible.

Loading `.glade` files

User interface designers are nothing new, but one of the really cool things about GTK has long been the existence of libglade. It’s a library which takes the output of a one of the GNOME user interface designers (such as such as Glade 3 or Gazpacho) and dynamically, at runtime, generates live Windows full of Widgets!

With the arbitrary Proxy retrieval sorted out, the beginnings of a binding of libglade was possible. None of the fancy stuff is there yet, but a .glade file can be loaded, and Widgets retrieved from the instantiated tree.

The JavaDoc for these classes clearly indicates that this is preliminary and subject to change. It may well all be blown away when GtkBuilder lands. We’ll see.

Testing framework

We’ve introduced the beginnings of a unit test framework. At the moment, this just evaluates various getters and setters without doing anything that requires the main loop. Despite this, the unit tests end up exercising the entire Proxy system discussed above; validating that the properties set and get and that the correct Proxy object is returned through a round trip is no mean feat.

You can run the suite from Eclipse, by specifying a JUnit 3 launcher on class UnitTests in the default package in tests/java, or by running

$ make test

the command line.

Further coverage

This release also sees the addition of FileChooser, FileChooserButton, FileChooserDialog etc. This is significant because GtkFileChooser is an interface in GTK, and GtkFileChooserButton implements it. We’d been putting off the question of dealing with GInterface (would it work or be a major problem?) for a while now. We were delighted to find that the design implied by the re-engineered bindings handled it cleanly, elegantly, and without any fuss. Another nice validation of our new architecture.

Finally, a number of new signals were exposed on Widget though these were mostly the result of doing live demonstrations at conferences of how easy extending the coverage of the new bindings is.

Memory management

We have successfully implemented full GObject memory management in java-gnome 4.0 using GLib’s ToggleRef mechanism.

A strongly referenced Java Proxy will not allow its GObject to be destroyed out from underneath it; meanwhile, as long as the GObject is still referenced by something other than java-gnome, an otherwise only weakly reachable Java object that Proxies it will not be finalized. When the situation does occur whereby the GObject is only referenced from java-gnome, and the Java object is no longer strongly referenced by any other Java objects, then the Java object can be garbage collected and the GObject will be unref()’d and destroyed.

You can watch the reference system in action if you set Debug.MEMORY_MANAGEMENT to true.

Huge thanks go to Vreixo Formoso Lopes who collaborated on the design, reviewed the implementation, and contributed test case code.

Build system improvements

A better detection of jni.h is done on Ubuntu, thanks to Michael Kedzierski. This makes java-gnome more likely to build out of the box on Debian-derived systems.

On the eve of release, Srichand Pendyala noticed that if you are running such a system, a package named libglade-dev needs to be installed. Of course, on more modern systems all the necessary dependencies are present merely by having GNOME installed in the first place. We’ll add a check for this Debian specific behaviour in 4.0.3.

The VERSION and APIVERSION constants were moved to org.gnome.gtk.Version so that anyone working on the Gtk main class isn’t forced to do a re-configuration every time they save.

Installation and Packaging

java-gnome 4.0 now has the standard make install command, and the equally standard --prefix option to ./configure.

$ ./configure --prefix=/usr
$ make
$ sudo make install

The install target understands the DESTDIR variable used by packagers to install to a specified prefix within a temporary directory.

See the README file for details.

The feature additions described above were done to bring java-gnome up to speed for the GTK & GNOME tutorial given at linux.conf.au. With that past, we’re not going to do any more manual mockups of code in what will be the generated layers. Focus now turns to designing and implementing the tool that will parse .defs files and output the translation code.

And so my firm returns to focusing on securing the funding needed to develop the project. Once more into the breach! When we succeed, the code generator will be our top priority and shouldn’t take more than a couple months to complete.

You can download java-gnome from ftp.gnome.org or clone a branch from mainline in our DVCS.

AfC

Fri, 05 Jan 2007

First tarball in the java-gnome 4.0 series released!

After 9 months of design, engineering, and mad coding, I am pleased at long last to make the first public release of the new java-gnome bindings.

We are now at the state where we have not just a theory but a working architecture that has been validated through interaction online and at conferences with the community around the world. My firm, Operational Dynamics, has sponsored the work to this point, and we are now seeking revenue to carry out the engineering necessary to take the bindings from a prototype with coverage of 0.1% to the level where it will be a fully usable binding. If you think your firm would be interested in engaging us to this end, or to ensure a piece of GNOME that you need is present, then please contact Andrew Cowie, andrew@operationaldynamics.com.

If it isn’t obvious, there’s still quite a bit of fluidity, and so I recommend to our downstream that it’s not quite yet time to package up java-gnome 4.0 for your distro. I’ll be in touch with the individual package maintainers in the coming months. Likewise for developers, yes, it’s exciting, but no, it’s not really ready for writing full apps with (though one guy did write something to show the cricket scores!). So while it is not really prime time (indeed, me even blogging about it smacks a bit of preemption), the open source vibe has ever been to release early and often so that others can participate. If you want to have a voice in the shape of the Java bindings for GTK and GNOME, you’re welcome to get involved.

The rest of this blog post is the release note from the NEWS file which you can read online or in the sources, of course!

java-gnome 4.0.1 (05 Jan 2007)

It’s not really a prototype anymore! the design works, and so the code that is here is forming the foundation of the new Java bindings for GTK and GNOME.

While there are some significant pieces of engineering that are yet to be done, and of course a universe of coverage yet to write, we’re pleased to mark the milestone of the prototype having proved itself to be stable and the strong foundation that we need. In this release:

Project documentation

Import project documentation, initially consisting of the re-engineering emails written by Andrew Cowie to the java-gnome-hackers mailing list, and expanded to include top level README and HACKING files, and a style guide for contributors to follow. All documentation [re]formatted in Markdown syntax so as to be renderable to web pages. See doc/design/ and doc/style/.

Project website

Create an entirely new website for http://java-gnome.sourceforge.net/, introducing sections “About”, “Documentation”, “Download” and “Interact” to discuss the the project as a whole, to be a home for the documentation, to provide instructions on how to get java-gnome, and information about the mailing lists and IRC channel, respectively. Also include a page thanking the previous generations of maintainers for their hard work and noting that the 2.x bindings have been unmaintained for some time now and are formally deprecated.

The website is no longer a wiki but is entirely within the source code of java-gnome itself. See the web/public/ directory; improvements welcome.

Major engineering

Quite significantly, the infrastructure to get a Proxy or Constant instance for any arbitrary C side pointer or enum is complete, involving significant work to the Plumbing classes, along with the corresponding C side code, especially in GValue.c

This was a necessary building block in order to complete the generalized getProperty() mechanism that, while hidden from public view, is nevertheless a major aspect of the GObject tool chest and is usable by bindings hackers when necessary. The generalized instance mechanism was the last major engineering hurdle that needed to be achieved in order to prove the new bindings design.

New coverage

Label, Fixed, Box, VBox, HBox and corresponding [working] mockup translation and native code.

Compliments to Srichand Pendyala from Bangalore, India for being the first external hacker to have a patch accepted to mainline! He contributed methods to Label and initiated the implementation of the Fixed class. In so doing, he also helped work the bugs out of the bzr bundle submission process. Awesome.

Build improvements

java-gnome now builds on Ubuntu and OpenSolaris in addition to its home turf of Gentoo. Thanks to John Rice of Sun Microsystems who provided the guidance allowing us to port Equivalence to Solaris some months ago, and Laszlo Peter, also of Sun Ireland, for several fixes to allow configure to recognize a wider range of Solaris environments.

The tiny example program that we have been using to validate the code, Experiment is now compiled by the build system if you so request. Try make demo.

API documentation

Extensive attention has been paid to the JavaDoc for the few methods that are presented so as to clearly set the standard required. The canonical JavaDoc for the project is available at the website with a stable URL and can be linked to.

All source code [comments, ie JavaDoc] have been spell checked! Initial top level overview.html and package.html files have also been written to help round out the JavaDoc.

java-gnome 4.0.0 (26 Nov 2006)

Initial release of the java-gnome 4.0 prototype, corresponding to the first public demonstration of the new bindings done at foss.in/2006 at Bangalore, India.

The prototype is fully functional and is intended to prove the design and architecture we have arrived at as a result of the re-engineering process. It includes both real wrapper layer classes that are the seed from which our public API will grow, along with the infrastructure that the wrapper layer depends on.

Initial coverage

Wrapper layer presenting the public API to developers: Widget, Container, Bin, Button, Window, along with complete translation layer implementations.

At present the bindings mock up the code that will be generated with temporarily hand written substitutes (ie, those in italics above) for both translation (Java) and native (C) layers. These will do until we receive the funding to make the code generator a reality; we certainly don’t want to be writing much more translation layer Java and C code by hand. Yuk.

Signal API

The defining aspect of GUI programming is, of course, that it is event driven. In addition to the “forward” direction of making calls to the native library and having return values bubble back up, there is the “reverse” direction of connecting callback handlers to the various signals that different Widgets offer, and having those signal events result in those handlers being invoked.

java-gnome 4.0 has an entirely new and redesigned signal connection and callback API. This functionality was demonstrated, and coverage of Button.CLICKED and Window.DELETE is now present and functional! The APIs used by GNOME language bindings to achieve this are some of the most voodoo I have ever seen. But it’s hooked up, and it works. It’s like black magic :)

Build

java-gnome is configured and built using Andrew Cowie’s Equivalence build scripts. It builds on Gentoo and should build on a Debian or Fedora derived system as well. See README for further details.

At the moment, java-gnome is a single source package. That may or may not change.

The source code is available via bzr. Again, see README.

You can download java-gnome from ftp.gnome.org but at this stage you’re just as well off to get the code from mainline in our DVCS. I’m pleased to say it’s moving quickly. :)

AfC

Thu, 04 Jan 2007

Spell checking Java source code

Dammit, Jim, I’m an Engineer, not an English major!

API documentation is incredibly important, but unlike tutorials or other reference documents that can be written in conventional text editors or word processors with nice built in spell checkers, the public API of a Java project is conveyed through JavaDoc comments that are contained in source code right next to each class and method as they are declared. We necessarily find ourselves with tons of generated web pages covered in prose… and full of spelling mistakes.

So I’ve had it in mind for a while to run aspell over the java-gnome sources. Raving insanity, that’s for sure. Spell check source code? Are you mad?

Well, being suitable euphoric on New Year’s Day, I decided “what the hell, why not” and gave it a try. One thing that was obvious was that I was going to end up with a ridiculous number of unknown words that were going to need adding. Rather than filling my personal dictionary (the one in $HOME) with tons of project specific crud, I used aspell’s -p option to specify a new word list in the project’s top level directory. Easy enough with bzr — they’ve got a bzr root command that tells you the path of the project root. Nice.

Here’s the command line I used:

aspell -x -c -p `bzr root`/.aspell.en.pws -H Button.java

It worked pretty well. The tokens sure did add up in a hurry, though. Java language keywords? Ok, no problem. Class names? Sure, makes sense to add them — many had already turned up when spell-checking other documentation in the project. But uh oh: it wants to know about getLabel( and num and x and every other bit of source code. Yikes. Quite the pain to add all that stuff while working through the files just to get to the JavaDoc and normal comments in order to fix the spelling in the text there.

But worth it… we now have spell checked API documentation!

What would really be neat is to write a little module for aspell that adds a mode that understands to only spell check stuff between /* and */ characters. The -H flag above tells aspell to ignore HTML markup, and there are modes for LaTeX and others. So hopefully a “source code” mode would be feasible, and I could start again and have a slightly better signal-to-noise ratio :)

Happy New Year!

AfC

Mon, 25 Sep 2006

java-gnome 2.14.3 released

Last week the java-gnome project did a point release of the stable branch of the Java bindings allowing you to write GTK and GNOME programs from Java. The 2.14.3 set of tarballs consist of:

glib-java      0.2.6
cairo-java     1.0.5
libgtk-java    2.8.7
libgnome-java  2.12.5
libglade-java  2.12.6
libgconf-java  2.12.4
libvte-java    0.12.1
libeds-java    0.5.1

As one would expect of a version number like 2.14.3, this release consists mostly of bugfixes and minor changes improving stability.

Of particular note are large contributions of patches and cleanups to the example code which ships with the sources of each library in doc/examples from the Sun team in Ireland. You guys are terrific. This effort was led by Michal Pryc.

You can download the tarballs making up this release from ftp.gnome.org. Anyone interested in learning more about writing GTK and GNOME applications using Java can either go to our website, java-gnome.sourceforge.net , or join us in #java-gnome on GIMPnet.

Meanwhile, work continues on what will become the java-gnome 2.16.0 set, which should hopefully be released soon.

As has been noted in several emails to both the java-gnome-developer and java-gnome-hackers mailing lists, we are engaged in a re-engineering of the Java bindings around the GNOME libraries, and so a major API break is expected after the java-gnome 2.16 series.

AfC

Mon, 18 Aug 2008

java-gnome 4.0.8 released!

java-gnome 4.0.8 (15 Aug 2008)

New coverage and continuing improvement

Signal naming change

Build changes

Documentation, examples, and testing

Conclusion

Looking ahead

Sun, 06 Jul 2008

Using Eclipse’s source code formatter from the command line

Mon, 05 May 2008

java-gnome 4.0.7 released!

java-gnome 4.0.7 (30 Apr 2008)

Drawing with Cairo

New coverage and continuing improvement

Internationalization support

Build changes

Documentation, examples, and testing

Summary

Looking ahead

Sat, 05 Apr 2008

Cairo drawing from java-gnome

Tue, 12 Feb 2008

java-gnome 4.0.6 released!

java-gnome 4.0.6 (12 Feb 2008)

Continuing Improvement

Documentation

Looking ahead

Mon, 26 Nov 2007

java-gnome 4.0.5 released!

java-gnome 4.0.5 (26 Nov 2007)

TreeView

Continuing Improvement

Screenshots

Building and requirements

Looking ahead

Wed, 21 Nov 2007

Automatically rendering and capturing screenshots

Snapshots

Off screen

Wed, 26 Sep 2007

java-gnome 4.0.4 released!

java-gnome 4.0.4 (26 Sep 2007)

Documentation improvements

New coverage

Internals

Looking ahead

Tue, 31 Jul 2007

java-gnome 4.0.3 released!

java-gnome 4.0.3 (31 Jul 2007)

Generated translation and JNI layers

New coverage

GList and friends

Flags

Build improvements

Thread safety

Looking ahead

Fri, 08 Jun 2007

The code generator cometh

Old way

New way

The codegen branch

Public API

Tue, 29 May 2007

Multi-threaded GTK applications - Part 2: java-gnome

Java’s synchronization primitives

What others wouldn’t dream of considering

But isn’t that going to be really slow?

java-gnome thread safety in action

Conclusion

Mon, 12 Feb 2007

java-gnome 4.0.2 released!

java-gnome 4.0.2 (12 Feb 2007)

Major bugfixes and refactorings

Loading .glade files

Testing framework

Further coverage

Memory management

Build system improvements

Loading `.glade` files