Friday, December 21, 2007

The Oil and Gas Industry using Eclipse

It has been too long since I last posted so I decided to show off some of what we've been working on at Kelman Technologies.

The screen shots show off the IPE (Interactive Processing Environment). It is our latest generation of seismic processing tools based on Eclipse RCP and other Eclipse projects. Before you get your mouse in a knot this isn't a sales pitch. The IPE is not for sale. We use open source tools to turn our intellectual property into tools for "in house" use.

The IPE is the front end to our geophysical processing system. We use it to help our clients find and enhance oil and gas reserves around the world. Developing software for this industry is a challenge because of the immense amount of data that needs to be managed and processed. We deal with extremely large computer clusters, terabyte sized disk farms, exabyte sized tape systems and insane visualization requirements. There are very few industries in the world that have the same high performance computing needs as the oil and gas industry.

Cross plots have been a big challenge. How do you visualize 20 million or more data points? We have tried a number of different scientific plotting libraries that have all failed to handle the large amount of data points. Early 2008 we are starting work on our own plotting library to replace SGT which replaced FreeHEP.

This is what the earth looks like to a geophysicist. A seismic plot can easily be 3 feet by 20 feet long on paper. Convert that to a usable image on the screen and you have a 12,000 by 23,000 pixel image. That is 276,000,000 pixels or about 828 megabytes of memory. Our users expect to have dozens of these images open at one time. Libraries like Java Advanced Imaging were never designed for this sort of work load.

You are seeing seismic data plotted as wiggles and as a variable density plot. Seismic data files can be as large at 350 gigabytes with the requirement that any seismic trace be retrieved for immediate viewing. SWT has been surprising good at keeping up with the performance demands.

We used BIRT for our spectral analysis editor. It made for a fast development cycle but the business oriented charting isn't up to plotting thousands of data points. The guys working on BIRT have been very responsive to our requests but it's not a high performance scientific charting API. Work on a replacement starts in 2008. We'll still use BIRT for our management related reports.

The seismic data keeps its own processing history. It's an audit trail of how the raw seismic field data was processed into a final client ready dataset. It can take hundreds of processing steps and months of CPU time to generate one seismic dataset. The history alone for a single file can reach 100 gigabytes. Being able to create our own custom IDocument saves us from loading the whole file into memory.

Quality control of the seismic data is another big challenge. The volume of data makes it impossible for a human to inspect all of the project data.

Managing the volume of files has been in nightmare in the past. I've been impressed with how well the resources plugin combined with the Common Navigator Framework has been able to keep up with our demands.

The Eclipse User Assistance API has been used heavily. We have offices all around the world so we have to provide easy access to good quality help documents. We are using Camtasia to create built in videos. I don't want to get a call from our users in Libya at two in the morning.

We've written a smart job deck editor that has all of the features of the Java editor but it works with our proprietary processing language called kismet. Our seismic processors are a bit like software programmers. They write job decks that tell our seismic processing system how to manipulate the seismic data.

Our clients send data in a bewildering array of data formats. Using our standard database editor has reduced the confusion experienced by our users. As new formats appear we create a new translator and plug it into the IPE. All of our plotting plugins can then take advantage of the new data.

Kismet jobs can report errors and warning. We have our own kismet builder/nature. We get the best of the Eclipse IDE for our users who have no programming experience. Editor user assistance, hover help, and other great built in features makes it the editor of choice.

We used iBatis to access our PostgreSQL job tracker server. It's an unorthodox use of iBatis (the IPE is not web based) but it makes for a fast development cycle and easy maintenance. Our users can monitor a job on any cluster computer from the comfort of their desk.

For those times when our client sends us data in a mystery format with no file extension. We can load any binary file into our binary editor. We even have an ascii table view to help with conversions from hex to octal to decimal to ascii. I wonder if we'll need a slider ruler view?

Another sore point I have with the available image libraries. The seismic industry image standard is TIFF on steroids. We can't load a typical TIFF image into an SWT Image object because it will use over 2 gigabytes of memory. We're still working out performance kinks on this one. I'll be looking at Batik and libTiff in 2008. The SWT TIFF support wasn't designed with us in mind.

Our front end to the Portable Batch System (PBS) developed by NASA back in the early 1990s.

We use PBS to manage our computer clusters. Users can submit hundreds or thousands of processing jobs with a simple click of a button. It's grid computing on steroids.

The processing requirements vary depending on the state of the seismic data. We still have to manually configure our computers for peek performance. That is why we have so many different processing queues. I know it sounds like we're in the age of ENIAC but we still haven't found computer hardware that is flexible enough to handle all our high performance processing needs. Heck, we're looking at moving our processing system onto GeForce graphics cards to squeeze out every last FLOP.

Job, jobs and more jobs. Managing thousands of jobs that could run for weeks is getting easier. Using an Eclipse view to filter and sort jobs has made life a little less stressful for our users. More advanced tools are in the works for 2008 and 2009 as we try to figure out better ways for our cluster computers to manage themselves.

Bless the wiki. We use it more and more to keep our internal users informed. The Eclipse platform allows us to search our internal doc, our wiki, and our Eclipse based info center with the click of one button. It's great.

We are pushing the Eclipse RCP to extremes that I know were never envisioned by the Eclipse team back in the 2001. Although we've found a few performance weaknesses we've never regretted our decision to adopt Eclipse RCP.

Thursday, September 20, 2007

IBM Lotus Expeditor and Symphony

Dreams of IBM Lotus Symphony components appearing inside our Eclipse RCP application have quickly faded when reality tapped me on the side of the head with a large SOA middleware wiffle bat.

IBM Lotus Symphony is built on top of Eclipse RCP and IBM Lotus Expeditor. That is a key point. Your existing Eclipse RCP applications need to be ported to the IBM Lotus Expeditor environment. For example, the three IBM Lotus Symphony applications run inside the Expeditor.

For those who are interested, start with the following articles. I'm assuming you are comfortable developing Eclipse RCP applications. In fact, only Eclipse 3.2 RCP applications. If you have made the jump to Eclipse 3.3 you will have to wait. Also, you don't need to download IBM Lotus Symphony.

The last article shows how to port our favorite Eclipse RCP Mail Application into the IBM Lotus Expeditor SOA Middleware universe.

Wednesday, September 12, 2007

Sigue Sigue SIGTERM

Our IT group wondered if our Eclipse RCP based application could handle a linux system shutdown. They kept catching hell for rebooting computers at night to keep the batch processing system running. Our users would lose unsaved work and get grumpy. Grumpy seismic processors are not a pretty sight.

The trick is to teach the Eclipse RCP application to listen for a SIGTERM by adding a shutdown hook.

Please note that this does not work on Windows. More about this in a future blog.

public class IPEApplication implements IApplication {
  public Object start(IApplicationContext context) throws Exception {
    final Display display = PlatformUI.createDisplay();
    Runtime.getRuntime().addShutdownHook(new ShutdownHook());  }
    // start workbench...

The shutdown code must be run in the UI thread and should not be run if the workbench is being closed by other means. All dirty editors are automatically saved. This avoids prompting the user who is probably at home sleeping when their computer is shutdown. Finally the workbench is closed.

private class ShutdownHook extends Thread {
  public void run() {
    try {
      final IWorkbench workbench = PlatformUI.getWorkbench();
      final Display display = PlatformUI.getWorkbench()
      if (workbench != null && !workbench.isClosing()) {
        display.syncExec(new Runnable() {
          public void run() {
            IWorkbenchWindow [] workbenchWindows = 
            for(int i = 0;i < workbenchWindows.length;i++) {
              IWorkbenchWindow workbenchWindow =
              if (workbenchWindow == null) {
                // SIGTERM shutdown code must access
                // workbench using UI thread!!
              } else {
                IWorkbenchPage[] pages = workbenchWindow
                for (int j = 0; j < pages.length; j++) {
                  IEditorPart[] dirtyEditors = pages[j]
                  for (int k = 0; k < dirtyEditors.length; k++) {
                             .doSave(new NullProgressMonitor());
        display.syncExec(new Runnable() {
          public void run() {
    } catch (IllegalStateException e) {
      // ignore

Monday, September 3, 2007

Cheat Sheet Editor

Eclipse Europa (3.3) has a new Cheat Sheet Editor which is a vast improvement over the XML editor used in the Calipso (3.2) release.

  1. Create a cheat sheet, File->New->Other...->User Assistance->Cheat Sheets.
  2. Enter a filename and click Finish.

The new Cheat Sheet Editor will appear. Select the Item and you will see the definition and command sections appear to the right. Click the Browse... button to see the new Command Composer.

The Command Composer shows a list of available commands and their parameters. Gone are the days of hunting through eclipse plugin.xml files looking for command ids.

  1. In the Command Composer, select Window->Preferences.
  2. Use the Preference Page combo box to select General->Editors->File Associations.
  3. Click OK.

You now have a cheat sheet item which will display the File Association preference page.

If you want to see what happens inside Eclipse, display the ParameterizedCommand class and set a break point at executeWithChecks(Object, Object). Now run your application in debug mode, display your cheat sheet and press "Click to perform".

The command org.eclipse.ui.window.preferences is defined in org.eclipse.ui/plugin.xml. Its default handler is ShowPreferencePageHandler which is found in the org.eclipse.ui.workbench plugin.

Read up on the following extension points and use the ShowPreferencePageHandler as an template to build your own commands that can be accessed from a cheat sheet. See New menu contribution extension for readings on how to use a command in a menu, toolbar, or popup menu.

A logical representation of a task. A handler does the work. See wiki and help guide.
A handler does what the command represents. See wiki and help guide.

Thursday, August 30, 2007

New Europa default presentation

An internal note in the WorkbenchPage code lead me to think that our legacy code was broken due to a valid API change. Boris Bokowski has correctly pointed out that it is a bug 202280.

Eclipse Europa has a new default presentation that includes a new minimize/maximize behavior and new tab treatments.

I like the new feature but was a bit puzzled at the API changes made to IWorkbenchPage. Calls to IWorkbenchPage.html#isPageZoomed() always return false. Our application would never call IWorkbenchPage.html#toggleZoom(IWorkbenchPartReference) and our users became a bit bewildered by a view taking up the whole view stack.

IWorkbenchPage page = workbench.getActiveWorkbenchWindow().
if (page.isPageZoomed()) {

Instead you need to call IWorkbenchPage.html#getPartState(IWorkbenchPartReference). It returns one of the following states.

State of a view in a given page when the page is zoomed in on the view stack. (int)0
State of a view in a given page when the view stack is minimized. (int)1
State of a view in a given page when the view stack is in it's normal state. (int 2)

In our case a simple call to restore the view to its original state is all thats needed.

IWorkbenchPage page = workbench.getActiveWorkbenchWindow().

Wednesday, August 15, 2007

High performance graphics with SWT

When dealing with high performance graphics using a Graphics Context is not always the fastest way to render images.

We discovered this when trying to render a seismic variable density plot. On a typical two monitor system we were rendering an partial image of 2,000x800 pixels. Repeated calls to GC.drawPoint() or GC.fillRectangle() returned extremely slow rendering times. A whole seismic panel could be 10,000x4000 pixels. A large dataset could have 15,000 panels (about 500 Gigabytes) so you see we have to do things quickly.

The answer was to draw directly into an ImageData by setting each pixel using an indexed palette. In the image above text and grid lines are drawn using the Graphics Context while the colour pixels are set using ImageData.setPixel(). It sounds slow but ends up being extremely fast.

    PaletteData paletteData = new PaletteData(new RGB[] 
                   {new RGB(255,0,0), new RGB(0,255,0)});
    ImageData imageData = new ImageData(48,48,1,paletteData);
    for(int x=11;x<35;x++){
        for(int y=11;y<35;y++){
    Image image = new Image(display,imageData); 

There is a great article on using SWT Image called Taking a look at SWT Images Summary which the code example above was taken from.

SWT Canvas default behaviour

This eclipse.platform.swt news group posting started me thinking again about platform differences in Eclipse RCP and Java. Without a bit of care you could end up having DOUBLE SECRET SWT.DOUBLE_BUFFERED! My apologies to Harold Ramis.

The default behaviour of org.eclipse.swt.widgets.Canvas does support SWT.DOUBLE_BUFFERED on Linux GTK but not on Windows.

import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Canvas;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;

public class CanvasTest {
 public static void main(String[] args) {
  Display display = new Display();
  Shell shell = new Shell(display);
  Canvas canvas = new Canvas(shell, SWT.NONE);
  System.out.println("SWT.NO_REDRAW_RESIZE: "
    + stateToString(canvas.getStyle(), SWT.NO_REDRAW_RESIZE));
  System.out.println("SWT.DOUBLE_BUFFERED: "
    + stateToString(canvas.getStyle(), SWT.DOUBLE_BUFFERED));
  System.out.println("SWT.NO_BACKGROUND: "
    + stateToString(canvas.getStyle(), SWT.NO_BACKGROUND));
  System.out.println("SWT.NO_MERGE_PAINTS: "
    + stateToString(canvas.getStyle(), SWT.NO_MERGE_PAINTS));

 private static String stateToString(int style, int styleFlag) {
  return (style & styleFlag) != 0 ? "on" : "off";

Run on a Windows platform has the org.eclipse.swt.widgets.Canvas set as:

           SWT.NO_REDRAW_RESIZE: off
           SWT.DOUBLE_BUFFERED: off
           SWT.NO_BACKGROUND: off
           SWT.NO_MERGE_PAINTS: off

Run on a Linux GTK platform has the org.eclipse.swt.widgets.Canvas set as:

           SWT.NO_REDRAW_RESIZE: off
           SWT.DOUBLE_BUFFERED: on
           SWT.NO_BACKGROUND: off
           SWT.NO_MERGE_PAINTS: off

Thursday, July 26, 2007

New menu contribution extension


I started this post to help me and hopefully you understand what is involved in using org.eclipse.ui.menus. I'm assuming you have a good understanding of Eclipse RCP and you would use this as a reference guide. It stems from my fustration in using the Eclipse wiki and finding help with the new features of Eclipse Europa.

The Eclipse Europa (3.3) release has changed the org.eclipse.ui.menus extension to improve menu, toolbar, and popup placement and visibility. The idea is to move away from the old extensions.

  • org.eclipse.ui.actionSets
  • org.eclipse.ui.editorActions
  • org.eclipse.ui.popupMenus
  • org.eclipse.ui.viewActions

Where to start

I've listed the extensions with links to appropriate Eclipse help page and wiki page.

Scared yet? It is well worth the effort to learn and thus avoid, as much as possible, the legacy action extensions listed above.

What do you get? A very flexible and powerful way of creating menus, toolbars, and popup menus. I found the legacy action extensions to be difficult to implement complex menu and toolbar interactions and I ended up writing less java code.

A logical representation of a task. A handler does the work. See wiki and help guide.
A handler does what the command represents. See wiki and help guide.
Bind a command to a key. See wiki and help guide.
Bindings can be organized into a context. See wiki.
Give a command an image.
See RCP Text Editor Example.
See Creating an Eclipse View.
Add custom additions to the main menu, main toolbars, and view/editor context menus, toolbars, popup menu, and trim. See wiki and help guide.

Important to understand

Command Core Expressions needs to be understood to make your commands visible and handlers enabled.

Read up on Basic workbench extension points using commands.

How the old maps to the new.


Example code can be found in org.eclipse.ui.examples.contributions.

Development strategies

Keep notes of the commands you are creating and how they can be reused.

  1. Create or reuse a org.eclipse.ui.contexts.
  2. Create a command category.
  3. Create a command.
  4. Create a command image (optional).
  5. Create a binding schema.
  6. Create a binding for a command.
  7. Create a handler.
  8. Create a menu contribution.

Debugging and tracing

Check out this wiki page for details on a few debugging strategies.

Common mistakes

  • Commands added to a view toolbar must have either a label or icon defined in the menuContribution or icons defined in the org.eclipse.ui.commandImages.
  • Forget to look at org.eclipse.ui.menus.MenuUtil for correct spelling of menu constants.

Friday, July 20, 2007

Europa help, gone without a trace

The* plugins have had all of the tracing features removed for the Europa release. Why? Only this bug 197335 report will possibly tell.

They did leave the .options file in the plugins so that your run configuration appears to support help tracing. It doesn't so don't waste your time trying.

What you can do is set a break point in It will show you the help context id being used. Most problems are due to simple typos in your help *.xml file or in your view/editor/dialog call to PlatformUI.getWorkbench().getHelpSystem().setHelp(Control, String).

Wednesday, July 11, 2007

I come to praise TPTP and bury it

I tried out the new TPTP (Test & Performance Tools Platform) tool yesterday. In the end I was able to get it to work but there are a number of very annoying problems.

The first issue is the new monitor page in the run configuration will randomly forget its settings. If Memory Analysis is selected make sure you edit its options to track object allocation sites. Otherwise you'll have a hard time tracing object allocation back to your source code.

The big issue is with the Java Profiling options. Manually setting a content filter set does not work as expected. The Default filter set does not work as expected (already reported to Eclipse). The filter below hides all org* classes instead of only showing org.apache* classes.

    org*                 *    EXCLUDE
    org.apache*          *    INCLUDE

Specify the filter below worked only for me in our group.

    com.kelman.*         *    INCLUDE

Everyone else had to use the following.

    com.kelman*         *    INCLUDE

The potential issue is Java 1.6 is not supported by TPTP. For the project I'm working on it is not an issue since Java 1.5 works extremely well for us.

The documentation for TPTP is overly complex for someone wanting to simply run an application and do a bit of performance analysis.

Now that I've buried TPTP its time for a bit of praise.

Once we figured out the monitor configuration problems the tool worked like a charm. A great new feature is the integration of the Agent Controller. If you are debugging an application on a local machine, TPTP will start the Agent Controller. No more worries about installing and managing the Agent Controller as a service.

TPTP by default tries to show you as little as possible. I think this is a good thing. It keeps TPTP responsive and forces you to think hard about what packages or classes should be included in the performance analysis.

Tuesday, July 10, 2007

BIRT is not evil

I'm at the point in my project that most programmers dread. It is report generating time! Yeah, its not very exciting stuff to code. With the growing buzz around BIRT (2.2) (Business Intelligence and Reporting Tools) I thought I would give it a try.

I'm using BIRT with Eclipse Europa (3.3). Talk about a nice reporting tool. Building a report using the design tools was easy and obvious. I hooked it up to one of our Postgres databases (8.2.3) using a type 4 JDBC driver without any problems. A little SQL coding for a query and I was dragging and dropping fields into the report.

Creating customized formats for the duration fields and integrating the report into our application wasn't obvious so I went and bought the only two books available about BIRT.

BIRT: A Field Guide to Reporting (The Eclipse Series)
by Diana Peh, Alethea Hannemann, Nola Hague
ISBN: 0321442598
Integrating and Extending BIRT (The Eclipse Series)
by Jason Weathersby, Don French, Tom Bondur, Jane Tatchell, Iana Chatalbasheva
ISBN: 0321443853
Both are easy to read, gave me additional background to how BIRT works and most importantly saved me a good deal of time learning how to use BIRT and integrating it into our application.

Sunday, July 8, 2007

Adding Go Into and Go Up to CNF view

I have always liked the Go Into and Up features of the navigation view in the Eclipse IDE. Here is a simple way of adding it to your CNF (Common Navigator Framework) view. I'm assuming you are familiar with CNF, if not check out Common Navigator Framework.
You will need to add the Go Into action to your CNF view toolbar and the Up feature to the CNF view actions.
public class GoIntoAction extends Action {
    private IWorkbenchPage page;
    private CommonViewer commonViewer;

    public GoIntoAction(IWorkbenchPage p, ISelectionProvider selectionProvider) {
        page = p;
        commonViewer = (CommonViewer) selectionProvider;

    protected List getSelectedResources() {
        ISelection selection = commonViewer.getSelection();
        List resources = null;
        if (selection.isEmpty()) {
            resources = new ArrayList();
        } else {
            resources = ((IStructuredSelection) selection).toList();
        return resources;

    public boolean isEnabled() {
        List resources = getSelectedResources();
        return (resources.size() == 1) && (resources.get(0) instanceof IContainer);

    public void run() {
The GoIntoAction is added to the CNF view actionProvider extension. See here for Michael Elder's article on defining the viewer.
public class GoUpAction extends Action implements IViewActionDelegate {
    private ProjectExplorerView view;
    private CommonViewer commonViewer;

    public void init(IViewPart aView) {
        view = (ProjectExplorerView) aView;
        commonViewer = ((ProjectExplorerView) aView).getCommonViewer();

    public boolean isEnabled() {
        return !(commonViewer.getInput() instanceof IWorkspaceRoot);

    public void run(IAction action) {
        view.setInput(((IResource) commonViewer.getInput()).getParent());

    public void selectionChanged(IAction action, ISelection aSelection) {
Now add the GoUpAction to the CNF view toolbar.
<extension point="org.eclipse.ui.viewActions">
I extracted the icons from the org.eclipse.ui.ide_3.*.jar file and dropped them into my plugin. You will need the following method in your CommonNavigator implementation.
    public void setInput(Object aObject) {
        IResource resource = (IResource) aObject;

Wednesday, June 27, 2007

Debugging context sensitive help

This feature was removed from the Europa (3.3) release. See bug 197335 for details.
Context sensitive help allows you to assign a help page to an SWT widget. When it works it is a thing of beauty. When it doesn't it can be a small nightmare to debug. I'm not going into details about how to use context sensitive help since it is well documented in the Eclipse IDE. What isn't covered is how to use tracing to help you debug.
In your run configuration click on the tracing tab. Enable tracing and select the following plugins and all of the trace options for each plugin.
When you run your application/plugin you will see tracing information. Warning: the first time you activate the help plugin you will see a flood of trace messages. If you display the Problems View and press F1 (Ctrl F1 on linux) you will see the following.
The context id is "problem_view_context" and is associated with plugin "org.eclipse.ui".

Friday, June 15, 2007

ViewPart content description

I hadn't realized that a ViewPart can place a short description at the top.
public VintageView extends ViewPart {
    public void createPartControl(Composite aParent) {

Thursday, May 31, 2007

Adding a cheat sheet menu action

Creating a cheat sheet using the org.eclipse.ui.cheatsheets.cheatSheetContent extension is well documented. What is not documented is how to add the cheat sheet action to your application help menu. You need to make your application plugin depend upon the org.eclipse.ui.cheatsheets plugin. Then add the following extension:
<extension point="org.eclipse.ui.actionSets">
      label="Cheat Sheets"
        label="Cheat Sheets"

Thursday, May 10, 2007

Plugin your images

Let AbstractUIPlugin handle sharing images with your plugins. It has its own ImageRegistry that handles storing, disposing, and retrieving of images.

A simple override of initializeImageRegistry(ImageRegistry) and you are on your way to Image sharing bliss.

public class Activator extends AbstractUIPlugin {
    public static final String ID = "rcp.eclipse";
    public static final String MY_IMAGE_ID = 

    protected void initializeImageRegistry(ImageRegistry registry) {
        Bundle bundle = Platform.getBundle(ID);

        ImageDescriptor myImage = ImageDescriptor.createFromURL(
                               new Path("icons/myImage..gif"),
        registry.put(MY_IMAGE_ID, myImage);

Put your image into workspace/rcp.eclipse/icons/myImage.gif and then access it in your code.

AbstractUIPlugin plugin = Activator.getDefault();
ImageRegistry imageRegistry = plugin.getImageRegistry();
Image myImage = imageRegistry.get(Activator.MY_IMAGE_ID);