Friday, December 31, 2010

EMF and RAP and Virgo, oh my..

Many of us who've spent time traipsing through the Eclipse wilderness are probably familiar with this story...

Tis a beautiful, sunlit 'morn and you're wandering around the world wide web when you come across a bright shiny page that says something like "Eclipse ABC does [exactly what you want to do] and [as we've worked out all of the old bugs] it is [now] [relatively] [unless you're using..] easy to integrate with XYZ". There is even a set of step-by-step directions to get your from point ABC to point XYZ. Now, even though you've been down this road many times before, hope springs internal in a [not so much anymore] young man's [or woman's] breast and you're up for a little adventure, as long as you come back before lunchtime. And in fact, the first part of your journey is pleasant, idyllic even. But then..

"..hmm.. this barren outcropping isn't on the map. And when did this tree trunk fall across the path? I think I already missed lunch. Still have to find a way around that somehow...maybe through this swamp over there. No, that definitely wasn't the way to go, let's back-track a bit. In fact, maybe it's time to head back, it's getting late. And what, waste all of the time I've already spent exploring this area? Let's climb up over this talus slope and see if we can get a bit of a perspective on things. CRACK!! ZZZTTT!! Great..almost zapped by a bolt of lightning and now I'm caught in a thunderstorm...typical."

So, you've struggled home at nine pm, wet, cold, tired and grumpy, and even though you missed dinner as well, your significant other greets you brightly. "Well, how was your day, dear?"

And somehow, your will is not broken. You've learned to accept this kind of thing as a natural part of the life of an explorer living on the edge. [OK, laying it on a bit thick here..] In fact, the very next day, you decide that there has to be a better way. Naturally, this being the land [ecosystem, if you prefer] of Eclipse, there is a better way -- or at least that's what the project page for a new wonder project subtly suggests. So you naively and dutifully follow their map, and head out for another morning of exploring, promising yourself that this time you'll definitely return before noon whether you've found the end of the rainbow or not. And...guess what? You walk out through the forest about 100 meters, take a left and then another right, push a few stray logs out of the way, and suddenly a clearing opens up and a beautiful maiden [um, what's gender neutral for "maiden"?] is standing in the middle holding a... OK, let's stop there.

The point is that, let's face it, often things in the Eclipse ecosystem, and oh, just about every other software ecosystem I can think of, just don't always really work as advertised. Sometimes they work really well, but there are glitches. Other times they don't work at all. Unfortunately you don't discover that until you've wasted enough time that you are then willing to waste even more time trying to solve all kinds of bugs and issues that really should have been solved a long time ago. By someone else. That's why it's especially nice when something does work. Exactly as advertised. The first time you try it.

I've been experimenting with solutions to putting EMF models on the web for a year or two now off and on. There are two really cool efforts out there, one based on RAP and the other based on GWT, and both of which Kenn Hussey, Ed Merks and others have put a lot of effort into. After minor fits, starts and glitches -- most self-induced -- I've been able to get cool EMF editors working on the web with relatively minimal effort -- insanely minimal if you compare to the time it would take to do a custom EMF front-end. But wait, did I say "on the web"? I should have said, "on a web page, launched from Eclipse". For I have not really been able to get past the point of having a nice little web app show up in self-hosted mode and actually being able to deploy it -- in other words, so that other people could actually use it.

I've tried and tried Tomcat hosting, but I guess I'm too dumb or impatient for it. Inevitably I got lost in the vagaries of WAR file creation, missing dependencies, obscure console messages and bizarre configuration issues, culminating through the deep dark labyrinth of the Tomcat logging system. Then -- wisely exercising good judgment and ignoring sunk costs for once -- gave up. This last time though I had something that I really really wanted other people to be able to use, and some quality time to devote to it and determined to solve it. I was encouraged in this effort because people have really understood what needs to be done to make this process more straightforward. For example, there is Holger Staudacher's GSOC effort for RAP WAR products [Where is Tupac when you really need him?].

Unfortunately this wasn't the end to my pain. And judging from the messages on the posting it hasn't ended everyone else's either. This isn't to diminish Holger's effort at all. To the contrary, it is the fact that this is such a difficult and log-across-the-path thing to take on that makes efforts like this so admirable and important. If all integration challenges were easy, we wouldn't need to worry about integration. All of the issues have to do not with the tools themselves, but with the target for those tools and the complexity of the underlying mechanisms, none of which the team has any control over. This is similar to the Buckminster and B3 effort. Efforts are needed because it isn't like Tomcat and Jetty and servlet bridges and other endless dependency issues are going to go away anytime soon, and dragons [or at least logs-in-the-path] be there.

So after stumbling around a bit, I thought [I'm getting to the point, finally] why not take a look at this Virgo thing? After all, as the RAP website says, there are two choices: Embed a server in Equinox, or Embed Equinox in an existing servlet container. And when you think about it, does it really make sense to layer one dynamic runtime technology on top of another if you absolutely don't have to? Like, isn't that just asking for trouble?

As usual, the documentation looked reasonable if not simple, and had the characteristic Eclipse modesty: "Here are the steps we used to deploy RAP on Virgo. We know the process can be improved, but just to give you an idea of how it works, try the following:" So I followed the steps, stumbled once or twice and [in less time then it has taken me to write this blog post and perhaps for you to read it] I had the RAP demo app up and running! With the latest [1.4] version of RAP even.

OK, great, but I'd actually gotten that far with Tomcat before. What I just knew wouldn't work was getting the EMF RAP stuff to work with that. I didn't quite get all of the console stuff and what not, but I just thought to myself, "what if I just dropped the EMF bundles into the same place, modified the demo .plan file to include them, and.. No way that could possibly work, right? Not only did it work, but it actually worked before I thought it would. I was ready for endless rounds of editing, rebuilding, restarting, reloading, re.. But instead, I saved the changes to the .plan file and Equinox/Virgo automagically hot-loaded them and ran through the dependencies there and then. I cleaned a couple of things up, and then I navigated to the URL, and there it was. Seriously, I haven't been that impressed by a technology that "just worked" since the first time I built an EMF model and editor. I'll say it: The Virgo/RAP/EMF stack is going to be -- already is -- a kick-ass combination. Light-weight development, light-weight deployment, industrial strength, web-based MDSD.

And actually, the steps are even simpler than outlined in the docs above (but it's likely you'll want to refer back to the details there). Oh, one special note, I've had it with being a build weeny. What I'm doing below is dumb, dumb, dumb. No product builds, no ant, no Maven, nothing but you, Eclipse and your m 'n f'in, drag 'n drop'n mouse.

  1. I'll assume you have a working model in Indigo EMF RAP. Note that there is currently a minor issue
  2. there.
  3. Download Virgo Kernel and put it wherever the hell you damn well feel like.
  4. Download Runtime RAP (I used 1.4) and drag the contents of plugins into [virgo]/repository/usr
  5. Download The OSGi Autostart Jar and put it [virgo]/repository/usr.
  6. Go to your copy of Eclipse and copy-drag the following stuff from plugins into the [virgo]/repository/usr :
  7. Export your plugins to [virgo]/repository/usr
  8. Create a file called whateverthehellyouwant.plan [OK, does the wold really need yet another dependency artifact type?] with something like the text below, obviously updating the version numbers when you come across this page in six months [when none of it will work anymore anyway], then copy it to [virgo]/pickup:
    <?xml version="1.0" encoding="UTF-8"?>
    <plan name="virgo-emf-rap-demo" version="1.4.0" scoped="false" atomic="false"
    <!-- present in virgo 
    <artifact type="bundle" name="org.eclipse.equinox.common" version="[3.5.0, 3.7.0)" />
    <artifact type="bundle" name="" version="[3.2.100, 4.0.0)" />
    <artifact type="bundle" name="org.eclipse.equinox.preferences" version="[3.3.0, 4.0.0)" />
    <artifact type="bundle" name="" version="[1.3.1, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.core.commands" version="[3.6.0, 3.7.0)" />
    <artifact type="bundle" name="" version="[3.5.1, 4.0.0)" />
    <artifact type="bundle" name="org.eclipse.core.contenttype" version="[3.4.100, 4.0.0)" />
    <artifact type="bundle" name="org.eclipse.core.runtime" version="[3.6.0, 3.7.0)" />
    <artifact type="bundle" name="org.eclipse.core.expressions" version="[3.4.200, 4.0.0)" />
    <artifact type="bundle" name="" version="[4.2.1, 5.0.0)" />
    <artifact type="bundle" name="" version="[3.5.0, 4.0.0)" />
    <artifact type="bundle" name="org.mortbay.jetty.util" version="[6.1.23, 7.0.0)" />
    <artifact type="bundle" name="org.mortbay.jetty.server" version="[6.1.23, 7.0.0)" />
    <artifact type="bundle" name="org.eclipse.equinox.registry" version="[3.5.0, 4.0.0)" />
    <artifact type="bundle" name="org.eclipse.equinox.http.registry" version="[1.1.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.equinox.http.servlet" version="[1.1.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.equinox.http.jetty" version="[2.0.0, 3.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.rwt" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.rwt.q07" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.jface" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.ui.workbench" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.ui" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.rap.ui.views" version="[1.4.0, 2.0.0)" />
    <artifact type="bundle" name="org.equinoxosgi.core.autostart" version="[1.0.0, 2.0.0)" />
    <artifact type="bundle" name="org.eclipse.emf.common" version="[2.6.0, 3.0.0)"/>
    <artifact type="bundle" name="org.eclipse.emf.ecore" version="[2.7.0, 3.0.0)"/>
    <artifact type="bundle" name="org.eclipse.emf.edit" version="[2.7.0, 3.0.0)"/>
    <artifact type="bundle" name="org.eclipse.emf.ecore.xmi" version="[2.6.0, 3.0.0)"/>
    <artifact type="bundle" name="org.eclipse.emf.rap.common.ui" version="[2.7.0, 3.0.0)"/>
    <artifact type="bundle" name="org.eclipse.emf.rap.edit.ui" version="[2.7.0, 3.0.0)"/>

  9. Got to command line and type:
    cd [virgo]
    export JAVA_OPTS=-Dorg.ogsi.service.http.port=10081
    [Mac users:] export JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

  10. Go to your browser and paste in: http://localhost:10081/thesamethingthattheworkedineclipse

That's it. Hope you find your way through the forest as easily as I did. If you don't, then just remember that you're probably not alone, the world is full of scary places, and even when something doesn't make any sense there is probably still something to learn. Take a listen to what Tupac is saying:
“I know it seems hard sometimes but remember one thing. Through every dark night, theres a bright day after that. So no matter how hard it get, stick your chest out, keep ya head up.... and handle it.”
Come to think of it, Tupac is probably not the best source for words to live by. So perhaps better to hear how "T Bone" Stankus does it:
I said, "Hey! Hey, weird little whiners, I am on a quest To dream the impossible dream. Walking down the road one day, doo-dah, doo-dah, I said, "Hey kids, I'm looking for the truth of life. Where do I go, who do I see?" They said, "Slow down, mister, in order to find the truth of life, one must see THE WIZARD!" I said, "THE WIZARD? Well, where does this wizard, old wise one, live?" They said, "You see the big, green, glow-in-the-dark house up on the hill?" I said, "Yes, I see the big, green, glow-in-the-dark house up on the hill. There's a big, dark forest between me and the big, green, glow-in-the-dark house up on the hill. And a little old lady on a Hoover vacuum cleaner going 'I'll get you, my little pretty, and your little dog, Toto, too!'. I don't even have a little dog, Toto."


  1. I'm glad you had a good first experience with Virgo etc. Thanks for taking the trouble to write it up in such an interesting way.

    Windows users, or should I say travellers in the Windows wilderness, please note that unzipping Virgo "wherever the hell you damn well feel like" is fine so long as you feel like unzipping it at, or very close to, the root of a drive, since Windows has problems with long paths which Virgo cannot easily avoid. Oh and you need to use 7zip or similar as the Windows built-in support for unzipping zip files doesn't work.

  2. @Glyn "Interesting", eh? ;D Yes, I left out that small minority of users of that archaic platform. I've run into this issue plenty for other users. So Windows users, try c:/WETHYW.plan. Unless of course you're one of those poor folks trying to get real work done on a "managed" machine in which case you can put it C:/WhateverTheHell/PlaceYour/Adminsistrator Has Set Aside/ThatLeavesYouAbout14CharactersToPutAnyOtherPathIn/AndThat ProbablyDoesntHaveTheRightPermissionsAnyway.

  3. I would also avoid spaces, be it Windows or whatever.

    Miles, I love the article :-) So true and I'll be trying Virgo one day.

    BTW, why don't just use Eclipse itself? (i.e. the same way as deploying Eclipse RCP?)

  4. Hi Hendy, thanks. :) Yes, spaces are bad, though I'm always surprised how well they on OS X. What I still have a hard time believing is that 20+ years on a company with a market cap of 238B hasn't figured out how to retrofit their flagship project so that it can handle paths larger than 256 characters. I mean they got through the 8.3 revolution ok and they were even able to handle 32 and now 64 bits. Ahem, not with the same binaries, mind you. It's not like they had to, oh a) transition from one completely different kernel, file-system and API to another or b) transition their entire customer base from big-endian to little-endian processors. Oh well, I guess they have higher priorities.

    I digress.. The point to the RAP stack as I understand it is that you want to be able to get out of any dependencies on Eclipse development environment per se. But the super neat thing about OSGi is that essentially all of "Eclipse" is is just collection of co-habitating OSGi bundles. And as I understand it Virgo is really "just" an OSGi host. When you assemble a RAP back-end, you're putting together the (nicely constrained) set of minimal parts of Eclipse that you need to support RAP. So that means I think that when we deploy to Virgo, we are using Eclipse itself, or at least getting as close to that goal as we can, with the loosest possible coupling possible.

    Now, it would be nice -- and I'm thinking it is safe to assume this -- that Virgo will become essentially *the* distributed / server-side Eclipse RT, which takes the Eclipse stack from a position of being locked to the desktop to being deployable anywhere. For Eclipse developers that has to be tremendously exciting.

    The deployment tools haven't caught up yet, but my guess is that they will really really soon. Already people are using Virgo to support all kinds of runtime scenarios that were previously hadnled by other mechanisms.

  5. BTW, deployment of an EMF-based GWT app is actually pretty easy (i.e., a button click) if you're open to do it on Google's App Engine...

  6. Yes, though in this case I didn't want to be tied to a specific vendor. I think GWT is a really great option but for rapid development it has a surprising amount of development time complexity in general over RAP -- for example having to compile the javascript as a separate step. Also, I still don't think there is a way to bring in EMF models from an external data store. There are advantages to each and it is really great to have the option of either one or even both!

  7. Thanks for this. I'm about to go mad going through the hoops you've already gone through. I simply need to expose EMF and some other code through REST or some other web services. EMF does not run on its own, so I'd rather use an OSGI container. So is it server side Equinox or Virgo? Am I going crazy here or there is an overlap? I'd appreciate comments (really) This post is a good starting point anyway.

  8. I think the simplest description of Virgo -- as I understand it anyway -- is server side OSGi + Equinox running on top of TomCat, but you don't have to worry about the TomCat part. On top of that, you simply need to place the EMF bundles and the appropriate eclipse dependencies. That's it.

    The EMF crew have been working on very transparent support for REST. I'm not sure what the status is on that, but in any case it wouldn't be at all difficult to grab the queries and squirt something back. It's only a matter of parsing the HTML query string, executing the EMF search, and building the JSON or XML string or whatever.


Popular Posts

Recent Tweets

    follow me on Twitter