Monday, September 7, 2009

Return to Athena

OK, this heroic title theme is probably a little old. But I thought a wrap up report on my efforts to get the Eclipse CBI infrastructure up might be helpful to one or two folks. I got back from my retreat a week ago (a story I might relate in a less technically minded post) and dived back into the depths. And I'm a bit embarrassed to say that its taken a lot of that week to get things working -- I seem to be one who is always discovering the special cases. But that's not all down to Athena difficulties -- in the process I've discovered and cleaned up yet more of the AMP codebase and I think it conforms pretty well to the Eclipse way now. That's another thing about builds. A lot of the time we'd like to think that its the fault of the build infrastructure but often its more a matter of the greater rigor of the build process that has masked issues that our local IDE has let us get away with.

Anyway here are some observations in fairly random order..

  • skipPack=true is useful if you want to test locally. I found that the update manager does not work with pak'd jars when running locally. Perhaps P2 is relying on something on the web server side..?

  • The build scripts appear to be simply searching for the occurrence of strings to determine wether a given map entry type is being used. They do not appear to respect the comment lines. I guess I should file a bug report on this one -- it cost a bit of time because I have a dodgy SVN setup on my home laptop. I'm actually still not sure what is going on with that (and does the mysterious Java 134 error msg have anything to do with it) but I'm trying to learn not to fix things for which I have a work-around, i.e. if ti works on a different machine, just use that instead!

  • There is a somewhat lengthy description at http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started of how to setup your local build to use a local cache CVS site but I'm not sure what scenario that would really be helpful in. I just use COPY and it works fine. I suppose that using the CVS approach might excercie some issues that you wouldn't run into with a plain COPY. And I'm not sure wether the local build copies over the binary files or not. If it does, you could get different build results if say your local environment happens to place some artifacts in the copied directories that aren't cleaned out by the PDE build. Not going to worry about it!

  • A useful hack when building locally is to pack up your difficult dependency(s) into a zip file and refer to them in your dependencyURL. It doesn't matter what you use for the URL, because it will use the cached file instead. For example, GEF3D doesn't yet have a build or update site. (We've been collaborating a bit on this so hopefully the build I've got working will help them get one up soon.) So instead of solving the the SVN stuff right away, I put that off until everything else worked. So I created a zip and added an entry for http://download.eclipse.org/tools/gef/fake/gef3d.zip. You have to be careful to have your zip file packaged up correctly or it won't work. That's easy to get wrong..your file structure needs to conform to the standard structure, i.e. eclipse/plugins/some.plugin.

  • On the other hand, you have to be careful to remove these fake dependencies when you begin testing the real fetched versions! The fetch part then fails silently; your build succeeds but using the previously downloaded files.

  • But in any case getting the SVN map file right has been a major headache. Again, the only reason I want this is to grab the GEF3D sources, but I can't seem to let this one go as it seems that it should be so easy. The Athena build seems to mangle things in the fetch script; or perhaps this is a PDE Build issue. So I tried the sourceforge style entry and after a couple of hours of fiddling (see below) I found a magic incantation that seems to work -- plugin@org.eclipse.draw3d=SVN,trunk,http://dev.eclipse.org/svnroot/technology/org.eclipse.gef3d,,org.eclipse.draw3d@HEAD

  • It would be super nice if there were a built-in dependency analyzer so that one could find the root plugin issues easily. As it is, if you have a bad dependency you get the whole tree and you have to walk the tree by hand to find out the root cause. Overall I think this is another demonstration of the ultimate weakness of the standard build systems out there. They are by nature linear (batch) approaches and so usually you can't do post hoc or inline analysis of issues. There is no semantic information in logs! You end up having to parse them back if you want to do any useful analysis. In a parallel lifetime, I would love to work on a simple semantic logging system that would allow structured analysis of this sort.

  • Here is something else that I would love to do given time. The map files are pretty well structured - it would be nice to whip up a quick XText editor that would at least check syntax and more ambitiously provide a mechanism for hooking in semantic checks -- wouldn't it be nice if you could check that that SVN url is correct while editing the map file itself?!

  • I'm still not sure how Athena determines what goes in the sdk vs. runtime vs. example zips so for now I'm just building the examples manually. Anyone come across documentation for this?

  • I found this very nice article on hosting update sites from a sourceforge site and so I've moved the Ascape update site over to sourceforge now. Unfortunately due to the usual licensing issues Ascape the Swing dependent stuff will remain hosted there. People will have to go to the Repast site for Simphony integration and LWJGL for the GEF3D dependencies. The most important thing for me is to have a seamless user installation experience, and I thought through pretty carefully how to set up dependencies so that users would never need to get a dependency that wasn't necessary for their given task. It's a trade off though, as I've ended up with quite a few project features.


Anyway, the really good news is that I have the AMP update site working now :D as well as a sourceforge hosted site for Ascape. I've got my hudson build request in and am anxious to try all of this out. If I can get the automated build working it will definitely have been worth the effort.
And Nick, I will try to sift through this and see if there is anything useful for Wiki.

I'll close with a bit of navel gazing. I've been thinking a lot lately about the effort and intelligence we -- meaning software developers -- put into solving problems, and what drives us to solve particular problems. Basically, why do I waste time with things that I really shouldn't. Getting through the build process has provided useful (if a bit uncomfortable) insights..

It was a real pain troubleshooting the SVN issues because I had to wait for all of the CVS project files to be downloaded before seeing the results of attempted SVN fixes. What I should have done was to set up an SVN only releng test for the GEF3D parts, but that would have required setting up a stand-in GEF3D feature, copying and editing the releng files etc. And of course I thought it would only take a few more cases of trial and error to fix the issue. But the dumb thing is that I kept thinking that even after spending an hour and a half on it. But I just couldn't drop the feeling that if I had gone to all of that trouble it probably would have ended up being an easy fix anyway, and that just a few more tries and I would have the problem solved.

And really, I didn't have to do any of this at all, because I could have just uploaded my current GEF build and built a perfectly good distribution from that. (And now that I do have it working, I've discovered that an issue with the current version means that I have to use a locally modified version of the code for my build anyway.) That would have meant that I might need to revisit the issue at some point in the future, but the point of YNGNI is that that point might never come.

How often does this kind of thing happen to you? It's just another example of how people are very bad at making risk benefit estimates. I need a good heuristic for when to stop throwing good money after bad and bite the bullet. (Two lame idioms in one sentence!) But I think there is a more going on here...

Do I take this stuff too personally? I feel like it should work, and I don't want to be defeated by a few lines of configuration code. So I bang my head against the wall trying to figure it out. Ego. But if I took just a single step back, I might realize that that that time could have been spent making dinner for my family or holding my five month old. Is there something inherently obsessive about our profession? If so, how can that be transformed into a healthier approach?

2 comments:

  1. Hi Miles, I picked up some of the ideas, and entered them as input to the Eclipse B3 project - http://wiki.eclipse.org/PDE/Incubator/b3/WishList.

    ReplyDelete
  2. Thanks Henrik! I'm also going to point to a nice follow up from Nick Boldt http://divby0.blogspot.com/2009/09/re-return-to-athena.html.

    ReplyDelete

Popular Posts

Recent Tweets

    follow me on Twitter