"English [insert your favorite academic discipline here] is such a cut-throat field. Everyone fights like cats and dogs because the stakes are so low." -- attribution unknown
Politics? What's that word doing in a technical discussion?
Before you run away, I'm not talking about the predilection on some mailing lists to slip from an innocent reference to H1-B visas into a full scale us-vs.-them left-vs.-right e-riot. After some very spirited and interesting discussions but also a lot of hurt feelings and intolerably low signal / noise, we finally had to ban all political discussion on straight-talking-java and move them to straight-talking-java-politics. Which no one bothers to subscribe to. Hmm..
Anyway, my point is that politics really is inseparable from all human endeavors. In fact one could say (I'm about to) that software architecture is often dominated by political concerns. Politics isn't just about disagreement, influence peddling and ego. OK, sometimes it seems that way. Those Java developers out there who are old enough to remember the AWT / JFC / AFC UI framework wars will know what I'm talking about here. XML schemas make up another vast terrain for people to fight over. ODF vs. OOXML ring a bell? (Points to the first person who recognizes the common player here.)
But politics at its best is also about exploring the perceived boundaries between various interests and concepts and creating a common ground that can civilly, if imperfectly, accommodate them. Well, isn't that what an API is about? So this isn't all Betamax v. VHS. People often simply have legitimately different ways of describing the elephant. There are extreme examples, but the politics of software architecture goes on everywhere. Whenever there is more than one person in the room, there will be different solutions that need to be reconciled and that's as true for an internal company application framework as it is for an IEEE standards proposal. This isn't a problem at all actually -- in fact one of the most rewarding aspects of software development can be the process of exploring, discussing and resolving various points of view and coming up with a well abstracted solution that is useful to everyone.
It is both amusing and annoying that the general public perception of programmers is of the lone anti-social pasty-faced hacker sitting in the dark staring at a computer screen holding a bag of Doritos. But those of us in the profession know that while we may resemble this caricature at crunch-time, software development is at its heart an intensely social enterprise, perhaps even more so than say law or medicine. Offhand, I can't think of another profession outside of academic research that is so dependent on inter-personal communication.
Open Source software can be a beautiful example of how politics of design can be done well. Open Source also has countless exemplars of the above quote. (What, there is competition in Open Source?!) Is it just me, or has anyone else noticed that the smaller the user base and the less money and prestige involved the more competing solutions and petty rivalries there seem to be? These would seem to be the very conditions that would most cry out for a coherent vision! Is it just that the law of the jungle doesn't take hold at this scale, or is there something more complex going on here?
Why should I care?
Well, perhaps you're actually involved in a field where there are a number of solutions competing at cross-purposes, and you'd like to do something about that. I'll draw from my own experience with Agent-Based Modeling tools, but I'm sure you can think of many other examples. Actually, ABM is not a perfect fit for the opening quote. For one thing, the stakes are quite high -- I still think that ABM has the potential to bring us closer to a deep understanding of the social sciences than any methodological advancement in the last 50 years at least. And our field is full of folks who agree about a lot of things, have rich experience in a variety of fields and are exactly who you want to be hanging out with in a bar at 2 am in Santa Fe, NM talking shop. But still.. Why is it that while there are now two major IDEs for the entire Java development space, there are at least seven decent ABM software toolkits in general use? You could say that it is about maturity -- and here I'm talking about software, not personality -- but we've been developing these tools for over 10 years now. It seems so wasteful. Why are there still so many people working on exactly the same features in parallel when we could be working on sets of collaborating capabilities?
I think that the conditions for the balkanized state of affairs has to do with a very well known tradeoff: the payoff for cooperation is lower than the payoff for defection. There is not (yet!) a large market for ABM tools, hence no private investment. And there is very little public money either. The federal research agencies are really leery of providing money for enabling software tools, and most of the institutions involved in software development are really bad at getting it. So ABM tools end up being supported as a kind of sideline to concrete research projects (or for a company like Metascape, on our own time). This means that incentives all revolve around making the tool developer's host organization happy. And it is much easier to argue to higher-ups that "our really unique software is going to put our institution on the map" than "we should participate with five other institutions on an Open Source software project". (It is also a lot easier to write grants for a project you control, incidentally.) On the other hand organizations can't justify much internal investment because its hard to make a clear business case, and the existing tools work "good enough". At my former company, ABM work built around Ascape brought in a very large part of the revenue -- at the end we were the only thing keeping the boat afloat -- but only token investment went back into the toolset.
So the tools that come out of this process tend to be sub-optimal. By creating little fiefdoms, organizations can lock in a set of users and potential customers / collaborators. Generally, the one tool that does everything pretty well will be the most successful, but there will be little incentive to improve on a feature by feature basis. On the other hand, a tool focussed on some particular benefit (performance, ease of use) will be able to protect a small but stable user base but won't be able to grow to fill out other niches. This whole frustrating state of affairs is actually what drove me to found Metascape in the first place. ABM software needs some kind of dedicated effort if it is ever going to get past the present barriers to acceptance, and I don't think that can happen purely at an academic or government institution. But at the same time, that effort has to facilitate rather than conquer.
Hey, that's all absolutely riveting, but what the hell does it have to do with development of meta-models for multiple targets?
OK, OK, I'm getting there. This all demands a core technology that is powerful, adaptable and transformative. So here's the take-away: Meta-Models can be used to break through API hegemony and facilitate truly open solutions.
I originally began exploring DSL and Model-Driven Approaches because I became convinced that they are the only way around limitations of current GPL API approaches -- see my paper on the original score system -- and because there are some wicked cool things we can do with generative approaches. What if there were a way to allow users to design complete agent-based models and then run them anywhere? Code generation would allow this. A non-developer could use a point-and-click interface to develop a model, play with it on a generated web-enabled application, and then send the very same model off to generate and run on a 100-node massively parallel machine. This alone is more than reason enough to want the capability to generate to multiple platforms and APIs.
But I got really enthusiastic when I realized that meta-models are an excellent way to route around the above political problem, or at the very least make it less of a burden to tool users. (Remember them?) With abstract representations, code could be generated for any number of existing ABM frameworks. Freed from API lock-in, users could choose the software tool that worked best for the task at hand. Just as importantly, tool developers could focus on their areas of strength. More broadly focussed efforts could concentrate on providing an open robust infrastructure that would integrate smoothly with those more focussed tools. I found this all to be a pretty compelling vision.
Perhaps.. but sounds suspiciously like a re-run of a few years back, when everyone was developing their own XYZML that was going to save the world.
There is a strong resemblance at first glance, and it is true that one can easily create nice XML schemas from ecore meta-models. But meta-models are much richer. They carry deeper semantics and most importantly a framework like EMF provides a whole set of interoperable services. Together this provides much more than a standard structure for everyone to agree on (or endlessly argue about). Instead, a model developer can create an abstract model that reflects the current state of the art for existing APIs, and at the same time provide a whole host of useful services centered around that model. Since MetaABM is built on ecore, we're able to offer our users all of the great graphical and text editing, persistence, code generation, query tools and so on. And yes, at the same time we can provide a simple non-platform dependent XSD for those who don't want to have Eclipse dependencies. So we're providing both a common representation (I don't like care much for the word "standard") as well as a common set of core tools that enable people to use that representation in countless different ways.
Yeah, but now you're replacing API lock-in with meta-model lock-in. "Meet the new boss..."
Curses, foiled again! Look, clearly companies and individuals have a self-interest in these efforts. IBM doesn't support Eclipse out of pure altruism and there are certainly advantages to technical leadership on an effort like this. But personally I get more joy out of knocking barriers down than in creating new ones. And for folks interested in a truly accessible playing field, MDSD approaches are hard to beat. The potential for lock-in is really attenuated by M2M transformation approaches. You don't like my model? Create your own model and a mapping and you have a fully automated get out of jail free card. And if we can support really painless customization of the generation process there is almost infinite freedom to do what you want with any given model.
It is true that this process won't work very well if there isn't a genuine effort to be ecumenical. And now we return to Part I; to what extent is our target wagging the model's dog? Happily, API target favoritism toward that has a dynamic counter-balance. If meta-model A is not sufficiently implementation neutral, you can create a meta-model A'. Then, if you still want to use A's target, you can map A' back to A.
Now as I pointed out above, most software politics really isn't the realpolitik variety. Its just a matter of people seeing things a bit differently or carrying a different history and the process of resolving, melding and combining their ideas and perspectives is one of the most creative and rewarding aspects of architecture.
In the MetaABM case, our work started out as part of the Repast Simphony effort and so some aspects of the design were directly influenced by that. This is true down to the level of some meta-class names; i.e. Context and Projection. On the other hand, the overall approach that Tom Howe and Nick Collier came up with was already very general and so the resulting meta-model is also very general. Generally at the highest level of a domain space there is a good deal of agreement about basics like object types and even class hierarchies. For example, every UI toolkit needs to support some kind of UI component, and those components are usually placed in some kind of container which exists in some kind of frame. Generally differences arise more at the level of how the various pieces interact, and how we get them to do what we want them to do. But in designing the original score system, I had a clear design goal to avoid putting those kinds of interactions into the model. And, yes, sometimes this lead to implementations that were sub-optimal for a pure Simphony meta-model or even a pure Java meta-model or conversely to compromises that were later undone when MetaABM became a separate project.
A lot of these decisions have to do with dancing between the contradictions of under-specification and over-specification, between freedom and constraint. As a simple example, I felt that the data types were too language / machine specific and so replaced the "double" type with "real". Wouldn't it be nice to generate a version of your model that supported pure rational numbers? With the flick of a switch you'd be able to discover wether your model had any floating point related artifacts. Now someone might argue that as an intrinsic part of any model people should be specifying floating point as that is the implementation that is most generally available, but I much prefer a solution that allows people to defer this decision to generation time. There are other areas in the the model that can and should be made more implementation neutral and flexible -- such as the overall structure definition -- and that's going to be something we'll tackle for the new AMF acore implementation. As always, feedback and discussion about all of this is highly encouraged and appreciated!
I've shared my theoretical and philosophical rationales for why we would want to transparently and flexibly support many different APIs from a single meta-model. In the final part, I'll explain how we did it using some very cool technology from the openArchitectureWare folks -- technology that will soon be part of the core Eclipse offerings.