BlackBox

What are some of the basic steps that I’ve followed as I create my DSLs (especially with respect to Groovy)?

1. What are your spoken and unspoken requirements?
2. Who is the intended user? (this will help you decide some of the other questions later)
3. Has this kind of problem been solved before?
4. How did they solve it? (Language, Toolset, Techniques?)
5. How closely does that solution match our requirements?
6. What can we take away from that solution (if it doesn’t match the requirements)?
7. What kind of DSL do you want overall? (Internal or External)
8. Prototype, prototype, prototype – even if it is an Internal DSL you will still want to start out with a working prototype that your intended audience can play with and give you feedback on. This should be an iterative process to refine the language.
9. External DSL – how to implement?

I will cover the first 8 of these, and I will leave the External DSL for a later post (sorry . For this I’ll use the steps that I’m currently going through on my own code to illustrate my own thought process.

Requirements

So, what are my requirements for my main two DSLs – QueryBuilder and ViewBuilder (this doesn’t exist yet, but starting on the planning right now). For QueryBuilder I wanted a clean syntax that makes it easy for developers to create constraints/restrictions on a database query. Ultimately the query could be executed by Hibernate, JPA, iBatis, whatever, so it should be ORM-agnostic. This actually isn’t too hard as they all have this concept of a criteria on an object/table and restrictions for that query. And the same goes for ViewBuilder, I want an HTML-like tree syntax that I can specify components in as well as attributes that will be applied to the view when it is built for me.

User

This one is easy, I am targeting Web application developers, initially I am targeting Tapestry Web application developers, but since the framework will be pluggable (probably around the 2.0 time) you can specify what specific pieces you want to use (Tapestry/JSF + Hibernate/JPA + Spring 1.x/2.x/Guice, etc.).

Has this problem been solved before?

In the case of the QueryBuilder, yes. Grails currently has an implementation called HibernateCriteriaBuilder. This has a nice syntax for specifying criteria and restrictions on that criteria. As for the ViewBuilder I don’t think this has been solved yet, but I will be looking. There is something similar in Groovy’s core for creating arbitrary HTML, which is close but not exactly what I’m looking for as I will need to generate more than just HTML from it.

How did they solve it?

Both of these were solved using Groovy’s BuilderSupport object which allows you to build up arbitrary “nodes” from a collection of method, property and closure calls. Very powerful and a little difficult to wrap your head around the first 10 times you look at it (alright, maybe it was just me, but that took a bit to fully understand what the hell it was doing).

How closely does that solution match our requirements?

First, for the HibernateCriteriaBuilder it actually will build up a real, live Hibernate criteria and use it, which unfortunately is not what I want. I need to generate the Java code that will come out of it. And the MarkupBuilder gets me part of the way there but not quite what I’m looking for.

What can we take away from that solution?

I decided to take away the syntax from HibernateCriteriaBuilder and looked at how they implemented it to see what I could do with that. And for the ViewBuilder I am going to extend BuilderSupport itself and use that to build up my nodes of whatever objects (so more than just plain HTML text, actually objects that represent things).

What kind of DSL do you want overall?

Definitely shooting for internal I think, but that doesn’t preclude having some kind of external DSL for the class creation stuff (that was a giant pain to do in Groovy/Java – just not all of the information I actually needed). So, having some kind of custom language for defining models could definitely come in handy down the road, but I haven’t decided anything yet. Plus, as I learn more Groovy and get feedback/suggestions I may stumble across something that simplifies my design greatly.

Prototype, prototype, prototype

This, aside from requirements and research is probably the most valuable step. Once you decide what you are going to do, how in the world do you do it? Especially if you’re dealing with Meta-programming (which can be a hard concept to wrap your head around)?

Here is where my experience will hopefully prove valuable to someone out there, here are the steps that I took when developing my QueryBuilder:

1. Write the Unit tests first, start with the simplest test you can think of for how your language should work
2. Run it – yeah I know its going to fail, but prove it to yourself
3. Make it go green – whether you go ahead and create a simple class that has one method in it that does what you want do that
4. Write another 2 tests – think about them, how are they going to change that class you just wrote, how much more complex is it going to make the logic you need to write to make sure you do what you expect.
5. Fix the first test – is it time to extend DelegatingMetaClass yet? Or maybe try something simpler like a Category? Maybe the new ExpandoMetaClass will work. Try a few of them out, but I don’t recommend you dive right into extending BuilderSupport, not yet, not unless you have already used it before and understand fully what it does. Throw in lots of println "calling so and so with these arguments". These help identify that what you are expecting is actually happening while you write your DSL.
6. Did that work? Good, write it so the next test succeeds.
7. Continue doing 4, 5, and 6 until you are happy and ready to show it to someone
8. Let them play with it, get their feedback. Rinse, repeat, refactor. (Here is where I would suggest you revisit the BuilderSupport idea, does it make sense? Remember not everything is an arbitrary nested tree, so not everything will benefit from using this structure.)

Well, I hope that this is helpful to someone . I really have gotten the DSL bug and (mostly) enjoy writing them. If you run into a weird error, throw it out to the Groovy list, someone should be able to help you out.

Addendum

I meant to add this in when I first started thinking about this post. When I started writing my QueryBuilder I didn’t start out with it generating anything but simple select strings. This helped me visualize the proper nesting of things without getting too complex. Once that was working right I changed my unit tests to expect the actual Java code I wanted back and changed the logic of what the DSL output.

I’ve been working like a fiend on Chama and am just about ready for an alpha release (not much but will have some basic model and dao creation from a groovy dsl paired with database goodness). Like a good programmer I have been building stuff up in chunks so that I can test everything separately and now this morning I started finally integrating everything together and blammo! something strange was happening.

The first part of this puzzle lies in QueryBuilder.groovy, this class is a builder that eventually will get called from inside of a finder method (like findAll()) and get parsed by another meta class, but more on that in a minute. Inside of QueryBuilder I do some checks on whether we are passing a closure or not, if we are then I set the delegate class to the current QueryBuilder instance and call the closure. Thus, when the closure continues processing the next interior method of the closure gets called it calls the QueryBuilder again allowing me to create new restrictions inside the current criteria. Running my tests and everything works as expected. Now for the strange part.

In the next part I created a new class JavaModelDAOBuilder.groovy which will actually parse a Groovy script and call each method in turn through a custom meta class called MethodInterceptor. This meta class has now essentially taken first responder position to any method calls that happen on the Groovy script. Which actually sounds good on the surface, until I discovered that a closure is actually “owned” by its class. Now that the class has a meta class it will intercept all calls to the closure, completely bypassing the delegate.

After some digging I can see how this makes sense and after sending a message to the list I got some great responses. The final solution that Jochen(Blackdrag) came up with is to set a flag so that meta classes can “delegate” to another delegate. I think that this is definitely a workable solution, but for now I have some kind of hacky solution in the MethodInterceptor so that it looks for anything that the delegate should actually be handling and pass it back to that instance.

The good news is that it looks like this will be fixed in the near future, for those of us really pushing Groovy’s Meta Object Programming (MOP) and I think I’m going to be joining the dev list so that I can watch those conversations (especially the MOP-related ones). On another note I’m working on a few more Meta-related posts that will hopefully be of interest to some people .

Alright, so I lied. I decided to use this example elsewhere and definitely needed something more flexible. So I spent a couple of hours (and one weird Groovy compilation bug later) and got my HOM implementation so that it uses closures to find stuff. It still uses findAll, but now it makes the determination from the closure passed into it. Now, theoretically you can implement any “message” you want and associate it with a closure in ArrayListMetaClass and voila you have a new operator for any collection. Of course HOM should be more flexible as well, but that I will leave up to others as far as implementation (unless the bug bites me again and I want to expand it a bit more).

Here then is the new relevant code for ArrayListMetaClass:

def whereClosure = { method, args, item ->
    List list = InvokerHelper.asList(args);
    if(list.size() == 0) {
        if( item."${method}"() ) {
            return item
        }
    } else {
        if( item."${method}"(args) ) {
            return item
        }
    }
}

def unlessClosure = { method, args, item ->
    List list = InvokerHelper.asList(args);
    if(list.size() == 0) {
        if(!item."${method}"() ) {
            return item
        }
    } else {
        if(!item."${method}"(args) ) {
            return item
        }
    }
}

def queryTypes = ['where': whereClosure, 'unless': unlessClosure]

…

def Object getProperty(Object obj, String prop) {
    if(queryTypes.containsKey(prop)) {
        this.hom = new HigherOrderMessage(obj, queryTypes.get(prop))
        return this.hom
    } else {
        return super.getProperty(obj, prop);
    }
}

So, the new items that you’ll notice here is that I decided to go with the closure idea and created two new closures. The whereClosure takes three arguments, the method we’re calling, the arguments to that method and the actual item that needs to get called. Here’s what happens when this closure gets passed in to the HOM object:

def Object invokeMethod(String methodName, arguments) {
    def closure = this.prop.curry(methodName, arguments)
    return this.object.findAll(closure)
}

What we are doing is currying the closure that was passed in and setting the method name and arguments (as they are passed in to invokeMethod) and then passes the curried closure into the findAll() method. Of course I’ve added in a new unit test as well to test my new unlessClosure.

    newList = list.unless.retired()
    assert newList.size() == 3
    assert newList == nonRetiredCollection()

Again, if you want the code, you can grab it here

A friend of mine sent me this link a while back on Higher Order Messaging in Ruby, which tracks back to this original post on HOM in Ruby and eventually tracks back to a presentation given on HOM in Objective-C.

After reading the original post I decided to try my hand at a HOM in Groovy. For me it was mostly a learning experience so that I could get my feet wet with attempting to write a DSL. And it was a great learning experience, unfortunately there are a few stumbling blocks for my implementation in the Groovy language. But without further ado he is my very rough stab at it (You are welcome to take this code and make something more of it, as I stated this was primarily a learning experience and I have no real desire at this point to take it further).

Here then is what I did, first with the test case:

void testHOM() {
    def list = createCollection()
    def newList = list.findAll{ boomer -> boomer.retired()}
    assert newList.size() == 5

    list = createCollection()
    newList = list.where.retired()
    assert newList.size() == 5
}

def createCollection() {
    List boomers = [ new Claimant(name:"Joe", age:65, gender:"M"),
                     new Claimant(name:"Jan", age:61, gender:"F"),
                     new Claimant(name:"Bob", age:50, gender:"M"),
                     new Claimant(name:"Ana", age:60, gender:"F"),
                     new Claimant(name:"Tim", age:60, gender:"M"),
                     new Claimant(name:"May", age:65, gender:"F"),
                     new Claimant(name:"Fay", age:59, gender:"F"),
                     new Claimant(name:"Jay", age:70, gender:"M") ]
    return boomers;
}

The first part simply shows that we are looking through all of the retired claimants, which is decided by this code in Claimant.groovy:

def boolean retired() {
    return (((gender == "M") && (age >= 65)) || ((gender == "F") && (age >= 60)))

}

The second part is where we actually start using Higher Order Messaging. What is the idea behind HOM?

In short the idea is to provide a way to pass messages to objects (in this case “where”) and let the object deal with it (in this case by calling the retired() method on each item in the collection. Let’s take a look at how to actually do that. First we need to create a MetaClass that will intercept method calls (although not all method calls are intercepted which I have filed a bug on already). Here is the HigherOrderMessage object that will get returned when where is called.

package hom;

import org.codehaus.groovy.runtime.InvokerHelper

class HigherOrderMessage {
    def object
    def prop

    def HigherOrderMessage(obj, prop) {
        this.object = obj
        this.prop = prop
    }

    def Object invokeMethod(String methodName, arguments) {
        List list = InvokerHelper.asList(arguments);
        println list
        if(list.size() == 0) {
            return this.object.findAll{item -> item."${methodName}"()}
        }
        return this.object.findAll{item -> item."${methodName}"(arguments)}
    }
}

This helps us keep track of what object we are currently working on within a given HOM (in our case a collection of Claimants and it also helps us track what operation to perform on them, where in this case. As I said earlier, this is an extremely simple implementation, this could greatly be expanded to do a number of things and this implementation is by no means the most ideal (for instance you don’t want to keep track of where, you really should have a closure or something else, hopefully you get the idea).

As you can see by my poor code example, I’m not even checking to see what operation we’re performing, quite honestly I got fed up with some stuff and just wanted something that worked, so you get to see my poor code . But essentially all this class does is call the actual method that is tacked on to the end of the HOM call list.where.retired() on each item in the collection using Groovy’s closure for findAll.

Next, is the actual part that intercepts the where property call and returns this object, ArrayListMetaClass.

package groovy.runtime.metaclass.java.util

import groovy.lang.DelegatingMetaClass
import groovy.lang.MetaClassRegistry
import org.codehaus.groovy.runtime.InvokerHelper
import hom.HigherOrderMessage

class ArrayListMetaClass extends DelegatingMetaClass {
    def parent
    def collection
    def hom
    def queryTypes = ['where', 'unless']

    ArrayListMetaClass(final Class myClass) {
        super(myClass)
        initialize()
    }

    ArrayListMetaClass(MetaClassRegistry registry, final Class myClass) {
        super(myClass)
    }

    def Object getProperty(Object obj, String prop) {
        if(queryTypes.contains(prop)) {
            this.hom = new HigherOrderMessage(obj, prop)
            return this.hom
        } else {
            return super.getProperty(obj, prop);
        }
    }
}

This class intercepts all calls to ArrayList through some Groovy magic. First there is the name itself ArrayListMetaClass, and the second is the package it lives in groovy.runtime.metaclass.java.util (it actually took me a couple of tries to get this right, originally I put it in groovy.runtime.metaclass.java.lang which only works for classes in that package, so make sure you just prepend the groovy.runtime.metaclass to whatever Java package you need to drop it in, or your own package for that matter).

Once it has intercepted the where property call it then returns a HOM object that we saw earlier. This could of course be extended to embed a closure instead of a simple string to the HOM and then it would be extremely flexible.

All in all this was an extremely interesting exercise and got me much deeper into the Groovy internals and jazzed about the DSLs that I need to write (currently working on those, will post more as I get some code ready for sharing). If you are interested in my poor code, you can grab it here.

So, I decided to use Yaml for all of my configuration needs, in order to use this I needed a library to interface with it, I found JYaml which seems to fit the bill nicely. Here’s a short example of what I’ve used it for in Chama.

--- !org.chama.db.DBConfiguration
dev: !org.chama.db.DBConnection
  dbName: chama_dev
  driver: org.hsqldb.jdbcDriver
  url: "jdbc:hsqldb:mem:"
  username: sa
production: !org.chama.db.DBConnection
  dbName: chama_production
  driver: org.hsqldb.jdbcDriver
  url: "jdbc:hsqldb:mem:"
  username: sa
test: !org.chama.db.DBConnection
  dbName: chama_test
  driver: org.hsqldb.jdbcDriver
  url: "jdbc:hsqldb:mem:"
  username: sa

DBConfigurationTest.groovy
class DBConfigurationTest extends GroovyTestCase {

    void testReadConfigFile() {
        DBConfiguration config = Yaml.load(new File("src/test/conf/db.yml"))
            assert config.test.dbName == "chama_test"
            assert config.dev.dbName == "chama_dev"
            assert config.production.dbName = "chama_production"
    }

}

I also found another package when I was having some difficulty with JYaml called JvYaml, but when I tried to save my configuration class it gave me a nested object exception telling me that I couldn’t save nested objects into my Yaml file. That seemed kinda stupid to me so perhaps I was doing something wrong, but I didn’t have time to try and figure out what it was and went back to JYaml.

Guillame Laforge (one of the Groovy programmers) posted his thoughts on using Groovy as a DSL.

With the help of closures, methods, operator overloading, adding
properties & methods to classes, with named parameters and so on, you
can quickly come up with a language of your own, a DSL, targeted at
the subject you’re dealing with.

This was in response to an article that ran on Artima “Creating DSLs with Ruby”, of which most of the examples could be easily modified to Groovy code.

The Groovy team has created a page, “Writing Domain-Specific Languages”, to showcase some of the examples of using Groovy as a DSL, which I will probably be committing more to as the Chama project continues.

So, where has all this DSL stuff been leading? One of my projects that has been sitting on the back-burner for a while has been Chama. Chama was born out of the frustration of writing the same boiler-plate code over and over for database-backed sites that are run on Tapestry. While you can certainly use something like Trails I was not happy with the additional amount of configuration I needed to do in order to get the forms to look and behave the way I wanted them to.

Why not use Ruby on Rails then, or even Grails if I want to stick with the Java theme?

A variety of reasons:

• I’m familiar with Tapestry, Hibernate, etc.
• I’m very familiar with Java
• I want to learn Groovy as a language
• I don’t want to switch out my deployment options to have to deal with deploying my apps to a new server and having to configure it all over again so that it’ll run RoR
• I like generators , no really this is an exercise that I want to undertake and if others find value in it that’s great
• Mostly to make it easier for me to get some app ideas I’ve had off of the ground

What will Chama use:

• Groovy as the primary language, with some Java thrown in where needed
• Most likely Yaml for configuration with JYaml for dealing with it
• Quite possibly several DSLs for each area (Model, View, Controller)

I don’t know yet exactly how far I’m going to take Chama, mostly I just want to get a tool together that will help me build stuff. If others find value in it and it gets extended from there and heads in a direction I never thought about, great! All the code will be available under an Apache License once I start working on stuff. I have some tentative dates setup in my Trac instance (what?! using Python for a Java app?). Comments are welcome, and I will use this blog to announce releases as they happen, the first milestone is set for 12/22/06 (oops was tomorrow, moving to two weeks, wow it’s been two weeks since I first set this up already).