Groovy DSL steps

Warner Onstine — Tue, 05 Jun 2007 13:30:00 -0700

What are some of the basic steps that I’ve followed as I create my DSLs (especially with respect to Groovy)?

What are your spoken and unspoken requirements?
Who is the intended user? (this will help you decide some of the other questions later)
Has this kind of problem been solved before?
How did they solve it? (Language, Toolset, Techniques?)
How closely does that solution match our requirements?
What can we take away from that solution (if it doesn’t match the requirements)?
What kind of DSL do you want overall? (Internal or External)
Prototype, prototype, prototype - even if it is an Internal DSL you will still want to start out with a working prototype that your intended audience can play with and give you feedback on. This should be an iterative process to refine the language.
External DSL - how to implement?

I will cover the first 8 of these, and I will leave the External DSL for a later post (sorry ;-). For this I’ll use the steps that I’m currently going through on my own code to illustrate my own thought process.

Requirements

So, what are my requirements for my main two DSLs - QueryBuilder and ViewBuilder (this doesn’t exist yet, but starting on the planning right now). For QueryBuilder I wanted a clean syntax that makes it easy for developers to create constraints/restrictions on a database query. Ultimately the query could be executed by Hibernate, JPA, iBatis, whatever, so it should be ORM-agnostic. This actually isn’t too hard as they all have this concept of a criteria on an object/table and restrictions for that query. And the same goes for ViewBuilder, I want an HTML-like tree syntax that I can specify components in as well as attributes that will be applied to the view when it is built for me.

User

This one is easy, I am targeting Web application developers, initially I am targeting Tapestry Web application developers, but since the framework will be pluggable (probably around the 2.0 time) you can specify what specific pieces you want to use (Tapestry/JSF + Hibernate/JPA + Spring 1.x/2.x/Guice, etc.).

Has this problem been solved before?

In the case of the QueryBuilder, yes. Grails currently has an implementation called HibernateCriteriaBuilder. This has a nice syntax for specifying criteria and restrictions on that criteria. As for the ViewBuilder I don’t think this has been solved yet, but I will be looking. There is something similar in Groovy’s core for creating arbitrary HTML, which is close but not exactly what I’m looking for as I will need to generate more than just HTML from it.

How did they solve it?

Both of these were solved using Groovy’s BuilderSupport object which allows you to build up arbitrary “nodes” from a collection of method, property and closure calls. Very powerful and a little difficult to wrap your head around the first 10 times you look at it ;-) (alright, maybe it was just me, but that took a bit to fully understand what the hell it was doing).

How closely does that solution match our requirements?

First, for the HibernateCriteriaBuilder it actually will build up a real, live Hibernate criteria and use it, which unfortunately is not what I want. I need to generate the Java code that will come out of it. And the MarkupBuilder gets me part of the way there but not quite what I’m looking for.

What can we take away from that solution?

I decided to take away the syntax from HibernateCriteriaBuilder and looked at how they implemented it to see what I could do with that. And for the ViewBuilder I am going to extend BuilderSupport itself and use that to build up my nodes of whatever objects (so more than just plain HTML text, actually objects that represent things).

What kind of DSL do you want overall?

Definitely shooting for internal I think, but that doesn’t preclude having some kind of external DSL for the class creation stuff (that was a giant pain to do in Groovy/Java - just not all of the information I actually needed). So, having some kind of custom language for defining models could definitely come in handy down the road, but I haven’t decided anything yet. Plus, as I learn more Groovy and get feedback/suggestions I may stumble across something that simplifies my design greatly.

Prototype, prototype, prototype

This, aside from requirements and research is probably the most valuable step. Once you decide what you are going to do, how in the world do you do it? Especially if you’re dealing with Meta-programming (which can be a hard concept to wrap your head around)?

Here is where my experience will hopefully prove valuable to someone out there, here are the steps that I took when developing my QueryBuilder:

Write the Unit tests first, start with the simplest test you can think of for how your language should work
Run it - yeah I know its going to fail, but prove it to yourself ;-)
Make it go green - whether you go ahead and create a simple class that has one method in it that does what you want do that
Write another 2 tests - think about them, how are they going to change that class you just wrote, how much more complex is it going to make the logic you need to write to make sure you do what you expect.
Fix the first test - is it time to extend DelegatingMetaClass yet? Or maybe try something simpler like a Category? Maybe the new ExpandoMetaClass will work. Try a few of them out, but I don’t recommend you dive right into extending BuilderSupport, not yet, not unless you have already used it before and understand fully what it does. Throw in lots of println "calling so and so with these arguments". These help identify that what you are expecting is actually happening while you write your DSL.
Did that work? Good, write it so the next test succeeds.
Continue doing 4, 5, and 6 until you are happy and ready to show it to someone
Let them play with it, get their feedback. Rinse, repeat, refactor. (Here is where I would suggest you revisit the BuilderSupport idea, does it make sense? Remember not everything is an arbitrary nested tree, so not everything will benefit from using this structure.)

Well, I hope that this is helpful to someone ;-). I really have gotten the DSL bug and (mostly) enjoy writing them. If you run into a weird error, throw it out to the Groovy list, someone should be able to help you out.

Addendum

I meant to add this in when I first started thinking about this post. When I started writing my QueryBuilder I didn’t start out with it generating anything but simple select strings. This helped me visualize the proper nesting of things without getting too complex. Once that was working right I changed my unit tests to expect the actual Java code I wanted back and changed the logic of what the DSL output.

"Groovy DSL steps" by Leo

Mon, 11 Jun 2007 05:13:26 -0700

Hey, Warner. I left made a response on my blog here.. It's always a great idea to outline a systematic approach like you have. A lot of people fail to do that, and they end up buying books that tell them how just to get that much out of it. Very good. I hope it helps a lot of people.

"Groovy DSL steps" by Warner Onstine

Thu, 07 Jun 2007 12:43:10 -0700

Guillaume, I definitely agree. I wanted to come up with some basics, but the user-level involvement definitely has to be there. It's hidden right now in the Prototype phase I talk about, but should be more fully discussed in a future post. I have several posts in my head right now so I'll jot this down as one to write up ;-)

"Groovy DSL steps" by Warner Onstine

Thu, 07 Jun 2007 12:40:06 -0700

This comment came from Guillaume Laforge (Groovy lead) on the Groovy users list:

This is a nice wrap-up of the steps to follow. The thing I feel is missing a little is the "invention" of the DSL in itself with the end-user. I think we have to stress that part more. You have to know what's possible to do with Groovy in terms of DSL, you have to speak at length with your users, and incrementally design on paper (or on screen) what the DSL will look like. I think this step should be emphasized a bit more in the approach.

"Groovy DSL steps" by Graeme Rocher

Wed, 06 Jun 2007 07:19:43 -0700

nteresting, though I find that in the beginning it seems like a good idea to extend BuilderSupport, but later you realise it is too limiting and overriding invokeMethod is far more flexible as far as the syntax options you have available to you

It of course depends on the DSL you are writing

Cheers Graeme

BlackBox : Groovy DSL steps

Groovy DSL steps

Requirements

User

Has this problem been solved before?

How did they solve it?

How closely does that solution match our requirements?

What can we take away from that solution?

What kind of DSL do you want overall?

Prototype, prototype, prototype

"Groovy DSL steps" by Leo

"Groovy DSL steps" by Warner Onstine

"Groovy DSL steps" by Warner Onstine

"Groovy DSL steps" by Graeme Rocher