BlackBox

I have been paying attention to Martin Fowler’s new DSL book in which he specifies several types of Internal DSLs. I’m not 100% on board with all of these yet, but here they are (of course these may change as this is a work in progress, so if he changes them it isn’t my fault :P):

• Expression Builder: A layer that provides a fluent interface over a regular API
• Function Sequence: A combination of function calls as a sequence of statements.
• Nested Function: Compose functions by nesting function calls as arguments of of other calls.
• Method Chaining: Make modifier methods return the host object so that multiple modifiers can be invoked in a single expression.
• Object Scoping: Put DSL code in a subclass of a class that provdes the DSL vocabulary.
• Closure:
  • Nested Closure: Express statement sub-elements of a function call by putting them into a closure in an argument.
• Literal Collection Expression: Form language expressions using literal collection syntax
• Dynamic Reception: Handle messages without defining them in the code
• Annotation: Data about program elements, such as classes and methods, which can be processed during compilation or execution.
• Macro: Define DSL expressions as macro definitions
• Parse Tree Manipulation: Capture the parse tree of a code fragment to manipulate it with DSL processing code.

Ok, so these are the basic categories as Martin has defined them. How do these map up to the tools and techniques we have in Groovy?

• Method/Property Missing - a special method that is called when a method can not be found allowing a developer to intercept the call
  • Maps to Dynamic Reception fairly well
• Categories - allows new methods to be added to any class at runtime
  • Could fit into Object Scoping or maybe a new one, Duck Typing? - not sure
• ExpandoMetaClass - a dynamically expandable bean
  • Again Dynamic Reception
• Closures/Nested Closures
  • Closures/Nested Closures
• Relaxed rules on the use of parentheses
  • Hmm, not sure where this would fit
• Built-in list and map syntax
  • I believe this is Literal Collection Expression
• Ability to dynamically add methods to an interface and have concrete classes also have the methods
  • Again, this sounds like Dynamic Reception
• Operator overloading
  • Hmm, again I’m not sure where this fits
• Annotations
  • Annotation

First up we have our builders (AntBuilder, XMLBuilder, Criteria Builder, etc.) these essentially mimic some other existing structure (an Ant build file, XML file, or SQL syntax). What patterns do these use? Just off the top of my head I think I see:

• Expression Builder
• Nested Closures
• Dynamic Reception
• Literal Collection Expression

JSON-lib’s Groovy interface has the following:

• Literal Collection Expression
• Dynamic Reception
• Operator Overloading (as)

My Groovy Higher Order Messaging DSL has:

• Dynamic Reception
• Nested Closures

My Groovy Math DSL has:

• Dynamic Reception
• Nested Closures (or possibly what would be considered Function Sequence in some cases, even though they aren’t functions but it might fit)
• Operator Overloading

GroovyLab has:

• Expression Builder
• Probably has more just don’t have time to dig into the code - I’m just looking at the quick examples)
• Operator Overloading

Guillaume’s Unit Manipulation DSL has:

• Expression Builder
• Dynamic Reception
• Closure/Nested Closure
• Operator Overloading

Ken Barclay’s GParsec has:

• Dynamic Reception
• Parse Tree Manipulation? (not sure if this qualifies for this or not)

GroovyRestlet DSL has:

• Literal Collection Expression
• Closure/Nested Closure
• Function Sequence

GSpec has:

• Closure/Nested Closure
• Function Sequence
• Dynamic Reception

Grails’ Domain Class Validation DSL has:

• Literal Colleciton Expression
• Closure
• Macros?

Tiago Antão’s Malaria Drug DSL has:

• Object Scoping
• Dynamic Reception

Human readable date modification has:

• Object Scoping
• Operator Overloading

Now, one of the big things that I think Fowler is missing here (and one that you can see several DSLs take advantage of) is Operator Overloading/Overriding. This, I think, is a key technique in DSLs and one that can make a DSL go from good to great. It’s all about letting the user think in the context they are in - be it math matrices, sql queries, file and string manipulation (overriding the << or >> operators). It’s all about the user’s context and making it easy for them to “think” in their domain and not have to worry about the language implementation (I use it quite heavily in my Math DSL to implement matrix operators for example).

I know I didn’t hit on all the DSLs I mentioned, but will try and come up with a wiki or something that people can contribute back to. In the mean time sound off in the comments if you have a DSL that I didn’t mention, or you feel my categorization is wrong (and why). I’m also curious if there are examples of some of the ones I didn’t touch on such as Annotations. This is an initial stab at this and as I said earlier I’m not 100% on board with how Fowler has organized things but I think it’s a good start.

• When dealing with EMC (ExpandoMetaClass) watch out for the use of the this keyword
• Closure delegates don’t always act the way you would hope when extending GroovyObjectSupport (at least I couldn’t get it to work right which is one reason why I switched to EMC)
• Overriding the or operator works fine until you get to an object that has overridden the equals operator. In this case Groovy kept trying to find out if my Matrix was equal to my MatrixColumn. Once I figured this out I just returned false if the MatrixColumn was being compared against a Matrix then it worked as expected.

The next part I want to add is the ability to track the EROs through Elementary Matrices. These are matrices that start out as 1s along the diagonal and they get the same EROs applied to them (this will be useful further down the line for other operations).

When dealing with matrices you need to do a lot of Reduced Echelon Form (REF) or Row Reduced Echelon Form (RREF) this consists of taking one of two Elementary Row Operations (EROs) and applying it to a row to get a matrix in the form of a diagonal (all 1s down the upper-left to lower-right diagonal with 0s everywhere else) or getting it into an upper-diagonal form (0s below the diagonal and integers everywhere else). The two EROs that you can consist of a) row interchange, where you can exchange one row with another; and b) row addition, where you can add one row to another multiplying that row by a scalar. What I would like to be able to do is something like this:

def A = new Matrix(//rows go in here)
A.r1 + A.r2*2 //which will add 2 times the second row and add it to the first and replace it
A.r1 % A.r2 //which will exchange rows 1 and 2 with each other

Another thought I had was to do this in some kind of “rowOperations” closure:

def A = new Matrix(//rows go in here)
A.rowOperations {
    r1 + r2*2
    r1 % r2
}

Which definitely reads better than the first (and of course that’s all part of a DSL how well does it translate the native tongue into code). But the issue that I have with either of these approaches (maybe, still ironing out the second in my head) is that the properties r1 and r2 don’t exist. These property accessors will need to be intercepted. Ok, not a big deal. But if I choose to implement this in MatrixRow (i.e. - actually return the row itself) it doesn’t allow me to modify the originating matrix (A in this case) only change the row. For the addition problem I think I can get around it as it is executing in a closure, but the row interchange is an issue. The individual rows don’t know they’re a part of a matrix and I’m not sure that knowing it allows them to do anything.

Anyways, just wrestling with this concept right now as it is key to moving on to the more advanced functionality in my matrix math. Any suggestions are more than welcome. One new feature that I really like will be overriding the or operator to provide Augmented Matrices (another key part). Normally you define an augmented matrix in one of three ways:

• A|0 - add a column of all zeros
• A|b - add a vector as the augmented matrix
• A|I - add an Identity matrix

The first two are used to solve the set of linear equations and a bunch of other things. The second is used to find the inverse of the matrix as you transform through EROs the matrix A so that it equals I (the identity matrix) and then the side that “I” was on now equals the inverse of A. I would say stop me if I’m boring you but it’s probably too late for that. I just wanted to point out how cool it was I can override the or operator to do something completely different that makes perfect sense in my given domain.

Requirements

So, what are my requirements for my main two DSLs - QueryBuilder and ViewBuilder (this doesn’t exist yet, but starting on the planning right now). For QueryBuilder I wanted a clean syntax that makes it easy for developers to create constraints/restrictions on a database query. Ultimately the query could be executed by Hibernate, JPA, iBatis, whatever, so it should be ORM-agnostic. This actually isn’t too hard as they all have this concept of a criteria on an object/table and restrictions for that query. And the same goes for ViewBuilder, I want an HTML-like tree syntax that I can specify components in as well as attributes that will be applied to the view when it is built for me.

User

This one is easy, I am targeting Web application developers, initially I am targeting Tapestry Web application developers, but since the framework will be pluggable (probably around the 2.0 time) you can specify what specific pieces you want to use (Tapestry/JSF + Hibernate/JPA + Spring 1.x/2.x/Guice, etc.).

Has this problem been solved before?

In the case of the QueryBuilder, yes. Grails currently has an implementation called HibernateCriteriaBuilder. This has a nice syntax for specifying criteria and restrictions on that criteria. As for the ViewBuilder I don’t think this has been solved yet, but I will be looking. There is something similar in Groovy’s core for creating arbitrary HTML, which is close but not exactly what I’m looking for as I will need to generate more than just HTML from it.

How did they solve it?

Both of these were solved using Groovy’s BuilderSupport object which allows you to build up arbitrary “nodes” from a collection of method, property and closure calls. Very powerful and a little difficult to wrap your head around the first 10 times you look at it ;-) (alright, maybe it was just me, but that took a bit to fully understand what the hell it was doing).

How closely does that solution match our requirements?

First, for the HibernateCriteriaBuilder it actually will build up a real, live Hibernate criteria and use it, which unfortunately is not what I want. I need to generate the Java code that will come out of it. And the MarkupBuilder gets me part of the way there but not quite what I’m looking for.

What can we take away from that solution?

I decided to take away the syntax from HibernateCriteriaBuilder and looked at how they implemented it to see what I could do with that. And for the ViewBuilder I am going to extend BuilderSupport itself and use that to build up my nodes of whatever objects (so more than just plain HTML text, actually objects that represent things).

What kind of DSL do you want overall?

Definitely shooting for internal I think, but that doesn’t preclude having some kind of external DSL for the class creation stuff (that was a giant pain to do in Groovy/Java - just not all of the information I actually needed). So, having some kind of custom language for defining models could definitely come in handy down the road, but I haven’t decided anything yet. Plus, as I learn more Groovy and get feedback/suggestions I may stumble across something that simplifies my design greatly.

Prototype, prototype, prototype

This, aside from requirements and research is probably the most valuable step. Once you decide what you are going to do, how in the world do you do it? Especially if you’re dealing with Meta-programming (which can be a hard concept to wrap your head around)?

Here is where my experience will hopefully prove valuable to someone out there, here are the steps that I took when developing my QueryBuilder:

1. Write the Unit tests first, start with the simplest test you can think of for how your language should work
2. Run it - yeah I know its going to fail, but prove it to yourself ;-)
3. Make it go green - whether you go ahead and create a simple class that has one method in it that does what you want do that
4. Write another 2 tests - think about them, how are they going to change that class you just wrote, how much more complex is it going to make the logic you need to write to make sure you do what you expect.
5. Fix the first test - is it time to extend DelegatingMetaClass yet? Or maybe try something simpler like a Category? Maybe the new ExpandoMetaClass will work. Try a few of them out, but I don’t recommend you dive right into extending BuilderSupport, not yet, not unless you have already used it before and understand fully what it does. Throw in lots of println "calling so and so with these arguments". These help identify that what you are expecting is actually happening while you write your DSL.
6. Did that work? Good, write it so the next test succeeds.
7. Continue doing 4, 5, and 6 until you are happy and ready to show it to someone
8. Let them play with it, get their feedback. Rinse, repeat, refactor. (Here is where I would suggest you revisit the BuilderSupport idea, does it make sense? Remember not everything is an arbitrary nested tree, so not everything will benefit from using this structure.)

Well, I hope that this is helpful to someone ;-). I really have gotten the DSL bug and (mostly) enjoy writing them. If you run into a weird error, throw it out to the Groovy list, someone should be able to help you out.

Addendum

I meant to add this in when I first started thinking about this post. When I started writing my QueryBuilder I didn’t start out with it generating anything but simple select strings. This helped me visualize the proper nesting of things without getting too complex. Once that was working right I changed my unit tests to expect the actual Java code I wanted back and changed the logic of what the DSL output.