Pouring over Grails code (and the gorm-jpa plugin code). Chime in if you’re interested in helping as this is secondary to what I want to get done. I’ll throw some code up to github soon (maybe tomorrow with an early, early version).

Posted via web from BlackBox Post

Here is the main issue Maven is designed to handle one language only for its compilation – primarily Java. This is rapidly starting to fall apart now that the JVM has some serious contenders besides Java, namely Groovy, JRuby, Jython, and Scala. There are some hackish things for getting Maven to deal with these, however it still only has one “true” source directory for compilation. A case in point is a mixed Java/Groovy project – trying to get IntelliJ to respect the fact that I want not one, but two main source directories is a pain in the ass as I have to either tell it not to sync everytime I launch or I have to refix the source directories each time I update the project.

This is what triggered this post – what happens when I want to do some Flex programming with some Java, ActionScript and MXML, all of these are true source files (and in Maven specfic directories). I’m sure some Mavenite is going to say that these should be separate projects, but in alot of cases they aren’t and it doesn’t make sense to actually separate them into sub-projects. And I’m equally as sure that some Ant-ite is going to come along and tell me to use Ant – I won’t. Ant is stuck in the past and I won’t use it unless I absolutely have to. I like the “imposed” structure that Maven offers. I can walk up to any project and know where to find stuff instead of blindly hunting around for hours trying to piece it together like some giant source-code jigsaw puzzle, blech.

So, I’m at cross-roads, what should I do? There is Ivy (for dependency management -which they got from Maven natch), but that requires Ant, blech. There’s also Gant, ok, this is better, but still Ant. GMaven, but I’m right back where I started.

Enter, Gradle, a new Groovy-based build system that offers alot of the benefits of Maven without locking you in and the flexibility of Ant. I’m not 100% sold on it yet, but I’m going to give it a shot for a relatively complex project that will have a fair amount of Java, Groovy and other fun resource files. It looks interesting in that all the build files are actually Groovy scripts that are a DSL (like Gant files). Here’s a short example with some dependencies (taken directly from their examples in the download).

import org.gradle.api.tasks.util.FileSet

usePlugin('groovy')

group = 'org.gradle'
version = '1.0'

sourceCompatibility = 1.5
targetCompatibility = 1.5

dependencies {
   clientModule(['groovy'], ":groovy-all:1.6-beta-1") {
       dependency(":commons-cli:1.0")
   }
   compile project(':groovycDetector')
   testCompile ":junit:4.4"
}

compile {
   exclude('**/Exclude.java')
   groovyExclude('**/ExcludeGroovy.groovy')
   groovyJavaExclude('**/ExcludeGroovyJava.java')
}

manifest.mainAttributes(myprop: 'myvalue')
metaInf << new FileSet(new File(srcRoot, 'metaInfFiles'))

test.options.systemProperties['org.gradle.integtest.buildDir'] = buildDir.absolutePath

I will try and keep everyone up to date as I continue to investigate this new build system. They also state that they are going to be supporting polyglot programming in the near future which I’m looking forward to.

• Groovy Higher Order Messaging (HOM)
• Math DSL Part 1 and Part 2
• GroovyLab
• Json-lib with Groovy
• Grails’ Spring Bean DSL
• Grails’ Dynamic Finders
• Grails’ Criteria DSL
• Grails’ Domain Class Validation DSL
• JParsec’s Groovy interface
• Guillaume’s Unit Manipulation DSL
• AntBuilder (as well as XMLBuilder, SwingBuilder, etc. – builders, essentially)
• Ken Barclay’s GParsec
• Human readable date manipulation such as in Guillaume’s example
• Tiago Antão’s Malaria Drug DSL
• GSpec Behavior Driven Development

I know there are more out there but this is a good starting point for sure.

Now, on to the actual categories of DSL “types”

I have been paying attention to Martin Fowler’s new DSL book in which he specifies several types of Internal DSLs. I’m not 100% on board with all of these yet, but here they are (of course these may change as this is a work in progress, so if he changes them it isn’t my fault ):

• Expression Builder: A layer that provides a fluent interface over a regular API
• Function Sequence: A combination of function calls as a sequence of statements.
• Nested Function: Compose functions by nesting function calls as arguments of of other calls.
• Method Chaining: Make modifier methods return the host object so that multiple modifiers can be invoked in a single expression.
• Object Scoping: Put DSL code in a subclass of a class that provdes the DSL vocabulary.
• Closure:
  • Nested Closure: Express statement sub-elements of a function call by putting them into a closure in an argument.
• Literal Collection Expression: Form language expressions using literal collection syntax
• Dynamic Reception: Handle messages without defining them in the code
• Annotation: Data about program elements, such as classes and methods, which can be processed during compilation or execution.
• Macro: Define DSL expressions as macro definitions
• Parse Tree Manipulation: Capture the parse tree of a code fragment to manipulate it with DSL processing code.

Ok, so these are the basic categories as Martin has defined them. How do these map up to the tools and techniques we have in Groovy?

• Method/Property Missing – a special method that is called when a method can not be found allowing a developer to intercept the call
  • Maps to Dynamic Reception fairly well
• Categories – allows new methods to be added to any class at runtime
  • Could fit into Object Scoping or maybe a new one, Duck Typing? – not sure
• ExpandoMetaClass – a dynamically expandable bean
  • Again Dynamic Reception
• Closures/Nested Closures
  • Closures/Nested Closures
• Relaxed rules on the use of parentheses
  • Hmm, not sure where this would fit
• Built-in list and map syntax
  • I believe this is Literal Collection Expression
• Ability to dynamically add methods to an interface and have concrete classes also have the methods
  • Again, this sounds like Dynamic Reception
• Operator overloading
  • Hmm, again I’m not sure where this fits
• Annotations
  • Annotation

First up we have our builders (AntBuilder, XMLBuilder, Criteria Builder, etc.) these essentially mimic some other existing structure (an Ant build file, XML file, or SQL syntax). What patterns do these use? Just off the top of my head I think I see:

• Expression Builder
• Nested Closures
• Dynamic Reception
• Literal Collection Expression

JSON-lib’s Groovy interface has the following:

• Literal Collection Expression
• Dynamic Reception
• Operator Overloading (as)

My Groovy Higher Order Messaging DSL has:

• Dynamic Reception
• Nested Closures

My Groovy Math DSL has:

• Dynamic Reception
• Nested Closures (or possibly what would be considered Function Sequence in some cases, even though they aren’t functions but it might fit)
• Operator Overloading

GroovyLab has:

• Expression Builder
• Probably has more just don’t have time to dig into the code – I’m just looking at the quick examples)
• Operator Overloading

Guillaume’s Unit Manipulation DSL has:

• Expression Builder
• Dynamic Reception
• Closure/Nested Closure
• Operator Overloading

Ken Barclay’s GParsec has:

• Dynamic Reception
• Parse Tree Manipulation? (not sure if this qualifies for this or not)

GroovyRestlet DSL has:

• Literal Collection Expression
• Closure/Nested Closure
• Function Sequence

GSpec has:

• Closure/Nested Closure
• Function Sequence
• Dynamic Reception

Grails’ Domain Class Validation DSL has:

• Literal Colleciton Expression
• Closure
• Macros?

Tiago Antão’s Malaria Drug DSL has:

• Object Scoping
• Dynamic Reception

Human readable date modification has:

• Object Scoping
• Operator Overloading

Now, one of the big things that I think Fowler is missing here (and one that you can see several DSLs take advantage of) is Operator Overloading/Overriding. This, I think, is a key technique in DSLs and one that can make a DSL go from good to great. It’s all about letting the user think in the context they are in – be it math matrices, sql queries, file and string manipulation (overriding the << or >> operators). It’s all about the user’s context and making it easy for them to “think” in their domain and not have to worry about the language implementation (I use it quite heavily in my Math DSL to implement matrix operators for example).

I know I didn’t hit on all the DSLs I mentioned, but will try and come up with a wiki or something that people can contribute back to. In the mean time sound off in the comments if you have a DSL that I didn’t mention, or you feel my categorization is wrong (and why). I’m also curious if there are examples of some of the ones I didn’t touch on such as Annotations. This is an initial stab at this and as I said earlier I’m not 100% on board with how Fowler has organized things but I think it’s a good start.

One possibility I’ve been tossing around is to turn it into a techniques or cookbook-style book. I’ve been reading through “Designing Interfaces” by Jennifer Tidwell. In this book she organizes the interface solutions into different sections and goes through each one with an example and what it solves and when to use it. I really like this idea (but others might not). Originally my book was a “How do I get started designing DSLs?” and “What in the world are they and why should I care?”.

Another thing I’ve been thinking about is ditching the Java part and making it Groovy only. This appeals to me because I feel that Groovy is much cleaner to implement an Internal DSL than Java is. Can you do DSLs in Java? Of course you can, but I’m not sure that I can give it the best treatment because it isn’t what I’m passionate about.

So, sound off in the comments, let me know what topics you’d like to see covered, what challenges have you faced in writing a DSL (or API/Framework) that you would have liked someone to trailblaze for you to show the dead-ends first?

I honestly can’t say how happy I am at this and will definitely test this functionality out with my Math DSL that I am working on.

I have one project I’m working on right now (not ready to talk yet about as I want more code in place before I unveil it, ) but this might be useful. Say that you wanted to duck-type your object for specific class instances. Overriding as may be one place to do it.

When would this make sense? Personally I think something like this makes the most sense in a library that provides some kind of functionality to a bunch of classes (rather than a one-off case). But I’m curious to see how others have used this functionality, would help to shed some light on other uses I hadn’t thought of.

• Create a new MetaClass using ExpandoMetaClass
• Define a new class that understood what row and matrix it was going to operate on (so it can have its own operator overloading)

Yes it was a little bit of extra work, but worth it. I also implemented my or operator overloading so that I can specify augmented matrices in three formats:

• A|I for the identity matrix (only works on square matrices)
• A|0 for the zero column, used for solving the set of linear equations
• A|b for a vector column, also used for solving the set of linear equations

As always it helps to learn from others’ mistakes (and oddities) so I’ll share mine along the road:

• When dealing with EMC (ExpandoMetaClass) watch out for the use of the this keyword
• Closure delegates don’t always act the way you would hope when extending GroovyObjectSupport (at least I couldn’t get it to work right which is one reason why I switched to EMC)
• Overriding the or operator works fine until you get to an object that has overridden the equals operator. In this case Groovy kept trying to find out if my Matrix was equal to my MatrixColumn. Once I figured this out I just returned false if the MatrixColumn was being compared against a Matrix then it worked as expected.

The next part I want to add is the ability to track the EROs through Elementary Matrices. These are matrices that start out as 1s along the diagonal and they get the same EROs applied to them (this will be useful further down the line for other operations).

So far I’ve implemented the following into my matrix math DSL:

• Matrix/Row equality – allows you to test if two matrices or rows are equal (i.e. – contain the same ordered set of numbers)
• Matrix addition/subtraction – You can add two matrices together if they are the same dimensions (2×2, 2×3, whatever, as long as the match in row and columns)
• Matrix/Row scalar multiplication – You can multiply a scalar value against a row or matrix
• Matrix multiplication – You can multiply two matrices together if the first matrix A’s (say 2×3) columns match B’s rows of the matrix you are multiplying against (say 3×2) which will produce a 2×2 matrix. If you did BxA then you would end up with a new matrix of 3×3
• Matrix powers – If the matrix is square (say 3×3) you can multiply it by itself to get the power of the matrix
• Left-shift operation – be able to dynamically create a Row or Matrix by adding new elements to the end

I still have a lot of work which is laid out on my Trac site for GroovyMath which leads me to my next conundrum.

When dealing with matrices you need to do a lot of Reduced Echelon Form (REF) or Row Reduced Echelon Form (RREF) this consists of taking one of two Elementary Row Operations (EROs) and applying it to a row to get a matrix in the form of a diagonal (all 1s down the upper-left to lower-right diagonal with 0s everywhere else) or getting it into an upper-diagonal form (0s below the diagonal and integers everywhere else). The two EROs that you can consist of a) row interchange, where you can exchange one row with another; and b) row addition, where you can add one row to another multiplying that row by a scalar. What I would like to be able to do is something like this:

def A = new Matrix(//rows go in here)
A.r1 + A.r2*2 //which will add 2 times the second row and add it to the first and replace it
A.r1 % A.r2 //which will exchange rows 1 and 2 with each other

Another thought I had was to do this in some kind of “rowOperations” closure:

def A = new Matrix(//rows go in here)
A.rowOperations {
    r1 + r2*2
    r1 % r2
}

Which definitely reads better than the first (and of course that’s all part of a DSL how well does it translate the native tongue into code). But the issue that I have with either of these approaches (maybe, still ironing out the second in my head) is that the properties r1 and r2 don’t exist. These property accessors will need to be intercepted. Ok, not a big deal. But if I choose to implement this in MatrixRow (i.e. – actually return the row itself) it doesn’t allow me to modify the originating matrix (A in this case) only change the row. For the addition problem I think I can get around it as it is executing in a closure, but the row interchange is an issue. The individual rows don’t know they’re a part of a matrix and I’m not sure that knowing it allows them to do anything.

Anyways, just wrestling with this concept right now as it is key to moving on to the more advanced functionality in my matrix math. Any suggestions are more than welcome. One new feature that I really like will be overriding the or operator to provide Augmented Matrices (another key part). Normally you define an augmented matrix in one of three ways:

• A|0 – add a column of all zeros
• A|b – add a vector as the augmented matrix
• A|I – add an Identity matrix

The first two are used to solve the set of linear equations and a bunch of other things. The second is used to find the inverse of the matrix as you transform through EROs the matrix A so that it equals I (the identity matrix) and then the side that “I” was on now equals the inverse of A. I would say stop me if I’m boring you but it’s probably too late for that. I just wanted to point out how cool it was I can override the or operator to do something completely different that makes perfect sense in my given domain.

Now of course I’m a little embarrassed that I announced the book before it was done, but that’s what happens sometimes. Not sure at this point whether or not the book will continue, taking some time to regroup/rethink things and see if it is something that I want to push forward with.

I will keep everyone up to date here on the progress of things once I’ve decided what to do. In the meantime I am going to get back to some of my back-burner projects (a few of which have some DSLs in them .

The book will cover both Java and Groovy techniques for writing DSLs, as well as have some general guidelines for writing DSLs. We also have a chapter on ANTLR and JavaCC for writing external DSLs using Java. I tried to cover as much as I could of Groovy meta-programming and some of the additional capabilities of Groovy that make it a good language to write an internal DSL on top of, but there’s only so much space .

We just sent off the book for peer review, which means its about half-way finished, maybe a little more. We are currently shooting for a release date sometime early next year, which looks very doable at this point, but I expect a ton of edits to come back from this first review.

On a final note I just want to say what a pleasure it has been working with the PragProg crew. I love, love, love their book build system and my editor Susannah rocks. I have had a blast working with them so far on this book and am looking forward to finishing this up and getting it published.