BlackBox : Tag groovy

Is Maven going away?

Warner Onstine — Sun, 06 Jul 2008 07:49:33 -0700

First, let me say, I like Maven - a lot. I’ve liked the idea, if not the execution, since 1.0 (yes, I know there is some serious bad blood for the implementation of 1.0, and yes I still liked it). But I’ve recently run into some irritating problems with Maven that would again, I think, require a major reworking of what it does.

Here is the main issue Maven is designed to handle one language only for its compilation - primarily Java. This is rapidly starting to fall apart now that the JVM has some serious contenders besides Java, namely Groovy, JRuby, Jython, and Scala. There are some hackish things for getting Maven to deal with these, however it still only has one “true” source directory for compilation. A case in point is a mixed Java/Groovy project - trying to get IntelliJ to respect the fact that I want not one, but two main source directories is a pain in the ass as I have to either tell it not to sync everytime I launch or I have to refix the source directories each time I update the project.

This is what triggered this post - what happens when I want to do some Flex programming with some Java, ActionScript and MXML, all of these are true source files (and in Maven specfic directories). I’m sure some Mavenite is going to say that these should be separate projects, but in alot of cases they aren’t and it doesn’t make sense to actually separate them into sub-projects. And I’m equally as sure that some Ant-ite is going to come along and tell me to use Ant - I won’t. Ant is stuck in the past and I won’t use it unless I absolutely have to. I like the “imposed” structure that Maven offers. I can walk up to any project and know where to find stuff instead of blindly hunting around for hours trying to piece it together like some giant source-code jigsaw puzzle, blech.

So, I’m at cross-roads, what should I do? There is Ivy (for dependency management -which they got from Maven natch), but that requires Ant, blech. There’s also Gant, ok, this is better, but still Ant. GMaven, but I’m right back where I started.

Enter, Gradle, a new Groovy-based build system that offers alot of the benefits of Maven without locking you in and the flexibility of Ant. I’m not 100% sold on it yet, but I’m going to give it a shot for a relatively complex project that will have a fair amount of Java, Groovy and other fun resource files. It looks interesting in that all the build files are actually Groovy scripts that are a DSL (like Gant files). Here’s a short example with some dependencies (taken directly from their examples in the download).

import org.gradle.api.tasks.util.FileSet

usePlugin('groovy')

group = 'org.gradle'
version = '1.0'

sourceCompatibility = 1.5
targetCompatibility = 1.5

dependencies {
   clientModule(['groovy'], ":groovy-all:1.6-beta-1") {
       dependency(":commons-cli:1.0")
   }
   compile project(':groovycDetector')
   testCompile ":junit:4.4"
}

compile {
   exclude('**/Exclude.java')
   groovyExclude('**/ExcludeGroovy.groovy')
   groovyJavaExclude('**/ExcludeGroovyJava.java')
}

manifest.mainAttributes(myprop: 'myvalue')
metaInf << new FileSet(new File(srcRoot, 'metaInfFiles'))

test.options.systemProperties['org.gradle.integtest.buildDir'] = buildDir.absolutePath

I will try and keep everyone up to date as I continue to investigate this new build system. They also state that they are going to be supporting polyglot programming in the near future which I’m looking forward to.

Groovy DSL roundup

Warner Onstine — Thu, 24 Apr 2008 09:50:00 -0700

This is my first attempt at categorizing some of the DSLs I’ve run across into some kind of logical grouping. First I’ll list out some Groovy DSL examples (sound off in the comments for those I’m not aware of, the more the better). I’ll go through the DSLs first and then look at the possible categories after the break. And the first two are mine that I created

Groovy Higher Order Messaging (HOM)
Math DSL Part 1 and Part 2
GroovyLab
Json-lib with Groovy
Grails’ Spring Bean DSL
Grails’ Dynamic Finders
Grails’ Criteria DSL
Grails’ Domain Class Validation DSL
JParsec’s Groovy interface
Guillaume’s Unit Manipulation DSL
AntBuilder (as well as XMLBuilder, SwingBuilder, etc. - builders, essentially)
Ken Barclay’s GParsec
Human readable date manipulation such as in Guillaume’s example
Tiago Antão’s Malaria Drug DSL
GSpec Behavior Driven Development

I know there are more out there but this is a good starting point for sure.

Now, on to the actual categories of DSL “types”

I have been paying attention to Martin Fowler’s new DSL book in which he specifies several types of Internal DSLs. I’m not 100% on board with all of these yet, but here they are (of course these may change as this is a work in progress, so if he changes them it isn’t my fault :P):

Expression Builder: A layer that provides a fluent interface over a regular API
Function Sequence: A combination of function calls as a sequence of statements.
Nested Function: Compose functions by nesting function calls as arguments of of other calls.
Method Chaining: Make modifier methods return the host object so that multiple modifiers can be invoked in a single expression.
Object Scoping: Put DSL code in a subclass of a class that provdes the DSL vocabulary.
Closure:
- Nested Closure: Express statement sub-elements of a function call by putting them into a closure in an argument.
Literal Collection Expression: Form language expressions using literal collection syntax
Dynamic Reception: Handle messages without defining them in the code
Annotation: Data about program elements, such as classes and methods, which can be processed during compilation or execution.
Macro: Define DSL expressions as macro definitions
Parse Tree Manipulation: Capture the parse tree of a code fragment to manipulate it with DSL processing code.

Ok, so these are the basic categories as Martin has defined them. How do these map up to the tools and techniques we have in Groovy?

Method/Property Missing - a special method that is called when a method can not be found allowing a developer to intercept the call
- Maps to Dynamic Reception fairly well
Categories - allows new methods to be added to any class at runtime
- Could fit into Object Scoping or maybe a new one, Duck Typing? - not sure
ExpandoMetaClass - a dynamically expandable bean
- Again Dynamic Reception
Closures/Nested Closures
- Closures/Nested Closures
Relaxed rules on the use of parentheses
- Hmm, not sure where this would fit
Built-in list and map syntax
- I believe this is Literal Collection Expression
Ability to dynamically add methods to an interface and have concrete classes also have the methods
- Again, this sounds like Dynamic Reception
Operator overloading
- Hmm, again I’m not sure where this fits
Annotations
- Annotation

First up we have our builders (AntBuilder, XMLBuilder, Criteria Builder, etc.) these essentially mimic some other existing structure (an Ant build file, XML file, or SQL syntax). What patterns do these use? Just off the top of my head I think I see:

Expression Builder
Nested Closures
Dynamic Reception
Literal Collection Expression

JSON-lib’s Groovy interface has the following:

Literal Collection Expression
Dynamic Reception
Operator Overloading (as)

My Groovy Higher Order Messaging DSL has:

Dynamic Reception
Nested Closures

My Groovy Math DSL has:

Dynamic Reception
Nested Closures (or possibly what would be considered Function Sequence in some cases, even though they aren’t functions but it might fit)
Operator Overloading

GroovyLab has:

Expression Builder
Probably has more just don’t have time to dig into the code - I’m just looking at the quick examples)
Operator Overloading

Guillaume’s Unit Manipulation DSL has:

Expression Builder
Dynamic Reception
Closure/Nested Closure
Operator Overloading

Ken Barclay’s GParsec has:

Dynamic Reception
Parse Tree Manipulation? (not sure if this qualifies for this or not)

GroovyRestlet DSL has:

Literal Collection Expression
Closure/Nested Closure
Function Sequence

GSpec has:

Closure/Nested Closure
Function Sequence
Dynamic Reception

Grails’ Domain Class Validation DSL has:

Literal Colleciton Expression
Closure
Macros?

Tiago Antão’s Malaria Drug DSL has:

Object Scoping
Dynamic Reception

Human readable date modification has:

Object Scoping
Operator Overloading

Now, one of the big things that I think Fowler is missing here (and one that you can see several DSLs take advantage of) is Operator Overloading/Overriding. This, I think, is a key technique in DSLs and one that can make a DSL go from good to great. It’s all about letting the user think in the context they are in - be it math matrices, sql queries, file and string manipulation (overriding the << or >> operators). It’s all about the user’s context and making it easy for them to “think” in their domain and not have to worry about the language implementation (I use it quite heavily in my Math DSL to implement matrix operators for example).

I know I didn’t hit on all the DSLs I mentioned, but will try and come up with a wiki or something that people can contribute back to. In the mean time sound off in the comments if you have a DSL that I didn’t mention, or you feel my categorization is wrong (and why). I’m also curious if there are examples of some of the ones I didn’t touch on such as Annotations. This is an initial stab at this and as I said earlier I’m not 100% on board with how Fowler has organized things but I think it’s a good start.

Thinking about picking the Groovy DSL book up again

Warner Onstine — Wed, 26 Mar 2008 10:30:00 -0700

But before I do I thought I’d ask everyone out there (who are interested in such a book), what would you like to see in it?

One possibility I’ve been tossing around is to turn it into a techniques or cookbook-style book. I’ve been reading through “Designing Interfaces” by Jennifer Tidwell. In this book she organizes the interface solutions into different sections and goes through each one with an example and what it solves and when to use it. I really like this idea (but others might not). Originally my book was a “How do I get started designing DSLs?” and “What in the world are they and why should I care?”.

Another thing I’ve been thinking about is ditching the Java part and making it Groovy only. This appeals to me because I feel that Groovy is much cleaner to implement an Internal DSL than Java is. Can you do DSLs in Java? Of course you can, but I’m not sure that I can give it the best treatment because it isn’t what I’m passionate about.

So, sound off in the comments, let me know what topics you’d like to see covered, what challenges have you faced in writing a DSL (or API/Framework) that you would have liked someone to trailblaze for you to show the dead-ends first?

Groovy Roadmap

Warner Onstine — Thu, 20 Dec 2007 16:36:36 -0700

So Guillaume posted the current road map to the list a couple of days ago and I put in my .02 for fixing the (in my eyes) broken operator overloading. And the bug I really wanted is now marked to be fixed in 1.6!

I honestly can’t say how happy I am at this and will definitely test this functionality out with my Math DSL that I am working on.

Overriding the 'as' operator

Warner Onstine — Thu, 06 Dec 2007 09:40:25 -0700

A friend of mine sent me a link to Charles Nutter’s blog on implementing interfaces using the as operator. While I was explaining how it worked to him I remembered that like many operators you can also override this one using asType(). This got us to thinking when would you use this functionality?

I have one project I’m working on right now (not ready to talk yet about as I want more code in place before I unveil it, :-P) but this might be useful. Say that you wanted to duck-type your object for specific class instances. Overriding as may be one place to do it.

When would this make sense? Personally I think something like this makes the most sense in a library that provides some kind of functionality to a bunch of classes (rather than a one-off case). But I’m curious to see how others have used this functionality, would help to shed some light on other uses I hadn’t thought of.

Math DSL Part Deux - Elementary Row Operations (EROs)

Warner Onstine — Mon, 26 Nov 2007 11:59:24 -0700

After sending out my query to the groovy user list I got back some ideas (no offense to Yann, just something I need to tackle on my own, so I didn’t look at your solution). What I ended up doing was two-fold:

Create a new MetaClass using ExpandoMetaClass
Define a new class that understood what row and matrix it was going to operate on (so it can have its own operator overloading)

Yes it was a little bit of extra work, but worth it. I also implemented my or operator overloading so that I can specify augmented matrices in three formats:

A|I for the identity matrix (only works on square matrices)
A|0 for the zero column, used for solving the set of linear equations
A|b for a vector column, also used for solving the set of linear equations

As always it helps to learn from others’ mistakes (and oddities) so I’ll share mine along the road:

When dealing with EMC (ExpandoMetaClass) watch out for the use of the this keyword
Closure delegates don’t always act the way you would hope when extending GroovyObjectSupport (at least I couldn’t get it to work right which is one reason why I switched to EMC)
Overriding the or operator works fine until you get to an object that has overridden the equals operator. In this case Groovy kept trying to find out if my Matrix was equal to my MatrixColumn. Once I figured this out I just returned false if the MatrixColumn was being compared against a Matrix then it worked as expected.

The next part I want to add is the ability to track the EROs through Elementary Matrices. These are matrices that start out as 1s along the diagonal and they get the same EROs applied to them (this will be useful further down the line for other operations).

Math DSL beginning

Warner Onstine — Fri, 23 Nov 2007 18:11:12 -0700

I decided to start working on this taking one of the examples I worked out for the book for math Sets (partially incomplete due to Groovy’s inability to directly override lessThan, lessThanEquals, greaterThan, greaterThanEquals) and to put into code some of the techniques I was learning in my Linear Algebra class.

So far I’ve implemented the following into my matrix math DSL:

Matrix/Row equality - allows you to test if two matrices or rows are equal (i.e. - contain the same ordered set of numbers)
Matrix addition/subtraction - You can add two matrices together if they are the same dimensions (2x2, 2x3, whatever, as long as the match in row and columns)
Matrix/Row scalar multiplication - You can multiply a scalar value against a row or matrix
Matrix multiplication - You can multiply two matrices together if the first matrix A’s (say 2x3) columns match B’s rows of the matrix you are multiplying against (say 3x2) which will produce a 2x2 matrix. If you did BxA then you would end up with a new matrix of 3x3
Matrix powers - If the matrix is square (say 3x3) you can multiply it by itself to get the power of the matrix
Left-shift operation - be able to dynamically create a Row or Matrix by adding new elements to the end

I still have a lot of work which is laid out on my Trac site for GroovyMath which leads me to my next conundrum.

When dealing with matrices you need to do a lot of Reduced Echelon Form (REF) or Row Reduced Echelon Form (RREF) this consists of taking one of two Elementary Row Operations (EROs) and applying it to a row to get a matrix in the form of a diagonal (all 1s down the upper-left to lower-right diagonal with 0s everywhere else) or getting it into an upper-diagonal form (0s below the diagonal and integers everywhere else). The two EROs that you can consist of a) row interchange, where you can exchange one row with another; and b) row addition, where you can add one row to another multiplying that row by a scalar. What I would like to be able to do is something like this:

def A = new Matrix(//rows go in here)
A.r1 + A.r2*2 //which will add 2 times the second row and add it to the first and replace it
A.r1 % A.r2 //which will exchange rows 1 and 2 with each other

Another thought I had was to do this in some kind of “rowOperations” closure:

def A = new Matrix(//rows go in here)
A.rowOperations {
    r1 + r2*2
    r1 % r2
}

Which definitely reads better than the first (and of course that’s all part of a DSL how well does it translate the native tongue into code). But the issue that I have with either of these approaches (maybe, still ironing out the second in my head) is that the properties r1 and r2 don’t exist. These property accessors will need to be intercepted. Ok, not a big deal. But if I choose to implement this in MatrixRow (i.e. - actually return the row itself) it doesn’t allow me to modify the originating matrix (A in this case) only change the row. For the addition problem I think I can get around it as it is executing in a closure, but the row interchange is an issue. The individual rows don’t know they’re a part of a matrix and I’m not sure that knowing it allows them to do anything.

Anyways, just wrestling with this concept right now as it is key to moving on to the more advanced functionality in my matrix math. Any suggestions are more than welcome. One new feature that I really like will be overriding the or operator to provide Augmented Matrices (another key part). Normally you define an augmented matrix in one of three ways:

A|0 - add a column of all zeros
A|b - add a vector as the augmented matrix
A|I - add an Identity matrix

The first two are used to solve the set of linear equations and a bunch of other things. The second is used to find the inverse of the matrix as you transform through EROs the matrix A so that it equals I (the identity matrix) and then the side that “I” was on now equals the inverse of A. I would say stop me if I’m boring you but it’s probably too late for that. I just wanted to point out how cool it was I can override the or operator to do something completely different that makes perfect sense in my given domain.

DSL book on hold...unfortunately

Warner Onstine — Tue, 30 Oct 2007 10:20:44 -0700

Well we got the first round of reviews back from the technical reviewers and there’s a lot of work to be done. So much so that we’ve decided to put the book on hold while we regroup and figure out if/how we are going to address the issues that the reviewers found with the book. In a lot of cases it is going to require a complete rewrite of the example (and hence the chapter itself). Personally I feel the core is there, but maybe the examples weren’t as fleshed out as they could have been.

Now of course I’m a little embarrassed that I announced the book before it was done, but that’s what happens sometimes. Not sure at this point whether or not the book will continue, taking some time to regroup/rethink things and see if it is something that I want to push forward with.

I will keep everyone up to date here on the progress of things once I’ve decided what to do. In the meantime I am going to get back to some of my back-burner projects (a few of which have some DSLs in them ;-).

DSL book in progress

Warner Onstine — Wed, 10 Oct 2007 09:44:00 -0700

Now that things feel a little bit more final to me I thought I would take this opportunity to announce the book that’s been keeping me from blogging. The tentative title is Creating DSLs using Java and Groovy and it will be published by Pragmatic Programmers. I started working on this idea after last years No Fluff Just Stuff where I saw Neal Ford speak on DSLs and I talked to him briefly about a project I was working on. His talk inspired me to push forward with using DSLs for code generation specifically using Groovy.

The book will cover both Java and Groovy techniques for writing DSLs, as well as have some general guidelines for writing DSLs. We also have a chapter on ANTLR and JavaCC for writing external DSLs using Java. I tried to cover as much as I could of Groovy meta-programming and some of the additional capabilities of Groovy that make it a good language to write an internal DSL on top of, but there’s only so much space ;-).

We just sent off the book for peer review, which means its about half-way finished, maybe a little more. We are currently shooting for a release date sometime early next year, which looks very doable at this point, but I expect a ton of edits to come back from this first review.

On a final note I just want to say what a pleasure it has been working with the PragProg crew. I love, love, love their book build system and my editor Susannah rocks. I have had a blast working with them so far on this book and am looking forward to finishing this up and getting it published.

Groovy DSL steps

Warner Onstine — Tue, 05 Jun 2007 13:30:00 -0700

What are some of the basic steps that I’ve followed as I create my DSLs (especially with respect to Groovy)?

What are your spoken and unspoken requirements?
Who is the intended user? (this will help you decide some of the other questions later)
Has this kind of problem been solved before?
How did they solve it? (Language, Toolset, Techniques?)
How closely does that solution match our requirements?
What can we take away from that solution (if it doesn’t match the requirements)?
What kind of DSL do you want overall? (Internal or External)
Prototype, prototype, prototype - even if it is an Internal DSL you will still want to start out with a working prototype that your intended audience can play with and give you feedback on. This should be an iterative process to refine the language.
External DSL - how to implement?

I will cover the first 8 of these, and I will leave the External DSL for a later post (sorry ;-). For this I’ll use the steps that I’m currently going through on my own code to illustrate my own thought process.

Requirements

So, what are my requirements for my main two DSLs - QueryBuilder and ViewBuilder (this doesn’t exist yet, but starting on the planning right now). For QueryBuilder I wanted a clean syntax that makes it easy for developers to create constraints/restrictions on a database query. Ultimately the query could be executed by Hibernate, JPA, iBatis, whatever, so it should be ORM-agnostic. This actually isn’t too hard as they all have this concept of a criteria on an object/table and restrictions for that query. And the same goes for ViewBuilder, I want an HTML-like tree syntax that I can specify components in as well as attributes that will be applied to the view when it is built for me.

User

This one is easy, I am targeting Web application developers, initially I am targeting Tapestry Web application developers, but since the framework will be pluggable (probably around the 2.0 time) you can specify what specific pieces you want to use (Tapestry/JSF + Hibernate/JPA + Spring 1.x/2.x/Guice, etc.).

Has this problem been solved before?

In the case of the QueryBuilder, yes. Grails currently has an implementation called HibernateCriteriaBuilder. This has a nice syntax for specifying criteria and restrictions on that criteria. As for the ViewBuilder I don’t think this has been solved yet, but I will be looking. There is something similar in Groovy’s core for creating arbitrary HTML, which is close but not exactly what I’m looking for as I will need to generate more than just HTML from it.

How did they solve it?

Both of these were solved using Groovy’s BuilderSupport object which allows you to build up arbitrary “nodes” from a collection of method, property and closure calls. Very powerful and a little difficult to wrap your head around the first 10 times you look at it ;-) (alright, maybe it was just me, but that took a bit to fully understand what the hell it was doing).

How closely does that solution match our requirements?

First, for the HibernateCriteriaBuilder it actually will build up a real, live Hibernate criteria and use it, which unfortunately is not what I want. I need to generate the Java code that will come out of it. And the MarkupBuilder gets me part of the way there but not quite what I’m looking for.

What can we take away from that solution?

I decided to take away the syntax from HibernateCriteriaBuilder and looked at how they implemented it to see what I could do with that. And for the ViewBuilder I am going to extend BuilderSupport itself and use that to build up my nodes of whatever objects (so more than just plain HTML text, actually objects that represent things).

What kind of DSL do you want overall?

Definitely shooting for internal I think, but that doesn’t preclude having some kind of external DSL for the class creation stuff (that was a giant pain to do in Groovy/Java - just not all of the information I actually needed). So, having some kind of custom language for defining models could definitely come in handy down the road, but I haven’t decided anything yet. Plus, as I learn more Groovy and get feedback/suggestions I may stumble across something that simplifies my design greatly.

Prototype, prototype, prototype

This, aside from requirements and research is probably the most valuable step. Once you decide what you are going to do, how in the world do you do it? Especially if you’re dealing with Meta-programming (which can be a hard concept to wrap your head around)?

Here is where my experience will hopefully prove valuable to someone out there, here are the steps that I took when developing my QueryBuilder:

Write the Unit tests first, start with the simplest test you can think of for how your language should work
Run it - yeah I know its going to fail, but prove it to yourself ;-)
Make it go green - whether you go ahead and create a simple class that has one method in it that does what you want do that
Write another 2 tests - think about them, how are they going to change that class you just wrote, how much more complex is it going to make the logic you need to write to make sure you do what you expect.
Fix the first test - is it time to extend DelegatingMetaClass yet? Or maybe try something simpler like a Category? Maybe the new ExpandoMetaClass will work. Try a few of them out, but I don’t recommend you dive right into extending BuilderSupport, not yet, not unless you have already used it before and understand fully what it does. Throw in lots of println "calling so and so with these arguments". These help identify that what you are expecting is actually happening while you write your DSL.
Did that work? Good, write it so the next test succeeds.
Continue doing 4, 5, and 6 until you are happy and ready to show it to someone
Let them play with it, get their feedback. Rinse, repeat, refactor. (Here is where I would suggest you revisit the BuilderSupport idea, does it make sense? Remember not everything is an arbitrary nested tree, so not everything will benefit from using this structure.)

Well, I hope that this is helpful to someone ;-). I really have gotten the DSL bug and (mostly) enjoy writing them. If you run into a weird error, throw it out to the Groovy list, someone should be able to help you out.

Addendum

I meant to add this in when I first started thinking about this post. When I started writing my QueryBuilder I didn’t start out with it generating anything but simple select strings. This helped me visualize the proper nesting of things without getting too complex. Once that was working right I changed my unit tests to expect the actual Java code I wanted back and changed the logic of what the DSL output.