A Groovy Decision

Note: This post was actually written at the beginning of the year, just never quite completed & posted until now.

When I started at the company we had 2 big process that were running in cobol ( well pretty much any way, the team had started rewritting large portions of the first project in Java before I joined ). Over the 3 years that I have been here we have completely removed the cobol from the first process. Now I am working on removing the cobol from the second process. The first thing that we relalized is that the heart of the 2 applications is exactly the same, of course it was litterally 2 copies of the same files with different names in the cobol code. So we want to reuse the heart of Project A that has already been rewritten, I'm not about to go copying and pasting the files like the cobol was. So we decided that we would create Project C which is nothing more than refactoring out the chunk of A that will be reused by B.

Project A was written when Java was first starting to be used here at the company and used a homegrown framework for Database connections, logging, and most importantly our Domain/Entity objects. This framework is a very slow memory hog that is difficult to work with from a coding perspecitive. All of the new code we currently write is using EJB 3 ( Hibernate for batch proccesses like these ) along with a little bit of our own framework that is more just a wrapper for the actual frameworks we use instead of all of our own crap code.

So that left us a few options:

1. Write Project B using the old framework ( framework is so out-dated that this was quickly ruled out.
2. Rewrite Project A using our new framework ( this is an eventual goal, but this would take at least 3-4 months with no customer benifit, so this was also quickly ruled out )
3. Write Project C as a webservice running somewhere that could be called by both A & B ( These proccess take roughly 1 hour to run today, and would make roughly 60-70 million times, sounds like a preformance problem to me )
4. Copy & Paste all the "reusable" code into project B and modify it to use our new framework. ( do I even need to say why we ruled this out )
   
5. Use a dynamically typed Language ( groovy was my preference going in ).

I got together with my team leader and his manager, to discuss our options. At first there was some resistance to something as new ( as in new to the company ) as groovy. Over the course of our discussion we seriously weighed all the costs and benefits of each option. I think at some point in our talks we had thought that each one of these was the best solution. However in the end the biggest reasons against a dynamic language were performance and training. Performance seams to be constantly debated and we eventually agreed that without writing the code and seeing what happens there is no way we would ever be able to trust any benchmarks. And as for training, well at some point every team's technologies need to progress otherwise our team would still be writing in COBOL.

So Groovy it is. Everything I read talks about Groovy being completely compatible with Java, so we'll start with renaming all of the files from .java to .groovy and go from there. I anticipate that this will not be as smooth as I dream, but hopefully not too bad. Stay tuned.

Are we ready to go Groovy?

So in my last blog post, Contemplating Learning, I discussed what would cause you to start looking for new technologies.

Well let me lay out a scenario for you of where I'm at right now in this decision process. We have Project A that uses an old home grown persistence framework. And Project B that uses our new framework that is JPA based. Now the problem is that our old and new frameworks are highly incompatible, to the point that we do not want them both on the classpath at the same time. Now the problem is that Project A & B need to share a very large peice of pretty complicated code. The inputs to the shared code our large domain objects. And the output is many changed values on those input domain objects. The plan is that Project C will now house the shared code.

Project A's domain objects are classes that extend a goofy base class that does the dao work. Project B's domain objects are POJOs with the addition of JPA annotations. Since we are purely a Java shop Project C now needs to house a thrid domain object that is really a DTO. All 3 of these domain objects will always need to stay in sync. And the real bummer will be the transition code, Project A/B has a domain object then converts it to a DTO for Project C, then after C returns it the DTO needs to be converted back into Project A/B's domain object. Which leads to 4 more places that we need to maintain all of the getters and setters for each of these domain objects.

Enter Duck Typing. For those unfamiliar with duck typing the idea is: if walks like a duck and quacks like a duck then it is a duck. So in the case of our domain objects as long as Project A's & Project B's domain objects have the same set of getters and setters ( which is actually enforced because of the fact that they need to map the same table just in two different frameworks ) then we can just call all the getters and not worry about what kind of framework we are using, or for that matter what type of class we have. This will allow us the flexibility to move to some other future framework as well, and as long as we still have getters & setters the code will work just fine.

So Groovy solves the problem. Are there other ways, probably. In fact Ruby, Python, Perl, etc. would all solve the problem is a similar way. So why groovy? The biggest reason is that this isn't a decision about me using a new language, but rather the team using a new language. The big advantage of Groovy is that it is sooo close to Java that the other developers on our team, who pretty much know Java and that's it, can hopefully pick this up pretty well.

Now I know I could use a whole bunch more features of Groovy to make Project C even better, but if there is one thing that I've learned from the great minds that I pick everyday at work its baby steps. So by writing the Groovy to be as close to Java as possible we can take that first little baby step towards freedom from our insistence that Java is the only language that code can ever be written in. Although I do see many places in our code base that Java makes the most sense, this isn't one of them.

Time to pitch my idea to the powers that be and see what we come up with.

Contemplating Learning

Over time we must continue to add new tools to our tool box. This is one of the most important things that every developer and every development team must do. If we, as developers, continue to do nothing to improve our knowledge then we will quickly become obsolete and be doing nothing but maintenance work on legacy code in languages like COBOL. If we, as development teams, don't continue to progress and attempt to use new technologies, tools, and/or languages our code will quickly become out dated. There is a reason that teams do not ( or least not normal teams ) choose COBOL for green field development. That said the flip side is true as well. We need to be cautious to not practice RDD ( Resume Driven Design ). Choosing to write some code in Groovy just because Groovy happens to be your new flavor of the day language doesn't make for a good business model, just like forcing all code to be written in Java just because that's what we've always done, doesn't make a good business model.

So the first question to answer is how do you know when its time to look at a new tool/technology/language. I think the starting point is pretty obvious...does your current set up prevent you from accomplishing something. Now it doesn't have to be a total block, maybe it just makes something really really hard. Or maybe it forces you to write some horrible hackish code that you just know is going to be next to impossible to maintain.

The tough part is the next decision... Should I/we use said new technology. If it is an "I" question then the answer is probably yes. If this is for some home project or something along those lines then there is much much less convincing to do. As the pragmatic programmers have taught us, you should be teaching yourself a new language every year. Same thing if it is an "I" for a new tool that you are personally going to use that could make you more productive without impacting anybody else on your team. Well then go for it. What's the worst that could happen, it doesn't work out and you have to go back to doing things the way you used to.

If it is a "we" question well then it gets a whole lot tougher. You have to consider things like: what benefit does this bring beyond just my one task at hand? How difficult is it going to be to maintain this code for a developer that has never used this before? What's wrong with finding a way to do it the way we always have? What risk is involved? Is there a step learning curve? If the project turns into an epic fail can we recover by switching back to some other solution? If we can then why are we not using that other solution in the first place?

I think one of the biggest problems we get ourselves into is exactly first question...maybe the technology doesn't provide any benefit beyond the current task at hand, but if it solves that problem shouldn't we say its good enough for that task?

Over the next couple of blog post's I'll use Groovy as an example of going through this process of choosing whether or not its time to introduce a new language.

NFSJ(TDD, DDT, and other practices of Agile Folk)

I went to a couple of talks by John Carnell on Leadership and the Lean process. The lean practices sounded kind of interesting but I never really figured out exactly what was going on through out the talks. The one part of the lean process that I picked up was making Value Stream Maps to help figure out where your process was wasteful. I don't know that I would ever do that because I think everybody here already recognized where we are wasteful, now the trick is figuring out how to make parts of the process better where they are wasteful. The part that is really good, and I think I will start using is his estimation stuff. He showed a spreadsheet where tasks were estimated as ether 20,40, or 80 hour tasks, and then there was some percentages for prototyping, design, testing, and some different risk factors that would help to generate a low..assume all risk factors were 0% and a high which was calculated given the risk factor percentages that you entered. He offered the spreadsheet to anyone that wanted a copy so I emailed him to get a copy that I can start playing around with. If I come up with something different after I start playing around with it I will post it here.

The last day I spent all day at Jared Richardson's talks on Agile Process type topics. All the talks and topics kinda blended together. One of the big things he kept going over and over was that nothing works if you don't have CI ( continuous integration ) and test automation. That was one thing that really gave me a good feeling about where we are at today in the company. We use Cruise Control and have all of our projects building there and unit tests running when they exist, so at least we have the foundation built. Tracer Bullets was a really neat idea too. Basically set up an API at each layer ( DB, DAO, computation, display, etc. ) that returns hard coded data. Then you can verify before any implementation happens that your protocols work and the communication is there. As far as myself I took a lot of Integration level testing ideas from these talks. Basically I neglect integration, and if there is one thing I took from all of his talks is that integration tests are just as important as unit tests. I liked his idea of writing a very high level integration test first, then implementing the lower level code, writing unit tests as you write code, with the goal of getting the integration tests to pass by completing the units. In the past when I have written integration tests its always been the other way around...write all the units, then as I build up to the integration level I write the integration tests. Another cool testing idea was DDT, defect driven testing. Maybe this can get people on board with Automated Testing in that if something is broken you write a unit test for it. Then as everybody gets better at it, start moving towards TDD. I will continue to say that we should enforce at least unit tests be written on all code checked in. Otherwise we are just going to continue to generate more and more code that is not covered and keep building up the Technical Debt. I have already, unintentionally, done a Blitzkrieg of sorts. As we wanted developers to start writing more tests ( we encourage it, but rarely does it get done ) we went through most of our projects and added a few unit tests to each so that there was examples and the framework set in most every project. Peer Code Reviews are another thing that I think we should start doing. Today I think that my manager is reviewing my code, but generally speaking this is done when we do a QA release, and at the point it is hard to go back and change things. I liked the idea of forcing a code review before checking anything in, and enforcing that you cannot have the same person do 2 code reviews for you. We are a pretty divided department, one team works on one set of code and the other works on another set of code, we are slowly breaking down the barriers between code and everybody is working more on everything. But I liked this idea that somebody on the other team could review your code too, thereby helping them understand what's going on with the other team in the process.

A topic that came up in a lot of talks, not just Jared's, was time boxing iterations. I really liked this idea of breaking down tasks smaller building lists of what you want to get done. I think a lot of us were already close to doing this using SharePoint and/or Xplanner, but listening to all the different ideas I think I feel better about knowing how to do it. On of the keys that I never really thought about is that every task should take between ½ day and 3 days to complete. Along those lines the idea of a daily meeting that is really a stand up couple of minutes to answer 3 questions: What did you do yesterday? What are you going to do today? What problems do you have? Then if you have a visible list everybody knows whats going on all the time. All of these talks and information about lists, daily meetings, etc. lead me to think that we might be better off working in a more team orientated environment. We all work together very well, but maybe instead of each developer getting assigned to a project we assign a project to a group of 2-4 developers. Everybody can work together to come up with a good design and everybody can grab random tasks for code, which would lead to a bit more diversity. I am terrible when it comes to writing a design for everything that needs to get done, and often I think it would be useful to have a second person working on the design with me to bounce ideas off of.

NFJS(Mocks, CI, and other tools of Agile Folks)

The first of several Agile talks I went to was Tools to Facilitate Agile Development. The first set of tools was all based around Unit Testing. Most of this was about Mock objects and Junit in general. We already use EasyMock and try to do some testing ( we are getting better all the time ). One Unit testing tool I had never heard of, and plan to explore more is JUnitPerf. One of our big problems at work is that our Jboss server PermGens A LOT. We have never been able to reproduce this in tests. Maybe using the load testing stuff here I will be able to reproduce it, then we can finally start debugging it. There was some good information on Measuring Design Quality and trying to get you're code closer to the Main Sequence...a good balance between abstractness and concrete implementation that actually does something. I like this, but as we try to introduce more tools and techniques I think this one is going to end up down on the list. We have been looking at Code Coverage tools on our own already and I think this is one of the things that we will try to implement pretty soon. Maybe not on everything, but at least on some stuff. I had never heard of, or much less thought about, using a tool like Jester that attempts to figure out of if you are writing "good" tests by modifying your code at run time to verify that if it changes the code your tests will start to fail. I like this, at least for now, as something that we should encourage people to run on their own while writing tests. One of the things Venkat mentioned about Jester is its ability to help you learn how to write good tests. But it also throws a lot of false positives, so I'm not sure I'm ready to throw that on Cruise Control just yet. I have run PMD many times on my own code, and I like it, but it also throws a lot of false positives, so until we get better Junit coverage, and eliminate the errors generated by FindBugs then I'm not sure I'm ready to jump on PMD or its CPD tool. As it is today we do not run FindBugs on our projects, except for the couple that I am working on. Mostly the reason so far has been that there are too many bugs reported and "why should we fix those before we fix the ones that the customer is pointing out to us" I'll continue to fight back on that one as I think we should get Find Bugs running on everything ASAP. Venkat also touched on CI, and over the last year we have gotten Cruise Control up and running, and within the last 3-4 months we now have all of our projects in Cruise Control and running the unit tests when they exist.

I also went to Venkat's talk Mocking to Facilitate Unit Testing. This was another talk where I have already been doing this for a while, and have introduced it to the rest of the team. I really went to this talk because I had introduced myself to EasyMock and taught myself how to use the framework, so I wanted to know if we were doing things right. As it turns out we are on the right track. The one thing that I took from this class was...”don't use a Mock Framework just for the sake of using a Mock Framework”. I'm not going to go back to my old code and change the tests, but I know from know on I'll really consider heavily whether or not I really need a mock object or if I can create a better one on my own. And as I look back on it, I really think that I will tend to lean on EasyMock more often than not because I like what it offers. The other thing that came from this talk ( at least I think it was this one ) was the idea of introducing Groovy to the company by writing unit tests in Groovy. This would offer a lot of the mock stuff right inside of it.

Jeff Brown gave a Test Driven Development with Groovy and Grails talk. There was actually not a lot here that I had not seen before, which kinda surprised me a bit. There were really two things that I took aways from this. First that I like TDD with a dynamic language because you can write a set of tests for stuff that doesn't exist and still run them. For Example...a class is going to have 3 methods that interact with each other. I can write up the unit tests and run them before the class is implemented. In Java you could write up the tests, but you definitely can't run them because the code won't compile. This is a pretty minor thing, but I really like that idea. The other was the coolness of the Expando class, because of Duck Typing you can build and Expando and use it as a mock for anything...really cool. Interestingly not a groovy specific thing, but he talked a bit about Canoo Webtest for testing grails applications, and I think we could really utilize that to test our Web stuff since today we do NO tests on our front ends. I think I will give that a try the next time I edit the front end stuff, which hopefully will be a long time from now.

NFJS ( Groovy )

In a word "awesome". I think this really comes down more to dynamic languages in General. I went to several of Jeff Browns talks on Groovy, and along the way I heard mention several times that this stuff can be done in most, if not all, dynamic languages.

One of the things I like about groovy is the fact that it interacts so well with Java. There are a couple of reasons that I really like this. 1. I'm coming from nothing but Java ( and a little Cobol ) over the past few years so it is really easy for me to transition to the new syntax. 2. We built a base class "application" that we utilize to drive all of our batch processing stuff, since Groovy can work with Java so well I can write an "application" class that extends this base application and still take advantage of all the functionality that we have already built.

I went the Thorough Introduction to Groovy talk even though I had written some Groovy before, it was all self taught so I saw this as a good opportunity to find out if I was doing it "right". I was happy to find out that I am definitely on the right track. I never knew about the Builders before, but I sure wish that I had because I wrote a whole lot of Java code to mimic that kind of simplicity for a project last year. Of course its not at all the same, because Groovy uses some meta programming to capture method calls to methods that don't exist to get the Job done, but I think my solution makes writing XML just as easy as the Builder does. Of course the big drawback is that I spent a few months writing the code and I can almost guarantee there are at least a few bugs in it because we just made it work for exactly our use case. The other thing in the intro that I wish I would have known about earlier is the Groovy Console. That is really handy for playing around with the syntax to figure out how to do things. I had previously kinda knew what Groovy Properties were, but since I never used them before I did not realize how powerful they could be in respect to less coding, and more importantly the ability to call constructors with a map of what properties you want set. I can't count the number of times I wish a particular domain object had a constructor with a given set of parameters when I was building my unit tests. One thing that I really wish more "intro" class/talks/websites would do different, and this one fell into the same trap but with a warning up front, is that closers are too often associated with loops. Jeff did go through a few other examples like the number.times method that accepts, but that can be done in Java with a for loop. I really like closures in that you can write more concise/cleaner code with them, but I'm still searching for a good example of something that would be just outright painful, or better yet impossible, to do in Java that a closure can do easily. Thinking about them it just feels like that example must be out there. There was just enough of a hint at Meta Programming here to get me to come back to the Meta Programming class later on, so I'll talk more about Meta programming with that class.

Meta Programming is Awesome. Jeff Brown gave a pretty cool talk on this stuff. This is one of those topics that I really never heard of and was totally blown away by how useful it is. It allows you to intercept any method call that you want to ( like AOP ) and do stuff. The big thing that standard AOP doesn't let you do is you can call methods that doen't exist and use Meta Programming to do something useful. The best example Jeff gave was the XML MarkupBuilder. Also using Closures to to add functionality to classes, as well as the more traditional MetaClass way of adding functionality to classes.

No Fluff Just Suff

I spent the weekend along with a bunch of guys from our team at the NFJS show in Green Bay: www.nofluffjuststuff.com. Overall I think the conference was very good. One of the big things that I took from the Conference is the idea of having a retrospective after you finish an iteration. One way to view this weekend was as an iteration of learning, so I'm gonna have my own little retrospective here.

Ok this got really long as I was writting it so I have broken it into several smaller enteries. And just a quick summary here.

10 Best things from the weekend:

1. Integration Tests are equally as important as unit tests
   
2. Peer Code review should be done before EVERY Checkin
3. Peer code review does NOT have to be done with somebody that is an expert in the code you are working on
4. DDT - Defect Driven Testing ( if there is a bug write a test before fixing the bug )
5. Groovy is well groovy
6. Meta Programming is Extreamly Powerful
7. You can manipulate Java objects at runtime from groovy using Meta Programming
8. Code Coverage should include all Unit and Integration/Functional Tests and System testing
9. Work in smaller iterations ( typically 1-4 weeks )
10. Within iterations break tasks down into things that can be completed in 0.5-3 days

The biggest thing that could impove me as a developer is...
Instead of simply writting unit tests I need to take a bigger picutre and make sure I write good integration tests as well.

The biggest thing that could impove the team..
Doing peer code reviews before every check in would help both the reviewer and the reviewe.

Update:
Here are links for the 3 detail entries...
NFJS ( Groovy )
NFJS(Mocks, CI, and other tools of Agile Folks)
NFSJ(TDD, DDT, and other practices of Agile Folk)