Saturday, 14 December 2013

Maven (not the space probe)

Until recently I had always used ant and ivy to manage building my Java projects. It works well except for one really annoying thing. Actually maybe it is several. When you have a team of programmers maintaining ant scripts it is easy for standards to drift, various projects have 'special needs' in their builds and that has to be catered for. Sometimes people just create projects with a different directory structure because they feel like it. I'm all for freedom of choice in these matters, except when it is late at night with a delivery in the morning and I am trying to work out someone's odd build arrangement that isn't working. Then my inner fascist surfaces and I wonder if we couldn't do better.

So I moved all (um, nearly all) my projects to use an included ant file containing all the standard targets we need for a build and the builds just have to conform to those. That worked quite well, then the really annoying thing turned up.

I wanted the builds to be easy to get started, so you can pull the project from source control and just run the build. I'm assuming ant is already installed, but I'm not assuming anything else. What about ivy? Okay, my included file looks for an ivy directory and pulls the ivy jar file if it isn't there. Ivy pulls everything else.

What about the include file? Well that has to be committed into the project. But it is the *same* include file in every project, committing multiple times violates DRY. Yuck. Plus when I change it I have to change it in lots of places, well that's why we say DRY, isn't it?

You've already seen the title so you know where I'm going. I thought I would try maven. Maven uses plugins to do those targets the include file delivers but, because plugins are just jar files they can be managed as dependencies, so maven will pull whatever plugins I specify from the repository. They've pretty much all been written already and I just have to conform to the standards they dictate. That means so does everyone else and we all understand they layout etc of our projects and how they are built. Great. Even better, you just have to have maven installed then any build you pull can be just run by invoking maven. No need to do tricky things with a dependency manager. The entire build+dependencies for the project is defined in a pom.xml file.

How does this work in practice?

Quite well, better than I expected in fact. But there are a some tricky bits that weren't immediately obvious.

First, all my builds use MaduraDocs which was basically an xsd to define the format, a couple of xsl files and a load of ant files to pull it together. Every build uses MaduraDocs. So it was immediately clear this would have to be a maven plugin. That was good. Rewriting the scripts into Java made them simpler and I could make use of some OO techniques to make things more flexible. I ran into one issue there: how does a project build when it depends on a plugin that is itself. MaduraDocs depends on the MaduraDocs plugin. Well, I cheated a little. I invoke the MaduraDocs code from one of the tests and that builds the documentation for MaduraDocs itself, rather than invoking the plugin.

The next challenge was MaduraObjects. MaduraObjects is a JAXB plugin. Not to be confused with a maven plugin. Though, and here's the tricky bit, there is a maven plugin for JAXB called maven-jaxb22-plugin. maven-jaxb22-plugin  accepts JAXB plugins and MaduraObjects is one of those. My initial approach was to simply add maven-jaxb22-plugin to my project and have it run, invoking the MaduraObjects code. Except it doesn't. For some reason I did not figure out it refused to find the MaduraObjects code so I had to try another approach. Again I fell back on tests. I wrote a test that invokes maven-jaxb22-plugin making sure the MaduraObjects code is on the classpath and that works. It results in some generated Java which then needs to be recognised by the phase that compiles the test code. To do that I use the build-helper-maven-plugin which can tell maven to include another directory in with the test code. If you use the build-helper-maven-plugin just right the Eclipse maven plugin uses it to recognise the generated source directory and adds it to the project classpath, that avoids various red Xs in Eclipse.

MaduraBundles is even more fun. The test code requires three small jar files to be built and loaded into a sweep directory. MaduraBundles loads small jar files as sub-applications and the tests have to try that out. Maven is *very* keen on the 'one project one output' approach and this project doesn't actually violate that, it just needs to generate some test jars. This is the most complex and the oddest of my projects. It works and is not so very complicated, but I suspect this is a rare case of extreme complexity.

Finally MaduraRules needed another maven plugin and this one generates source files, so it has the same issue that MaduraDocs has in that it must avoid invoking itself, and it needs to have the generated code added to the test code for compiling. In practice the code that does the work is invoked by tests in the MaduraRules project and the plugin is a simple wrapper to that code placed in a separate project.

These tricky bits took some working out but overall it ends up simpler, I think. One of the things that made me sceptical of maven initially was how I would handle the various configurations I was using in ant+ivy. When using Ivy I normally had configurations for test, compile, package, build and so on. I could define whatever ones I wanted and just use them. I could pull all the dependencies of, say, the build configuration, into a directory and use that as the classpath for the build. Specifically I would use this for invoking JAXB to generate Java from an XSD file or to generate rules from a rules file. I knew that maven had more strictness around this with a specific set of phases that, while it could be extended, was best left alone. How would I manage my builds?

The answer is delightfully simple. Because plugins can have dependencies and those dependencies are managed by maven directly you don't need a build configuration, you just specify the plugin you want. The dependencies of the plugin are distinct from dependencies of the project itself. So I don't need a build configuration anymore, nor do I have to write scripts to pull the build dependencies into a directory. As for compile, test and package there are equivalents in maven. Test dependencies are only used in the test phase, compile dependencies are everything the project needs to compile against and these are always packaged as well. There is sometimes a case where the project needs to compile against something but not package it, in this case you tell maven it is 'provided'.

There are some more things in maven that I haven't had to look at closely. It seems the pom.xml file can specify a parent to inherit, which means if you have a lot of common stuff in your pom files you can consolidate that into one file and inherit it. Superficially this looks like the included ant file I mentioned earlier but there is a key difference. The parent pom file can be kept in the repository and fetched by the maven dependency manager. That means you don't have to violate DRY, and you still don't have to jump through hoops to get the parent file into your project.

So for me the maven builds turn out generally simpler, more consistent and there is no violation of DRY.

Friday, 13 December 2013

Tama Lakes


This is a picture of the Upper Tama Lake near Mt Ruapehu which is in the middle of the North Island of New Zealand.

It is also a picture of my foot, but we'll come back to that. Know for now that the foot is deliberate, not a mistake.

There are two Tama lakes, an upper and a lower and they are both volcanic craters. Pretty much everything in this area is volcanic and they used places like this to stand in for Mordor in the LOTR movies. Ngaruahoe, the mountain they used for Mt Doom is the rising edge on the left of the picture.

To get to the lakes takes about three hours walk from Whakapapa Village, which you can drive to. I've walked this track three times now and the last time I did it the Whakapapa end was a bit dodgy and hard going. I'm pleased to say there has been a lot of good work done there. Much of the track would be okay to walk with a pushchair if you had to. There are some steps here and there but the rest of it is nicely gravelled walkways and quite a lot of it is flat.

However once past the lower lake the going is a bit tougher because the track degenerates to a 'poled route' by which you scramble up a steep slope and along a ridge. It's only about 45 minutes going and not too tough really, and from the ridge top you get this view to the upper lake. Then you turn around and go back.

Now about the foot.

You'll notice I am not wearing boots, I'm wearing sandals. I have boots but I've given up wearing them. The sandals, sometimes padded with socks as shown but usually not, really do the job for me. They don't compress my toes like boots do and even over rough ground they seem to protect my feet just fine. If I have to wade a stream they dry out faster (I take the socks off first). This track does not require stream wading, plenty of streams but there are bridges if the stream is bigger than a jump.

If I were walking through serious snow I'd probably wear boots, though earlier this year I was happily walking through light snow in sandals without socks. These are, of course, heavy duty sandals and they can take a bit of punishment. Mine are Tevas, but Mrs has a different brand so the particular type doesn't matter too much I think.




The socks also keep the sun off my feet which mattered on this occasion because I was silly enough to forget to put sunscreen on the rest of me (cloudy day... not really an excuse) and I paid for it later on my knees, nose and hands. But my feet were fine.

Saturday, 7 December 2013

The Waters of Mars

If you haven't seen it, 'The Waters of Mars' is an episode of Dr Who in which some alarming things are discovered about frozen water beneath the Martian surface, and some other alarming things are discovered about the crew of the first manned Martian base (a British one, of course). It is a good yarn and it gave me the idea for this post.

But this is not about strange things in the Martian water, it is more about the water itself and the fact that is seems to be there at all, or possibly it isn't now, but it definitely was once, which is more or less the point.

When I was a kid I read stories by Heinlein and Blish where people (mostly kids, actually) wandered about the surface of Mars and found it a challenging environment but not much more challenging than, say, Antactica. The air pressure was lower and it was a lot colder, but with some breathing apparatus and suitable clothing, trekking across the Martian wastes was practical enough. And there were signs of life. Lichens mostly but, in Heinlein's stories, more sophisticated plants as well as animal life and even intelligent life. There was a sense that the environment had once been a lot more friendly to our kind of life but it had changed for some reason and this was all that was left. The Dr Who episode followed a similar line.

The recent rover missions to Mars support the idea that Mars once had a decent atmosphere and running water on its surface. The two go together. Under very low pressure, such as that on Mars today, water boils even at low temperatures and doesn't exist long in liquid form. So to support the geological findings of old stream beds etc there needs to have been more atmosphere, and for the water to not be frozen, it needs to have been warmer.

The science fiction I read as a kid was more hopeful than the science at the time. Although a few decades earlier some astronomers had suggested there was a global civilisation on Mars, we all knew that it was too tough an environment to support any such thing. Very little atmosphere, harsh radiation and very cold. If there was any life there it was hard to see how it could have started, let alone survived.

Did you spot the mistake there? Actually there are two of them, but the second one is subtle.

The more obvious mistake is the problem of induction. We make observations and we assume that what we see is how things are, which is okay. Then we project those observations to say something like 'it has always been like this' which is not so okay. We also often project them to say 'it is like this everywhere', in this case everywhere on Mars but we are pretty quick to project our findings to everywhere in the universe. For example the speed of light, the law of conservation of mass and even the notion of cause and effect, we assume to apply everywhere. We do this because we don't have better information. When we get better information (evidence that Mars has had a different environment in the past) we find we were wrong and we have to adjust our thinking. Being willing and able to adjust our thinking is a huge virtue, clinging to outmoded ideas in the face of conflicting evidence is not, though sometimes people try and say it is. But adjusting our thinking doesn't solve the problem of induction. We might be wrong about everything because of this.

The less obvious mistake involves just why we keep on inducing things despite it being obvious that we keep getting it wrong. We insist on keeping to the simplest explanation we can come up with. It is so much simpler to assume that things are consistent across space and time, simpler to assume Mars has always been the same as it is now, so that is what we do. Then we find it is more complicated and have to adjust. The simplest explanation seems to be actually never the right answer, though it is a long tradition in science and any other form of knowledge that keeping things as simple as possible is a good thing. But we need to be clear eyed about this. We pick simple explanations because we like them, not because they are in any way better or more true. We simply choose to believe, for the moment, until we are proven wrong anyway, that the simpler explanation is correct.

We think we know how the Martian water got there, something like icy meteorites crashing into the planet, possibly even bringing the strange life forms in the Dr Who episode. But it might have been something completely different that is much more complicated. In fact, it probably was.