Thursday, 29 November 2012

Directed Questioning



This announces the latest version of Madura Rules, part of my Madura toolkit.

Madura Rules is a rules engine that plays nicely with Java objects, the rules refer to fields in the objects and enforce validation and derive new data. This all happens transparently, and with almost no api calls. You just write your code to talk to your objects and the rules just happen.

But this was missing something I used to have in an earlier engine I built about 20 years ago: Directed Questioning.

Let's say you have a really complicated data input scenario, something like an insurance form or a tax form. These things often have a lot of questions on them that users don't have to answer because the questions don't apply. There are a lot of statements like 'go to step 17 if you answered NO' and so on. If this is paper based (and it usually is) then the amount of paper is somewhat intimidating, even when the user might only have to answer 5 questions or so. And finding which five questions out of, say, 200 is awkward.

You could build a software system that codes this kind of form, of course, and people have. But they are hard to maintain. Adding a new field or two means you have to review what circumstances a user has to supply the information and then adjust your navigation and flows to ensure the right users answer those questions.

Directed Questioning bypasses that by using the fact that the rules engine knows all the relationships between these questions and can dynamically customise the UI as the user enters the data. Here is a simple example that calculates your BMI.
formula: Customer "BMI"
{
    bmi = weight / (height * height);
}
formula: Customer "Height Metric"
{
    height = heightMetric;
}
formula: Customer "Height Imperial"
{
    height = (heightFeet * 0.3048D) + (heightInches * 0.0254D);
}
formula: Customer "Weight metric"
{
    weight = weightKilos;
}
formula: Customer "Weight pounds"
{
    weight = weightPounds * 0.453D;
}
These are the rules the engine uses. They happen to be all formula rules. We can use if/then rules as well as constraint rules if we need to, but I am keeping this example simple. The first rule is the standard BMI formula and the rest of them allow the user to enter in metric or imperial units, depending on what they prefer.

All we are really interested in is the value of bmi. We can tell the rules engine to
go figure it with a simple Java call. What actually happens then is the engine looks at the rules and sees that it needs a value for height and weight. To get weight it needs to fire either the third or fourth rule, it doesn't matter which at this point.

So it tries the third rule, and that means it needs a value for weightKilos. There aren't any rules that deliver weightKilos, if there were the engine would go look at them, so it asks for weightKilos. That's the first question the user sees.

This user is happy to answer that and inputs a value, so the engine derives a value for weight. It does not have to ask for weightPounds and it knows that. Now it is looking for height.

To get height it tries the second rule and asks for heightMetric. This user is happy with metric weights but not length and he doesn't know his height in metres. We can tell the engine we 'don't know' that value and it will look for another way to get the height.

There is the third rule which needs heightFeet and heightInches so it asks for those, figures the height and then it can work out the BMI.
So, of the 5 possible fields to enter this user was only asked for 4. If he had known his height in metres it would have been only 2.

The questions are asked one at a time, because each answer might eliminate the need to ask the next one. The API is designed to be architecture friendly, it actually queries the engine for the next field to ask for, and how it is asked (ie what technology is used to ask it) is up to the application. My current preference for these things is to use Vaadin and I have a working example on rhcloud you can look at from your browser. Log in as admin/admin and click the BMI button on the main screen.

Tuesday, 27 November 2012

John Chilembwe

John Chilembwe is regarded as something of a martyr in Malawi today. I found a reference to him in a book about Malawi history I've been reading.

My grandfather was a missionary in Malawi 1913-1919. He wasn't a priest or a preacher, he mostly built houses, schools and churches for the mission effort. He wrote some notes of his time there and I've put them here. So I have been reading a few books about the early missions in Malawi, wondering if any of them refer to my grandfather (only one so far, he was a minor player in the scheme of things). But I did find references to John Chilembwe and my grandfather's comments on him were interesting to compare.

First, we have to know about Joseph Booth, an Englishman who arrived in Malawi in 1892 with his nine year old daughter. He writes of his arrival:
"...I reached this spot, weary, sickly and weak. My little daughter was with me... A drenching rain was falling. Several Angoni men were sheltering under a grass roof they had constructed on the boughs of that tree. They had made a fire there and were crouching round to keep warm... Yet these men out of their good nature got up instantly and insisted that myself and little child should sit around the fire while they stood close to the trunk of the tree in the rain to get what shelter they could."
Booth, unlike some of the English he met in Malawi, seems to have understood that the Africans were just as human as he was. As such he was regarded as a radical by some members of the English administration. His catch cry was "Africa for the Africans" which did not go down too well with the English settlers. I'm not sure just what the other missionaries made of him, but they were likely enough more sympathetic. A driving concern for the English missionaries was the slave trade. Raiders did good business in Malawi raiding villages and selling their captives on the coast. The missions' general approach was to generate enough local industry to make the slave trade comparatively unattractive. The Administration's approach leaned more towards using force to stamp out the trade.

Life for the missionaries was pretty tough. They seemed to die like flies over there, subject to tropical diseases unheard of back home. My grandfather came home alive, but with illnesses that blighted the rest of his life.

Back to Chilembwe, though. Booth hired Chilembwe as a 'house boy', basically a domestic servant, and proceeded to educate him along with the others in the mission he set up. He seems to have been a star pupil and part of the family. Booth's daughter, Emily, wrote of him years later with sisterly affection. Booth and he travelled to the USA in 1897.

By 1915 Chilembwe was running his own mission, helping with the rather too many disputes the local Africans had with the English settlers, and campaigning against the idea that Africans should be helping fight the English war (WWI). And that is when the real trouble started. The actual incident seems to have been quite minor and, though it was represented as an African uprising, it resulted in about three English deaths, several injuries and Chilembwe and his friends were pursued by police and shot. There was also an unsuccessful and possibly unconnected attempt to raid the munitions depot in Blantyre, one of the key towns in Malawi. The whole affair seems to have been over in a week apart from a Commission of Inquiry launched by the Administration which concluded Chilembwe had been treated unfairly and the issues he had raised were valid.

But where communication is tricky rumours can spread. My grandfather was quite some distance from the centre of it but he heard of the Chilembwe 'movement'. Here is what he says:
It was while I was building the church at Kayoyo that the "Chilembwe" movement started probably influenced by some enemy in Britain. Anyhow it never got beyond Blantyre where it started. The order was sent out for every boy to kill his master and thus free the country of whites so that Chilembwe would take charge. It was some while before I heard of this through the mail and I asked Stephen about it, had he heard about Chilembwe?
"Yes".
"Why didn't you tell me, were there not enough of us here to protect you?"
"Why should I cause you worry?"
Can we trust a native? And this was the man who was allotted the job of cutting my throat according to Chilembwe.
Of course it is more likely that Chilembwe issued no such orders, but it also looks like if he had the missionaries at least would have been safe enough. The Africans seemed to have a pretty clear idea of the missionaries' motives, understanding that these were good people doing their best, and they responded accordingly.

Ref: Malawi: History of the Nation, Bidglai Pachai

Friday, 23 November 2012

A Dangerous Idea

At the age of about 12 I was exposed to a dangerous idea. It would be a better story if this went all dark and blighted my life from then on but it didn't and I turned out fairly normal. But the idea still lurks and surfaces fairly often. Since it didn't blight my life I guess it won't blight yours either so I will share it.

There is a series of books by James Blish called 'Cities in Flight' about whole cities converted to space ships and lifted off the earth along with their industrial factories etc. They move from planet to planet selling their services to the inhabitants, all originally from Earth as well. Blish took this story into the far future. At times it is a metaphor for the Okie phenomenon in the 1930s and the cities are the itinerant tramps hawking their labour for meagre wages. They even call themselves Okies sometimes. In the final book he makes several sly references to Gulliver's Travels and around then the dangerous idea emerges not as a key part of the story, just as an aside. The image shown here refers to the second book in the series.

I should add that these books contain fascinating ideas, especially for something written in the 1960s, but the characters are mostly cardboard things with no depth, so not great literature.

In the last book one of these characters expresses a belief in something called 'stochasticism' which is a term that crops up in the real world in discussions about quantum effects and the unpredictability of various other things. Blish's character explained that his stochasticism view held that everything is random and the laws of the universe are an illusion. In a big enough universe that is truly random chances are there will be pockets of something that looks like order. We just happen to live in one of those pockets.

For now.

The orderliness of our particular part of the universe extends to as far as we can see, which is a long way but the universe is really, really big so that doesn't help. It is ordered at this point in time, for this few million or billion years or so, and that might change at any moment and everything we know would fall apart.

Is this just nonsense? Don't we know better?

Well, as far as I can tell, we don't. Science has done very well for us and scientific method is probably the most useful thing we have invented, but even without being very scientific we make assumptions about the consistency of the universe all the time. People who thought the earth was flat still assumed the sun would rise in the morning. Consistency, see? And we observe this consistency over and over, so it must be right, mustn't it?

Well, only if we can justify something called induction, which is what we are doing when we assume the observations we made in this lab on this date will apply in a different lab on another day. David Hume seems to have been the first to point out that we have no logical basis for that assumption. No matter how many times we observe something, as Karl Popper pointed out, there is always the possibility that the next observation will break our theory. And if the universe is really random we have even less basis for it regardless how often we have observed it before. It is an awful thought and if you tried to base your life on it you probably wouldn't bother getting out of bed in the morning. This is the dangerous idea.

To make things a little more awkward the rules of science hold that theories must be kept as simple as possible. This is Ockham's Razor. Which is simpler: the completely random universe which has natural pockets of seeming order, or the one with universal laws? It seems to me that the former has less to explain itself.

Of course no one does actually believe this. We continue making our observations, noting patterns in the way the universe seems to work and telling ourselves these are universal laws that apply across all time and space. Naturally some places have more gravity or more radiation etc than others so we're not saying all time and space is exactly like Earth is today, but we are saying that we can figure this out because there is something to figure out.

No, don't tell me there must be universal laws because they've been observed empirically. If you still think that you haven't been paying attention. Go back and read what I said three paragraphs ago. 


If I have explained this well enough you should now be convinced that, while the dangerous idea is a valid enough and, by the rules of science and philosophy, can be argued as preferable, there is no way you are going along with it. You are damn well going to believe in those universal laws because the alternative is simply awful.

Fine. I completely agree. I'm not going to endorse it either. I'm going to live in a universe where there are universal laws even if I can't prove there really are universal laws. Which reminds me of something:

I’m on Aslan’s side even if there isn’t any Aslan to lead it. I’m going to live as like a Narnian as I can even if there isn’t any Narnia.” - C.S. Lewis, The Silver Chair”

Writing Portable Mobile Apps

Since the rise of smart phones writing phone apps is all the rage, to the extent that it almost seems we have forgotten what we used to do before there were smart phones. There are tons of desktops and laptops out there armed with browsers, but my local bus company wants me to download an app to my phone so that I can basically look at their web page. So do my local weather service. Even my phone company wants to do this.

All of the above present the same information on a web page which I can access just fine using the browser on my smart phone. So why do they go to the trouble of developing mobile applications to do the same thing? Notice I said 'applications', plural. They typically develop for iOS and again for Android. The two use different APIs and different development tools so this does mean double the effort. But it doesn't stop there.  There are Blackberry, webOS, Symbian and Windows phones out there too and, assuming you think they are all worth supporting, you'll need to rewrite your application for those as well.

And all to present the information on your web page. Really?

The reason they want to create an app is probably one or more of the following:
  • The newly hired batch of geeks are getting bored
  • We need to present a cool image to our customers
  • We can monetize the app (ie sell it) through the various app stores.
Different app writers will have different reasons. The weather service I mentioned above wants me to pay for their app, so they're clearly into the 3rd reason. The phone company and the bus company give their apps away so I'm thinking one of the first two reasons applies to them. Or maybe once I use their apps I will find add-on pay for features, or maybe they have advertising.

None of these reasons are bad reasons. But they do seem a little pointless when what you end up with is just the same as the web site.

And there is a better way anyway.

In the early days of desktop computing the entire program, including its UI, always ran on the desktop machine in much the same way smart phone apps do today. Many desktop applications still work like this (MS Office, for example).

But these always need you to install something locally and as the Web took off applications started to be entirely web based. This was always difficult to manage well partly because the browsers were fairly dumb and partly because there were so many different technologies to integrate (html, javascript, JSP or ASP, backend languages like Java or C, databases etc). Then you had security and failover to manage.

But we moved on to smarter techniques using AJAX that could build really slick applications that were web based but look like desktop applications. Google's GMail and most of their other services are examples of this approach. They also published GWT, the framework they use to build their applications. There are others in this area too such as Dojo and Echo, so Google doesn't own this space.

But the interesting thing about GWT is it means you can forget about those other technologies and just write Java. The Javascript is generated transparently (including the code to handle cross browser inconsistencies). But it doesn't stop there.

Vaadin developed their product on top of GWT and Vaadin's approach simplifies the GWT stuff, provides a ton of out-of-the-box widgets (which you can add to) as well as a rich framework for building heavy duty Java applications at the server side. Using Vaadin you don't even have to worry about GWT, though you can tweak CSS stuff if you want (and you probably do want to) to add a richer look and feel. But CSS is just look and feel, not function. The CSS team can go work on their own while the application programmers work on the real stuff (oops, I just insulted all the CSS experts out there, sorry).  This separation is important. It ensures the two groups do not trip over each other.

Vaadin applications are designed for desktop browsers. How does this fit into your mobile ambitions?

The answer is Touchkit, a Vaadin add on which has widgets that look and behave like smart phone controls. The default style is iOS but you can tweak that for other environments, and you're still using the same application.

What about monetization? Oh, yes, you still want to see your app in the app store and charge for it. Okay, you can do that by surrounding it with a webview, but it is still the same application and it is still running in a browser. There is a small gotcha there in that Apple are a bit wary of webview applications, although they are allowing some. Also I'm not sure about serving up advertising in an application in a web view, probably fine, but it might need some work to get it going.

That gets you a browser based app that is installed like a native mobile app, but you only have one copy to develop and maintain. What are the downsides?

There are a couple. First, you do have to have a server running the application somewhere. So if you were developing a stand alone Tetris game this solution is not for you. Having said that Vaadin's Touchkit supports going offline (actually I think it is riding on GWT's support for this) and synching subsequent changes afterwards, so you're not quite tied to the internet. But you do need a server.

The second one is that you are still running in a browser. What the browser cannot do your app cannot do. So, location services are okay because HTML5 browsers support that. Camera access not so much. There is a work being done in this area with a product called PhoneGap which seems to understand how to drive all the mobile platforms I mentioned early on in this post. Actually they list 7. It takes HTML, Javascript and CSS and turns them into an app for the target platforms, including accessing the accellerometer, compass, camera etc. They do appear to have had some issues with Apple about whether their apps can be in the app store. But these seem to be resolved.

Vaadin applications, as far as the client is concerned, are precisely that (HTML, Javascript and CSS and the client doesn't know about the Java back end code running on the server). More about it working with Vaadin here. There is ongoing work in this area, but we should be able to write totally portable mobile web based applications.

Thursday, 8 November 2012

Courier Blues

I need to rant about couriers. Does anyone else get the same run around I do?
Okay, maybe my situation is a little odd.
We have a house in the country and an apartment in town. We only want to get our mail in one place so while all the neighbours in our country road have RD boxes (Rural Delivery, like a letter box, the postal service delivers most days) we get our mail in town.
Most of our neighbours in town get their mail in the boxes in the communal area and the postman puts it there most days. Well, when we first moved there we had mail stolen from those so we got a PO Box. This means there's a kind of mini locker at the nearest post office and the letters are placed in that. It has a lock, so it is secure.
Long ago I worked for the postal service here in NZ and I know a bit about the process. The PO Boxes are open at the back to allow sorting. But the public aren't allowed into that area (there are locks on the main doors etc) making it quite secure.
So, our usual way of getting mail is from a PO Box and this works perfectly every time, never a problem.
But often when I order something online they won't take a PO Box number because they want to use a courier. That's when the problem starts.
I have used the apartment address but they inevitably try and deliver while I'm out (and I am out a lot). It is pretty much impossible to arrange a time like 'anytime Monday morning', they'll come on Friday, leave a card and come back on Monday afternoon.
I've had cases where I stayed in all day and called the courier when no one arrived. They told me he had arrived but no one was there. Did he go to the wrong door and I didn't notice? It's a tiny apartment, that's not actually possible. They didn't say but I gathered the driver had gone to the wrong suburb that time.
After a few of those I took to using the work address. There is always someone on the desk in office hours so should be fine right?
Most of the time yes. Then a birthday present I ordered for Mrs failed to turn up. I still don't know what happened but I called the vendor and they promptly sent a replacement which (not their fault) arrived too late.
I tried putting a 'hold for pickup' instruction on it. That worked fairly well, but I did have to drive out to the airport for it and they had a bit of trouble finding it. I gather no one ever does this and they didn't really know how to handle it. After much humming and hawing a nerdy guy came out from the bowels of the place and said 'are you Roger Parkinson?' yes 'okay, I know where your stuff is'. And he did.
Okay, so this time when confronted with 'no PO Box' I decided to try the country address. Let's see what happens.
I can track FedEx on the web so I know it arrived in Auckland within two days of them sending it (good). Then it stayed in Auckland 'in transit' for a week. I phoned their Auckland office to find out what they were doing.
Well, they'd passed it to a local courier who were 'waiting for instructions'. It seems they decided the address I had given did not exist. The lady from FedEx added '...but I can see it on Google... okay, I will call them now.' I should make clear I think FedEx did a good job here, their only mistake was passing it to the local courier.
The local courier phoned me and I gave them directions to my house and I told them not to deliver on Thursday morning because I'd be out. He wrote this down (or I think he did, he asked me to repeat several points slowly etc).
Next morning (Thursday) I got a call from the local Rural Delivery lady asking where my house is. I could tell her, but I also pointed out that I was out. And I had told the courier company this only yesterday. They had passed the box to the RD people with none of my instructions. A quick conversation with the RD lady established where I live (she knows the road well, she drives it most days) and that I would be home the next day and, sure enough, she brought it this morning as promised.
The weak link in this chain, obviously, is that local courier. It seems that every interaction I ever have with any local courier (there are lots of them) is a problem. FedEx were fine and the RD people were fine.
I have removed several vendors from my list of people to buy stuff from because they won't ship to my PO Box, and this doesn't happen too often now. So
  • if you are a vendor think hard about who you ship through. Your stuff might be great, but if your customer gets this run around he will buy elsewhere, even if it costs more.
  • if you are an international courier (looking at you FedEx, but there are others) think hard about who you pass your stuff to. The locals can let you down and make you look bad even when it isn't your direct fault.
/rant