Gradle and Java Dependency Library Management

Recently I learned that I couldn’t easily merge the encryption library I settled on into my main one without pain. As this is a sporadically visited hobby, I don’t have the free time to get to the bottom of it. As we’re dealing with a signed jar around security, it’s possible I’d be pushing s**t uphill anyway.

You might be aware by now that I like to solve the trusting of 3rd-party library problem by simply glueing those libraries into the final executable. If I can’t have that ideal, I want a model that’s as close as I can get to it.  I decided I’d merge those dependency jars that are easy to merge, and ship the ones with issues as external dependencies.

I’m now as close to my ideal as I think I can push it, so below is some gory detail for fellow gradle fans who are especially interested in building Java applications. Gradle 1.11 was used for the scripting below.

First, I introduce some of my own properties into the project:


project.version = "0.2"

project.ext {
  title = "PersonalFinancier"
  libsDirectory = "libs"
  distributionDirectory = "dist"
}

Then I separate my dependencies into two lists. One list has the libraries I want merged. The other, the libraries I’ll ship separately.


List libsToMerge = [ 
  "com.jtattoo:JTattoo:1.6.10", 
  "com.cedarsoftware:json-io:2.5.2", 
  "de.erichseifert.gral:gral-core:0.9" 
]

List libsToShip = [ 
  "org.bouncycastle:bcprov-jdk15on:1.50" 
]

Then I introduce two new configurations, one for the merging of libraries, the other for shipping:


configurations {
  mergeLibs
  shipLibs
}

I then modify the dependencies to a) tie the relevant lists the their matching configurations, and b) ensure that the pre-supplied compile dependencies continue to rely on both sets of libraries:


dependencies {
  mergeLibs libsToMerge
  shipLibs libsToShip
  compile libsToMerge, libsToShip
}

I also modify the jar task. I first tell it to take the collection of merge libraries and zip them into the jar being produced. Then, as I build the manifest, I ensure that each of the libraries I need to ship are listed on the Class-Path of the manifest file. Finally, because I am developing in Linux, and would rather the build script do it, I’m waiting until the jar is cooked, then flagging the jar as executable.


jar {

  from {
    configurations.mergeLibs.collect { zipTree(it) }
  }

  manifest {
   attributes (
      'Implementation-Title': project.title, 
      'Implementation-Version': project.version,
      'Contact':  project.contact,
      'Built-With': "gradle-${project.getGradle().getGradleVersion()}, groovy-${GroovySystem.getVersion()}",
      'Created-By': System.getProperty('java.version') + ' (' + System.getProperty('java.vendor') + ')',
      'Main-Class' : mainClassName,
      'Class-Path' : configurations.shipLibs.collect{ it.getName() }.join(' '),
    )
  }
  
  doLast {
    file(jar.archivePath).setExecutable(true)
    println "$jar.archiveName created and is executable."
  }
}

Because I’m developing under Eclipse, I have a separate ‘libs’ directory for the Eclipse project to look in for dependencies. I make things easy for my Eclipse experience by introducing a task that takes all the project dependency libraries and syncs them to the directory that Eclipse knows to look in:


task syncLibs(type: Sync) {
  from configurations.compile
  into project.libsDirectory
  doLast {
    println "Synchronised library dependencies into directory ($project.libsDirectory)."
  }
}

Finally, I introduce a new task to distribute both the external libraries and the executable jar itself, all to to the distribution directory I specified as a project property:


task distribute(type: Copy) {
  dependsOn jar, syncLibs
  from jar 
  from configurations.shipLibs
  into project.distributionDirectory
  doLast {
    println "Distributed $project.title and support libraries to directory ($project.distributionDirectory)."
  }
}

It took me a while to figure out, but shipping an executable jar with a minimum of support libraries is something gradle handles gracefully.

Who I am and why I’m here (with Pocket Ninjas)!

Existential, no? Don’t worry. I’m participating in a blogging challenge to see what new things I can learn about this hobby. Regular subscribers be warned, this probably means I’m going to be more post-happy-chatty over the period of the challenge.

Today’s challenge encourages me to (re-)introduce myself and give you a feel for why this blog exists.

First, a brief intro around who I am.  My name is Lindsay. I’m a married father of two young children living in Australia, and have a 20+year career in software development. I have a PhD, but mostly forget to call myself ‘Doctor’.

Why am I here, posting to this blog? There are a bunch of suggested questions to the challenge for today, so let’s just run through them.

Why am I blogging publicly, rather than keeping a personal journal?

Would it surprise you that I cover a spectrum?  I have pen and paper and a physically co-located support network for the heavy personal mojo. I then have a number of work-related/hobby WordPress journals that I use as a running commentary on where I’m at with a particular project. Some of those private posts are coalesced once I’ve broken the back of an issue, and find myself ready to share what I’ve learned with the world here.

That word ‘help’ is key to what I find blog-worthy, as a decision to blog hinges mostly around whether it might help others.   I enjoy knowing that sharing insights into the challenges I’ve faced off against are useful to a wider community.

Wanna help a pocket ninja? Of course you do!

Wanna help a pocket ninja? Of course you do!

There’s a delightful tradition in software development of people blogging around the software problems they’ve had.  I’ve saved myself an immense amount of time over the years finding blog and forum posts that nail the technical challenge I’m wrestling with.  I like the blogs more, because there’s often interesting context around the issue that can be worth knowing about.

This blog started out as notes to myself on the snarlier problems I’d been caught up in that I thought others might also face, very much in the spirit of this tradition. I was surprised to find that I enjoyed the experience of knowing I’d helped when I’d see esoteric web search queries so tight I could tell someome else just arrived at a spot I knew well. They’d follow links withing seconds to minutes  that I sometimes spent hours trawling for, paring the signal away from the noise.

I’ve had over 27,000 views since I decided to host with WordPress in September of 2010.  A goodly subset of that number represents who I’ve somehow helped.  I guess you’d say that by now, I’m hooked on helping.

What topics do you think you’ll write about?

If history is any predictor, I’ll mostly continue writing about challenges overcome in hobbies or work. This mostly involves software development.  Strangely, my most popular post to date is the editing of EPS files with Inkscape. Who’d a thunk it?

Pocket Ninja Man! Doing the things a Pocket Ninja Man Can!

Pocket Ninja Man! Doing the things a Pocket Ninja Man Can!

Reflecting on it, I’m not here in the hopes of having a huge following. Though there are about 20 people now following the blog via Wordpress, I admit to wondering what these and other followers see in my eclectic bouncing from topic to topic.  I can only guess that you guys get enough signal:noise ratio that you don’t mind the uninteresting posts.

Who would you love to connect with via your blog?

Now.. that’s a question I’ve not considered until now.  First challenge delivers! I’d love to connect with anyone and everyone who shares similar interests. You’d probably be a software developer, vector graphics enthusiast, or gamer who especially enjoys Minecraft or Torchlight 2.

What's the collective noun for pocket ninja? A lint of of pocket ninja!

What’s the collective noun for pocket ninja? A lint of of pocket ninja!

That’s part of what I think is missing for me to date though. The general subconscious discomfort I’ve been feeling when I consider the trajectory the blog is on.  I’m currently wondering how to find or form a community of like-minded bloggers who don’t exactly have a “theme” but nevertheless have overlapping common-ground in the interests we share.

If you blog successfully throughout the next year, what would you hope to have accomplished?

Hmm… a goal that’s not measurable is just a wish, so after some careful consultation with my pocket-ninjas, factoring in their obsession with Belgian chocolate:

1) I’m averaging about 660 views per month at the moment.  Let’s aim for double that over the coming year, putting me at about 15,900 new views between now and then.  I’d like to have stats telling me I’d received 43,700 views by the end of April, 2015.

2) Double the number of subscribers at year’s end, putting me at 40 (ooh… uncomfortable, given the eclectic nature of the blog).

3) At least five other bloggers who I’ve been regularly chatting with in a kind of ‘like-minded blog-ring’. More discomfort right there.

Poor clipart graph never knew what hit it.

Poor clipart graph never knew what hit it.

There you go, a discussion around why I blog, with the timely support of pocket ninjas. The care and maintenance of pocket ninjas is something I take very seriously. Pocket ninjas flourish with heavy web traffic.  Don’t make me starve the pocket ninjas.  You wouldn’t like the pocket ninjas when they’re hangry.

5 things to know about AES Encryption in Java

Recent revelations around security on the Web have left me shaken, not stirred. If I can’t trust people in positions of power to respect the concept of ‘innocent until proven  guilty’, it strikes me that withdrawal of my trust would be a rather rational response.

To that end, I’ve been consumed this past week with figuring out how to password encrypt certain files I’m generating that I’d rather people spend effort on if they stumble across them.  I also freely admit that my inner Loki is mighty pleased by picturing the delightful dilemma others may face in trying to square off my ‘secrets most mundane’ against the inevitable sunken cost fallacies they’ll need to concoct to justify the discovery time.

I’ve settled on symmetric key encryption via Advanced Encryption Standard (AES) because a) I’m a cheap-arse, b) I’m too lazy to go the extra effort of grokking asymmetric key encryption, and c) AES has a good rep for the less challenging arena of symmetric key encryption.

Now that I have a working implementation, I thought I’d write something that I wish I had whilst I was getting up to speed. I’ve discovered 5 things that you should know if you’d like to implement AES in Java.

1. the Java SDK limits you to AES-128 by default.

Java’s SDK has been capable of offering ‘strong’ encryption by merging its Java Cryptography Extension (JCE) into the base library ever since Java 1.4.2.  Now, ‘strong’ encryption in Java is limited to the AES-128 variant . Java make the distinction between ‘strong’ and ‘unlimited’, as certain countries impose import-control restrictions, limiting the amount of cryptography they’ll allow.

If you want to build applications that allow ‘unlimited’ encryption from the Java SDK, you’ll need to download the Unlimited Strength Jurisdiction Policy Files. And yes, you’ll need to ship those unlimited policy jars as part of your final deployment.

Now, before you get your panties in a twist, even Bruce Schneier considers AES-128 adequate enough.  Yes, AES-256 is a bit stronger, but 128 will ‘do’ for a while yet despite what’s been thrown at it.

But… but… as a developer, I’m (now) totally aware that the difference between AES-128 and AES-256 is a single constant I use to tell the byte-array that holds my key how long to be.  It’e exactly the same effort to code, so why wouldn’t I go with stronger?  Ah the stories one may weave!

2. alternatives EXIST to the Java SDK for CRYPTO.

My favourite is the Legion of the Bouncy Castle (what an absolutely fantastic name for a cryptography library). And, because we’re not (yet) pandering to US cryptography law here in Australia, there’s no distinction drawn between ‘strong’ and ‘unlimited’.  Crippled Crypto cannot bunk in a Bouncy Castle!

I'm Serious!

I’m Serious!

Also, there’s the entire implementation to Java’s blessed JCE interface, giving it an air of ‘appropriate fit’ with respect to how Java prefers its crypto rolled.

Finally, because it’s been registered as an Australian charity, I get tax-deductions for my donations! Oh God! This just keeps getting better and better!

There are others (like Jasypt), but, well.. I came for the crypto, but stayed for the LOLs.

Now.. we’re ARE talking crypto. It’s dangerous to go alone, even if the platform is gravity defying. Take this, and this!

3. As a Black Box, AES is remarkably easy to use.

AES isn’t a trivial algorithm if you want to grok how it does what it does.  However, it’s dead easy to understand from the outside.  Here’s what I wish someone had told me before I started trawling web sites on AES trying to grok how to make use of it:

  • It needs just three pieces of information:
    •  A key. The length of bits in the key designates its strength.  AES-128 literally means AES with a 128 bit key. AES-256 means a key with 256 bits.
    • An Initialisation Vector (IV), helping to randomise the encryption as it starts.
    • The content you want encrypted.
  • So long as you use exactly the same key/IV pair, you’ll be able to decrypt whatever you’ve encrypted.
  • The IV must be the same size as the size of a block of data in AES.  This is ALWAYS 128 bits.  JCE ciphers rely on byte-arrays, and because a Java byte is the same size everywhere, the IV will always be 16 bytes long (128 bits / 8 bits per byte).
  •  As per the above, your AES key needs to be a 32 element byte array if you want AES-256.  Dial it back to 16 bytes for AES-128. Dead-easy.
Well... that was easier than expected!

Well… that was easier than expected!

4. Take Great Care in Consuming Web Examples.

So, I’m a novice when it comes to security with the software I write.  It turns out there are a lot of ways to do cryptogrphy code wrong, and the  popular examples out there are living proof of what ‘wrong’ looks like.

I’m not claiming that what I’ve built is tight. Only that it’s got less issues than the example code I started from. Also, please don’t take my supplying of this link as criticism.  I found worse in my hunt, and this one allowed me a great bootstrap. It focuses just on the AES algorithm, which is good for seeing the particular AES tree in the forest of crypto code that you need to ultimately have written. Would I deploy with it? No.  Am I grateful it showed only what was needed as I was de-nubefying? Yes.

However, there are things I now know that I feel you should be aware of before embarking on your own virgin AES journey:

  • Use SecureRandom whenever you need make things random. The basic Random class falls short. Specifically, password salt and IV should both be ‘very random’.
  •  Don’t reuse randomised IV, salts, etc across multiple encryptions. If you’re encrypting, you’re also re-randomising those things that should be random.
  • If you’re encrypting via a user-supplied password, do not just re-use the password as the AES key.  Salt and hash the password into the key so dictionary, brute-force and rainbow attacks become far less likely to succeed.
  • Consider salt size.  You want it big enough that as random numbers go, it stands a good chance of being truly unique. I’ve seen between 16 and 32 bytes as a rule of thumb here.
Example crypto on the web is butthurt for the brain!

Example crypto on the web is butthurt for the brain!

5. Decryption Context Matters. Think about it!

Ultimately, to decrypt something you’ve encrypted via AES, you are going to need the same key/IV pair that you used initially. Consider what you are going to do with that ‘context’ over the elapsed time between your encryption and decryption events.

The answer you come up with depends entirely on what you are trying to do with encryption in the first place.  I can’t answer for you what you should do here. However, I can briefly describe my own situation and reasoning for the final solution.

I am aiming for the password-encryption of a save-file, created via a project that I’ve released open-source. I have no interest in the code also managing the storage of the password anywhere. The user (me) must consider password storage external to the code. I also can’t rely on keeping the exact storage and retrieval tricks secret by locking the source-code up somewhere.

When it comes time to decrypt, by applying the same salt and hashing algorithm to the original password, the code is capable of deriving the original key.  The password salt will need storage over time, most especially because the advice is that it be random per encryption event.  The Initialisation Vector will also need storage over time for exactly the same reason.

So… the password is the missing external piece of the puzzle required to derive a valid decryption.  Also, I don’t want to be reliant on anything other than the encrypted file and the code for decryption. Note that I’m actively choosing a trade-off between convenience and security here.  Models involving storage of the salt and Initialisation Vector away from the encrypted content might be more secure, so I must be happy that the convenience of local storage is worth less security.

It just so happens that I’m (mostly) fine with this tradeoff.  I’m only “mostly” fine here because by storing the salt with the file, a rainbow table based on the salt can be derived once an attacker knows the file format.  It’s been mentioned that hiding the salt and IV don’t add a great deal of security anyway, so I’m all flippy-floppy about this particular decision.

Anyway, convenience wins and I store the password salt and Initialization Vector within the file along with the encrypted content.  I store all three data-types the same way, so casual inspection won’t drop visible clues to where one starts and another stops.

In storing context, you might be tempted to do funky stuff like, oh, interleaving the salt and IV data, etc.  Ultimately, whatever you do though is obfuscation that MUST be un-obfuscated for a successful decryption (skilled crackers know that you must unobfuscate again, and it helps to know ahead of time that the game is winnable). A common theme I’ve seen in my pouring over software and security is ‘Don’t think you can out-obfuscate a determined unobfuscator’.

Because the code is open-source, it won’t make a lick of difference because the obfuscation code would be available, so simple is best given the lack of extra benefit. I just glue the salt and IV to the encrypted content and save it all to file.

The weak-point that matters is the password itself. By choosing not to deal with its storage within the code, and using a salted hash to derive a key, I’m happy that a strong, memorised password will lead to a good enough encryption experience for my password-locked encrypted file.

Now, if you’re looking to do something different, I doubt my reasoning here would still fully apply.  At this stage, I suggest you pour over content from OWASP and Information Security Stack Exchange that seems relevant to the kind of application you’re considering building.

http://i2.kym-cdn.com/photos/images/original/000/001/946/1240596772330.jpg

Non-choice is not the choice you think it isn’t.

And there we have it.  All the nuanced AES learning I picked up whilst getting a working implementation going.  If you’re about to embark on your own AES journey, I hope this grants you fairer sailing.

Finally, if you’re interested in the end-result, here’s my contribution of an example implementing AES in Java with the BouncyCastle library.

Origami… Clipart… Bliss…

Origami!

So calming. I’ve missed you.

It’s been too many years. Well, until the weekend just past, that is. I’ve rediscovered you, and mashed you up with clipart, which is another hobby I’ve been tooling with in your absence, and I am well pleased with your love-child!

Come then gentle reader, on a retrograde temporal shift with me, back to Friday, 31st January, 2014.

Picture… darkness, for that’s what’s happening.  My eyes are lightly closed; and I’m on the very verge of sleep. Delightfully, a quasi-dream starts. Paper in our shared memory hands, folding, folding, and folding again. The clean, delightful head-space of knowing the fold so well that I’m coasting in ‘The Zone’, and deeply content with the movement, sans the story-teller who is now half-jokingly claiming ownership of something it knows doesn’t belong to it.

However, what is a story without a little drama, reader?

Enter beloved daughter, and zen-master, barging her way into the bedroom and allowing a warm orange light and household noises to shatter my dream-vision.  She demands that we share a story, and as the ghost-origami evaporates from our phantom fingers. Let’s accept the transition, and put our story-teller to good use.

But… that’s not the reason I re-discovered origami.

Allow us to now skip through the remainder of the night, through to the next morning. Open our shared memory-eyes with me reader, and ease into our Saturday morning with a quick scan of some social media with the phone that now also serves as my alarm clock, sleep and blood pressure monitor.

Experience my surprise of finding a new follower on Google+, and the realisation that her public posts are exclusively origami. Note my confusion. Yes, I list Origami as a hobby, but those trawling my digital footprint wouldn’t know it for the ‘proof’ I’ve left. And yet, there we are.  A delightful entry into Saturday, gazing at picture after picture of the folds she’s shared with the world.

But… that’s not yet the reason I re-discovered origami.

Come now with me to the kitchen. It’s early, and nobody else is likely to stir for at least another hour or two.  We eat a breakfast, you and I, and ponder what to do with the silence.  Origami… why not?  It’s been years.

However, what is a story without a little drama, reader?

So, let’s now search the house over, room after room, wondering where the hell the origami library and stacks of coloured, square paper have gotten to. It’s been years of inattention after all, and with a move of house, that stash could be anywhere.

Sneak quietly through the house with me reader, because I am certain that two bouncy munchkins will amp up the handicap on this rusty skill just a little too much for my liking.  Feel again with me the frustration of over-full cupboards, stuffed with interests that parenthood has had no time to revisit. Finally, sing internally with me, as I find the complete stash in a forgotten nook, and successfully extract it without waking my littlest female zen-master.

But… that’s still not the reason I re-discovered origami.

Sit with me now, and watch me burn through a number of squares, re-doing those beginner cranes until I no longer need the book as a crutch. See the peace doves that follow next, for now I’m moving into favoured folds. Finally, watch me fold the talking frog I hold a special love for.

Beloved Origami Frog

Beloved Origami Frog

And know gentle reader, that THIS is the reason I re-discovered origami.

For half-way through the folding of the frog, the chatter stopped in surprise, as the fingers took off and finished the frog without needing to check with the book. Experience again with me that it wasn’t just the fingers, but the faint mind’s eye view into the idealised construct. Three-dimensional, and calling like a siren to align  the physical paper in my hand match the phantom frog in my head.

Now, let us sample highlights from the day, for they serve to deliver more rewards than just that moment of wonder at witnessing my long-dormant skills come out of hibernation.

The children awake and are gifted with paper animals.

Games spontaneously form around the folds over the day.  Frogs and doves sharing meals of invisible worms and flies. Flapping, zooming, pecking paper. A fascinating circular game of “I completely wrecked it THIS time Dad! Bet you can’t bring the bird back again!”.

Eventually the scamp bested me, though I like to think I gave him pause to wonder how I managed to reclaim the bird from that scrunched ball so many times.

However, what is a story without a little drama, reader?

So… Let’s you and I fast-forward through to Monday night. Hold the memory of a stressful day wondering which event will land first.  The contract extension or the contract completion? There’s a month left of money in the ‘blessed’ till. Share if you will, my  discomfort reader. Know that I’ll likely need to call in the heavy artillery to give the HR team the incentive needed to make the magic of contract continuance manifest.

Note how I’m craving time to myself to unwind.  None of my usual tricks will work tonight, because I’m wound up tight.  I need to taste a little flow again, because there was damn-near none of it at work.

I haven’t engaged in a serious Inkscape project for a while, so I decide on the spot that tonight will be me drawing something.

What to draw?

Hmm… here’s my frog, still mostly in one piece.

Very well. I have no idea how hard it will be to reproduce the frog as vector-art, which is perfect. A stretch goal, but not so huge I don’t dismiss it as too big a leap.

Let us fast-forward through the drawing now. And why not?  Flow was held, and time stood still for me until I was too tired to draw any more. The vectorised frog is in a good place. There are spots that need touching up, but I won’t push it now that it’s suddenly gone from just after dinner to way-too-late for a school night in the blink of an eye.

Programming you see, has taught me the folly of having the story-teller try to fill the gap once the flow departs.

Moments of the story-teller surface through the drawing and quietly murmur things like “Ooh.. clever! I like that!”, and “Alright… I can see why that works now”. I note again, that as the story-teller, I’m just not clever enough to come up with some of these things myself.  The deeper moving currents own the flow, and don’t care a jot that ‘Linds’ now claims what can only be forged in that river of bliss. A place that Linds can see from a distance, but can’t enter without it suddenly all evaporating.

And now, we’re back in tonight.  Small touch-ups are made, and an experiment with a tool called SVGCleaner shrinks the finalised image to a third of its size with no visible degradation of quality.

I am well pleased with the end result.

Vectorised Origami Frog

Vectorised Origami Frog

Now gentle reader.  Now… we are done.

Origami… Clipart… Bliss…

MWCIT Source Code Released Under BSD-3 License

It’s been a ‘swimming through molasses’ project all things considered, but today marks a milestone where I can finally drop down a gear. After clearing things with my employer, I’ve just placed the source-code to Release 1.0 of the ‘Murrumbidgee Wetlands Condition Indicator Tool’ under a BSD-3 license, and hosted it with GitHub.

The BSD-3 license was suggested by the University when I was sniffing around for options on how the project client could retain access to the source if they need it once my gig here is up. Turns out there’s excellent reasons for BSD-3 (or more accurately, NOT a Creative Commons license which was my first choice), so I was more than happy to settle on the suggestion.

Now, before you get all excited, Release 1.0 of the MWCIT isn’t much more than a button launcher driven by config-files:

Relase 1.0 of the MWCIT,  doing what it mostly does.

Release 1.0 of the MWCIT, doing what it mostly does.

Still, if you’re a developer with a passing interest in a simple (but not trivial) example of a home-grown Model-View-Presenter (MVP) implementation, or you’re interested in how to coax NSubstitute into firing events out of a mock object, there might be something in it for you.

System Domain Model - MWCIT 1.0

System Domain Model – MWCIT 1.0

Now, button launchers can be made with far less effort, I know.

In my defense, the contract was looking for a Web deployment, but the things handed to me to integrate absolutely scream “Over my dead body!”.  I’ve not completely given up on a Web deployment, but right now, it looks like total re-write to web-enable some of these tools.

I settled on MVP so I could easily tear off the WinForms facing, and drop in a web interface instead (probably ASP) .  The plan was hatched in a moment of desperation, and I nick-named it “The Clown with A Tear-Away Face”.

Accurate call in hindsight, except perhaps for that bit about being gone without a trace.

A delightful side-effect is that object are all chatting to each other via interfaces thanks to the MVP reliance on Inversion of Control.  The suite of unit-tests ended up with excellent, and easy to achieve code coverage.

Now I think about it, I’m somewhat interested in seeing if the design document gets any link-hits, because there are so few examples of open-source software with designs that I’m aware of.

And finally, I was told in Mid-September (after losing 3 months to lawyers debating IP issues that weren’t issues, and other fun stuff) that the contract-variation had an unmovable milestone delivery of “something to show” by mid-December.  I didn’t get approval for my “poor-man’s integration” plan until mid-October.

The code above didn’t take too long to whip up (here in a flash).  The time was mostly spent with very poor-quality data-sets, out-of-date API calls in 3rd-party tools, and extremely large files that broke installer tech like a ‘monster with a taste for the sweet nectar of spinal column juice’ . Oh, and those last two data-sets that landed a week before my oft-repeated release-freeze date.

In hind-sight, a decision to go very light on the ‘software build’ aspect of things was an excellent call. I also learnt a bucket-load more about IP issues around software.  Here’s hoping that the next six months of this project allows some serious integration.

Example Box-Whisker Plot in Excel

All sorts of loose-ends need tidying up now that I’m past a software delivery crunch. Here’s one.

I was asked recently to do a box-whisker plot of some raw data out of this species optimisation framework I’ve been working on.The boss mentioned Excel in passing, so I got fixated on trying to do it with Excel, instead of the more sane approach of using a tool better suited to the task.

Turns out it’s bloody hard to do in Excel (peeps, consider R for out-of-the-can box-whisker plots). You need to go through the rigmarole of using a stacked bar graph to plot deltas for the Minimum, Maximum, 1st, 2nd and 3rd quartiles, all of which you’ll need to derive yourself. Here’s some detailed instructions that didn’t suck/weren’t wrong.

Excel Box-Whisker Plot

Excel Box-Whisker Plot

Finally, because I never want to waste that much time again getting Excel to do box-whisker plots, here’s an example Excel file that does one for you.

Configurable Data Directories via Inno Setup

Recently, I’ve been relying on Inno Setup installation scripts to build setup executables for my Windows installs. I’ve given up on .NET deployment projects, and even InstallShield Express (why does my brain insist on reading ‘Express’ as ‘CrippleWare‘?) given their limitations.

The two big wins with Inno Setup for me are 1) it’s freeware, and 2) if I need to do something funky, I can roll my own behaviour into the script with its Pascal scripting option.

Today’s post is how I wrote an installer wizard that prompts the user for a “Data Directory” that receives data that I don’t want sitting in the “Program Files” area of Windows, and a configuration file so I can re-install to a different data-directory and allow the program toggle between them via config file modifications.

I do this because a very wise man once pointed out to me that:

Data that evolves at different rates should be designed to evolve separately.

Programming tends to evolve more slowly than input data, which can typically evolve more slowly than output data. This approach is a tip of the hat to his wisdom. Also, as I’ve learned several times:

The programs can come and go, but if you let it, the data will outlive them.

So, back to Inno Script and getting it to generate a config-file at install time that points at the Data-directory we will store our data to. I’ve settled on INI files, given their ease of understanding for my target users, and Inno Setup’s favouring of them.

First, I define a directory under the [Dirs] section. The name of the directory is delegated to being resolved via a call to some code I define later called GetDataDir(). I check that the directory doesn’t exist. I ask that it not be uninstalled with the software, and I grant the user permission to modify the directory contents:

[Dirs]
Name: {code:GetDataDir}; Check: not DataDirExists; Flags: uninsneveruninstall; Permissions: users-modify

Then in the [Files] section, I identify the files I want (here an entire directory I’ve pre-defined with the constant {#DataDir}. I ask that it confirms any over-write that it might have to do of pre-existing data, to recurse through any sub-directory and again, that the contents are to be left intact on an uninstall of the software:

[Files]
Source: "{#DataDir}\*"; DestDir: "{code:GetDataDir}";  Flags: confirmoverwrite recursesubdirs uninsneveruninstall

Next, I write some Pascal code to prompt the user for a data directory. If an INI file doesn’t exist, it will initially supply text based on the default location for the user’s application data. If the file does exist, the wizard will use the value it finds in the file instead.

A warning I should point out here is that turned out to be a bit tricky to get working as intended.

Because I want the wizard to prompt the user for a data directory directly after they’ve been prompted for the program directory, I’ve learned that program directory constant {app} isn’t available yet when this code runs. I need to reach under the hood, and pluck the value out of the running Delphi API code via a call to WizardDirDrive(), which returns whatever the user accepted as the program directory in the install wizard’s previous step.

[Code]
// global vars
var
  DataDirPage: TInputDirWizardPage;
  SampleDataPage: TInputOptionWizardPage;
  DataDirVal: String;

function GetDataDir(Param: String): String;
begin
  { Return the selected DataDir }
  Result := DataDirPage.Values[0];
end;

function GetDefaultDataDirectory() : String;
begin
  Result := ExpandConstant('{localappdata}\{#MyShortAppName}');
end;

function GetIniFilename() : String;
begin
    Result :=  WizardDirValue() + '\{#MyShortAppName}.ini';
end;

  // custom wizard page setup, for data dir.
procedure InitializeWizard;
var
  myLocalAppData: String;
begin
  DataDirPage := CreateInputDirPage(
    wpSelectDir,
    '{#MyLongAppName} Data Directory',
    '',
    'Please select a directory to install {#MyShortAppName} data to.',
    False,
    '{#MyShortAppName}'
  );
  DataDirPage.Add('');

  DataDirPage.Values[0] := GetIniString('{#MyShortAppName}', '{#INI_DataDirKey}', GetDefaultDataDirectory(), GetIniFilename());
end;

function DataDirExists(): Boolean;
begin
  { Find out if data dir already exists }
  Result := DirExists(GetDataDir(''));
end;

Finally, whatever the user chose via the scripted code will ned to be stored in this INI file. I add an entry to the [INI] section telling the script to write the DataDirectory value to the INI file:

[INI]
Filename: "{app}\{#MyShortAppName}.ini"; Section: "{#MyShortAppName}"; Key: "DataDirectory"; String: "{code:GetDataDir}"; Flags: createkeyifdoesntexist

And there we have it. Inno Setup. It’s powerful, it’s free, and it allows me to tack hard against default behaviour if I need something non-standard. If you’re programming for Windows, and need a a top-notch installer that’s free, take it for a spin.