Matt Newman

Travel posts

2018-06-16T00:00:00+00:00

Sometime in the next couple of days I’m going to publish a bunch of back dated posts about travel I’ve done recently (or in some cases not so recently). A lot of these posts are taken straight from emails I sent to family and friends at the time I did the actual travel, I decided to post them here as email is where keystrokes go to die.

This is a bit of a departure from what has been the norm on this blog so far, and for those following via RSS I’ve introduced categories and separate feeds for the Software/Tech category and the Travel category in addition to the existing feed containing all posts. If you’d rather not hear about travel posts, please update your subscription :)

Technically, the only interesting part of this is handling the images I wanted to include in the posts. I’ve included a few images in the past, but never in a large quantity.

I use Google Photos/Drive for all my photos, and originally tried hotlinking the photos from there. If this was supported it would have been the best solution, as I wouldn’t have to duplicate the photos and it would be easy to include them (though some of them are quite large). Unfortunately (though unsurprisingly) Google Photos/Drive doesn’t support hotlinking, and any direct links to images expire after a while.

The next option was using the Google Drive ‘embed’ functionality, which involves inserting some HTML into the post which will load an IFrame. This worked, but:

it looked a little clunky on desktop and didn’t scale well on mobile
a Google IFrame means Google tracking cookies
performance was poor - loading all the images on one of the travel posts using this method resulted in a whole load of extra requests:

I ended up committing them into the Git repository that backs this static site, but that does mean the repo size is going to grow a lot more than it should. As it’s just me working on this repo though, I can always use git filter-branch as a last resort if I do want to move the images elsewhere. To avoid bloating the repo too much and to keep loading times low I did fairly aggressively reduce the file size with ImageMagick:

mogrify -sampling-factor 4:2:0 \
  -strip \
  -quality 85 \
  -interlace JPEG \
  -colorspace sRGB \
  -resize 50% \
  -auto-orient
  *

The final change was adding lazy loading using Lozad.js. Thankfully I include almost all the images on my site using a custom Jekyll include so I just had to include the library and change one file to get lazy loading across the board.

Vetted - tweaking importer performance

2018-03-03T00:00:00+00:00

Since the last post, I’ve started working on an importer to load data from the existing Access database. Work to date is on GitHub.

In the current domain model, there is a single aggregate root, the Client. The importer is written as a command line application which interacts directly with the domain, assuming an empty database (I might get to incremental imports in the future). At a high level, the importer currently:

Creates the clients
Adds any existing ‘notes’ about the client
- Notes are freeform text about a client, unrelated to any particular patient or transaction
- The existing application is a little limited in what can be entered into the main form, so notes have been used to make up the slack (e.g. in the existing data, there are numerous clients which have an email address or fax number in the notes field, as there is no first class input for these values)
Adds home and mobile phone numbers
Adds the ‘most common travel distance’ as a note

These steps are visible in the implementation of the importer:

override fun run(vararg args: String?) {
    val rows = accessDb.clientTableRows

    val newClientIds: Map<String, UUID> = createClients(
        rows,
        accessDb::postCodeFor,
        accessDb::stateFor,
        commandGateway
    )

    allOf(
        addClientNotes(rows, newClientIds, commandGateway),
        addPhoneNumbers(rows, newClientIds, commandGateway),
        addMostCommonDistance(rows, newClientIds, commandGateway)
    ).get()
}

First, clients are created, producing a Map of the old client ID to the new client ID. Once all clients have been created, all the other updates are applied (potentially concurrently).

Without giving away too much information about the existing data, the order of magnitude of the existing number of clients is 3, and the total number of events generated with the current importer implementation is at most 5x the number of clients (one ClientMigratedEvent, up to two ClientNoteAddedEvents and up to two ClientPhoneNumberAddedEvents).

My first pass at the importer was taking around 80 seconds to import everything into a PostgreSQL database. I know that premature optimization is the root of all evil, and that I don’t have anything resembling a working product at the moment, but this seemed far too high. Also, it was impacting my ability to iterate quickly with ‘production’ data, which is enough of a reason to look for improvements.

After looking at the generated schema and doing some sampling with VisualVM, I decided there were three options to investigate:

Asynchronous processing of commands
Serialisation format changes
Generated schema changes

In order to compare a full run of the importer pre and post optimisations, I want to be able to toggle the optimisations on/off from the command line. The following script has the toggle properties in place, and in the sections below I will use Spring config management to read these properties.

PASSWORD=$(uuidgen)

docker stop vetted-postgres ; docker rm vetted-postgres

docker run \
    --publish 5432:5432/tcp \
    --name vetted-postgres \
    --env POSTGRES_PASSWORD=$PASSWORD \
    --detach \
    postgres

./gradlew build

java \
    -jar importer/build/libs/vetted-importer-0.0.1-SNAPSHOT.jar \
    --axon.use-async-command-bus=false \
    --axon.use-cbor-serializer=false \
    --spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL95Dialect \
    --spring.datasource.password=$PASSWORD

The script above will allow me to evaluate the impact of any changes I make in a repeatable fashion.

Option 1 - Asynchronous processing of commands

I’m using the Axon framework, which handles a lot of the plumbing of building an application based on DDD & CQRS principles. By default when using the Spring auto-configuration, a SimpleCommandBus is used which processes commands on the calling thread.

I added some configuration to use a AsynchronousCommandBus with a configurable number of threads:

@Bean
@ConditionalOnProperty(
    value = ["axon.use-async-command-bus"],
    matchIfMissing = true
)
fun bus(
    transactionManager: TransactionManager,
    @Value("\${axon.command-bus.executor.pool-size}") poolSize: Int
): CommandBus {
    val bus = AsynchronousCommandBus(
        Executors.newFixedThreadPool(poolSize)
    )

    val tmi = TransactionManagingInterceptor(transactionManager)
    bus.registerHandlerInterceptor(tmi)

    return bus
}

I initially tried this configuration out with a pool size of 10. This reduced the time for the import to around 30 seconds, which is an improvement from 80 seconds but short of an order of magnitude improvement which should be possible. This led me to believe that there was either contention somewhere else, or that some of the constant factors are just too high at the moment.

Option 2 - Serialisation format changes

By default, Axon will use XStream to serialise events, which uses an XML representation. XML is quite verbose, and the Axon documentation even suggests using a different serializer.

Overriding the serializer is thankfully quite easy:

@Primary
@Bean
@ConditionalOnProperty(
    value = ["axon.use-cbor-serializer"],
    matchIfMissing = true
)
fun serializer(): Serializer {
    val objectMapper = ObjectMapper(CBORFactory())
    objectMapper.findAndRegisterModules()
    objectMapper.setSerializationInclusion(NON_ABSENT)
    return JacksonSerializer(objectMapper)
}

I opted for using Jackson with a ‘Concise Binary Object Representation’ (CBOR) JsonFactory. This resulted in a ~70% reduction in size for the serialized payload for most events. With XML:

postgres=# select avg(length(loread(lo_open(payload::int, x'40000'::int), x'40000'::int))) from domain_event_entry;
     avg
--------------
 433.69003053

and with CBOR:

     avg
--------------
 111.54379774

This didn’t have a huge impact on the run time of the importer, but is still a worthwhile optimisation.

Option 3 - Generated schema changes

You may have noticed in the SQL statments above that the current schema is using the PostgreSQL large objects functionality. From the PostgreSQL docs:

PostgreSQL has a large object facility, which provides stream-style access to user data that is stored in a special large-object structure. Streaming access is useful when working with data values that are too large to manipulate conveniently as a whole.

If we inspect the schema that’s being generated:

postgres=# \d domain_event_entry
     Table "public.domain_event_entry"
      Column      | Type | Nullable | Default
------------------+------+----------+---------
 meta_data        | oid  |          |
 payload          | oid  | not null |
 ...

The oid type here is an object identifier - a reference to a large object which is stored externally from the table. The events we’re writing are small enough that the overhead of reading them as separate streams is hurting performance rather than helping.

At least two people have had the same issue when using Axon with PostgreSQL, as evidenced by the questions on Google Groups and StackOverflow. The suggestion to customise the PostgreSQL dialect used by Hibernate seems to work, and further reduced the runtime to around 8 seconds.

Conclusion

Based on my very rough benchmarking, the three changes above have reduced the run time of the importer from around 80 seconds to 8 seconds. The code is all at the link above, and the optimisations are on by default.

There is surely more that can be done to improve performance, but that’s fast enough for now!

Vetted - importing data into an event sourced system

2018-01-30T00:00:00+00:00

See Vetted - a new project for some background.

A task I’m going to have to tackle sooner or later is importing data from the existing Access database. As I’m going to try my hand at event sourcing, this raises an interesting question:

Given an existing application with data stored in an relational database & no notion of events, how do you go about importing data into an event sourced system?

Options

At a high level, the initial options seem to be:

Try to reverse engineer/map the existing state to events
Have some sort of migration event event in your domain model (e.g. FooMigrated) which acts as a snapshot
Run everything through your new API as commands, and allow the API implementation to take care of creating the relevant events like normal

‘Recreating’ domain events

Option one above would be nice, but seems impractical at best and is more likely impossible. For every domain object (client, patient, payment, invoice, vaccination etc) I’d need to try and reverse engineer the real-world happenings that occurred to transition the object into its current state.

A ‘migration event’ as a snapshot

Originally when a colleague suggested this it conflicted with my understanding of the term ‘snapshot’. To me a ‘snapshot’ has always been about collapsing an event stream into a single event for performance reasons. When using this kind of snapshot, the original stream of events is still available.

The second kind of snapshot (which I didn’t see immediately) is a snapshot which is used as base data. When using a snapshot as base data, the collapsed state of the aggregate at the time the snapshot is all the information you have about the history of the aggregate.

It could also be argued that the migration is a meaningful domain event in its own right, and should be captured explicitly. A CustomerMigratedEvent could result in the creation of a new customer aggregate root in the same way that a CustomerRegisteredEvent does.

Run all existing data through the new API

It should be possible to write a script that reads data from the existing database, creates commands and posts those to the appropriate API. The relevant events would ultimately be created off the back of processing the commands, so all ‘legacy’ data should look exactly the same as anything created going forward.

The outcome is probably close to option one above, but with less manual work.

Next steps

So far I’ve been spending a lot of time on the technical concepts & design of an event sourced system, without doing much on the implementation side.

It’s hard to build a useful conceptual domain model without considering implementation issues, so I think it’s time I stopped debating concepts and wrote some code.

I’m planning to explore a little and gain an understanding of how building and executing commands would differ in practice from the ‘import event’ option above.

Acknowledgements

Thanks to Roman Safronov, Chris Rowe, Martin Fowler, Mariano Giuffrida, Jim Barritt and Nouman Memon for taking the time to reply and/or chat about event sourcing! Any good ideas are theirs, and errors are mine.

Vetted - choosing an appropriate license

2017-11-05T00:00:00+00:00

See Vetted - a new project for some background.

I’m hoping to start pushing some code soon. Before I do, it’s a good opportunity to do some reading into a topic that I am more ignorant of than I should be: software licensing. The following is what I’ve learnt so far.

Disclaimer: I am not a lawyer and this may be totally incorrect, and as such should not be used as the basis for any decision ever.

Once code is released as open source software, most common OSS licenses (GPL, BSD, MIT, Apache etc) do not allow for the revocation of rights granted under the license. This is a good thing. Imagine having to be prepared for any open source library/framework you’re currently using become proprietary software with no warning. Such uncertainty would severely limit the utility of open source software.

It is possible for the current copyright owners to relicense their creations. So, theoretically, any OSS software can be relicensed if all copyright owners agree. The important part is that this relicensing does not revoke the rights assigned under the previous license. So if I’ve released some software as open source software, I can decide a year later to relicense it and create a commercial version, with the following caveats:

I still own the copyright for the entire project
- I need to have been the sole contributor to the project, or to have ensured that contributors have assigned copyright to me for their work
The existing rights assigned under the OSS license remain in place
- If the license permits, anyone can fork the project at this point and develop/use their own version

Given what I’ve learned above, I’m planning to license the project under an OSS license, but I won’t accept any contributions until I’ve got some kind of Contributor License Agreement (CLA) in place.

This is likely the first of many posts where it might seem I’m researching a topic and deliberating a little excessively, given I have no working software or even a particularly interesting idea. It’s fairly premature to assume that there are going to be any contributors to this project other than myself. I don’t believe veterinary practice management software is so exciting that I am going to be swamped with contributions. However, as I mentioned earlier, this whole project is mainly a learning opportunity for me.

Further resources:

Vetted - a new project

2017-11-05T00:00:00+00:00

When I was in high school, I created a fairly basic application for managing a small veterinary practice. It’s written in Microsoft Access and is used by my parents to manage their mobile veterinary business.

I’m toying with rebuilding it as a web application. For its current users, the main benefits of this would be:

client contact details could be made available on a mobile device;
there will be no more (or at least fewer) issues with concurrent modification (merging two access database files that have been modified independently on different computers because Dropbox didn’t sync is no fun); and
it will be accessible anywhere with internet access, so that my dad could do accounts when he is away from home and has some downtime.

For me, it would mainly be a learning opportunity.

Naming things is not my strong suit, so I’m going with ‘Vetted’ for now.

I’m going to try to do the following throughout the project:

Apply domain driven design rigorously
Apply functional programming principles
Document failures and successes
Document my design heuristics
Develop in the open
Deploy continuously to ~~production~~ somewhere
Focus on adding the most valuable parts first (e.g. make phone numbers available online) & delivering vertical slices

I’m thinking that the basic architecture for now will be:

Single page application
Elm frontend
Kotlin backend
Maybe some kind of event sourced data store, as I’d like to see how badly I can shoot my foot off

While I want to build something functional, I also want to learn about a few techniques/patterns/tools that would be applicable on larger projects, so I might be making some choices which seems strange. I’ll try to call these out as they happen. I need to remember that I am not Google.

I’m hoping to publish posts more regularly about this project, so keep an eye out if you’re interested!

Notes

Design Heuristics

I attended the excellent Explore DDD conference this year and one of my favourite talks was Cultivating Your Design Heuristics by Rebecca Wirfs-Brock.

As defined in the talk, a heuristic is:

anything that provides a plausible aid (not a guaranteed aid) or direction in defining a solution but is ultimately irrelevant to the final product

Rebecca encourages everyone to consciously document and cultivate heuristics, learn others heuristics & discuss them and ultimately adapt (or wholesale replace) your own heuristics when appropriate.

It’s a great talk, and I’m going to try to document the heuristics that I find myself using while working on this project.

Develop in the open

I’m going to be writing about what I’m building, and the code will be available on GitHub. However, I haven’t yet determined how to license the project. My basic requirement is that I retain copyright and can relicense the project if that’s ever required.

Look out for a post on this in the near future.

Debugging slow bash startup files

2017-10-31T00:00:00+00:00

Recently I found that opening a new bash session (e.g. when opening a new terminal window) was getting a bit slow on my machine. I take reasonable care to make sure my dotfiles don’t get too crufty, and I keep them all in version control.

The following is a walk through of how I went about debugging the issue.

So, how does one go about profiling what bash is doing when starting a login shell/interactive shell?

My initial thought was to use some kind of system call tracing to see what files were being opened/executed. dtrace exists on OS X, so let’s try that:

sudo dtruss -ef bash

Sadly, the output isn’t overly useful due to System Integrity Protection. I don’t want to boot into recovery mode, so what are our options?

I regularly add set -o xtrace to my bash scripts … would the same thing work for my .bashrc? I added the line, and executed bash:

+ source /Users/mnewman/.bash_profile
++ export PATH=/Users/mnewman/bin:/Users/mnewman/perl5/bin:/Users/mnewman/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mnewman/.rvm/bin
++ PATH=/Users/mnewman/bin:/Users/mnewman/perl5/bin:/Users/mnewman/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mnewman/.rvm/bin
++ for file in ~/.{path,bash_prompt,exports,aliases,functions,extra}
++ '[' -r /Users/mnewman/.path ']'
++ for file in ~/.{path,bash_prompt,exports,aliases,functions,extra}
++ '[' -r /Users/mnewman/.bash_prompt ']'
...

It looks like that works (the above is showing the start of my .bash_profile, which is sourced from .bashrc). There is a lot of output there though, and we still don’t have any timing information. A little searching for variants of bash add timestamp to each line led me to an SO answer recommending ts. Looking at the manual page for ts:

$ man ts

NAME
       ts - timestamp input

SYNOPSIS
       ts [-r] [-i | -s] [format]

DESCRIPTION
       ts adds a timestamp to the beginning of each line of input.

       The optional format parameter controls how the timestamp is formatted, as used by strftime(3). The default format is "%b %d %H:%M:%S". In addition to the regular strftime
       conversion specifications, "%.S" and "%.s" are like "%S" and "%s", but provide subsecond resolution (ie, "30.00001" and "1301682593.00001").

       If the -r switch is passed, it instead converts existing timestamps in the input to relative times, such as "15m5s ago". Many common timestamp formats are supported. Note that
       the Time::Duration and Date::Parse perl modules are required for this mode to work. Currently, converting localized dates is not supported.

       If both -r and a format is passed, the existing timestamps are converted to the specified format.

       If the -i or -s switch is passed, ts timestamps incrementally instead. In case of -i, every timestamp will be the time elapsed since the last timestamp. In case of -s, the time
       elapsed since start of the program is used.  The default format changes to "%H:%M:%S", and "%.S" and "%.s" can be used as well.

So far so good, it looks like we could use ts -i and get the duration of every command! I’d like to try this out, but how can we redirect the xtrace output to ts?

Some further Googling led me to this SO answer, which suggests using the BASH_XTRACEFD variable to tell bash where to write its xtrace output. After some trial and error, I added a few lines to my .bashrc:

# open file descriptor 5 such that anything written to /dev/fd/5
# is piped through ts and then to /tmp/timestamps
exec 5> >(ts -i "%.s" >> /tmp/timestamps)

# https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html
export BASH_XTRACEFD="5"

# Enable tracing
set -x

# Source my .bash_profile script, as usual
[ -n "$PS1" ] && source ~/.bash_profile;

Upon restarting bash, this produces (a lot of) output in /tmp/timestamps, and each line contains an incremental timestamp, like so:

000046 ++ which brew
003437 +++ brew --prefix
025518 ++ '[' -f /usr/local/share/bash-completion/bash_completion ']'
000741 +++ brew --prefix

These particular lines tell me that a brew --prefix command executed and took 20ms.

With output like the above, I had enough info to track down a couple of slow loading scripts (like sourcing nvm.sh) and remove them from my .bashrc/.bash_profile.

Migrating a GitHub Pages blog with a custom domain to HTTPS

2017-07-06T00:00:00+00:00

At the time of writing, this blog is hosted on GitHub Pages, which has been working well since I set it up a few years back.

The only thing that has bugged me for a while now is that the whole site was served over HTTP, rather than HTTPS.

I wanted to move to move this blog to HTTPS, but with some constraints:

Continue using GitHub pages (it’s free and easy, I don’t want to manage a server)
No certificate renewal (smart me plans for stupid me, who would surely forget to renew a cert)
Continue using my domain (mdjnewman.me)
No cost

GitHub pages doesn’t support HTTPS for custom domain names, as it uses a certificate with a wildcard SAN of *.github.io.

CloudFlare offers HTTPS on a free plan, which Troy Hunt has written about before.

It looks like this will meet my constraints above - I get to keep using GitHub Pages, I don’t have to manage a cert (CloudFlare takes care of this), and I can keep using my custom domain.

The steps I followed to do this were relatively simple:

Exported a zone file from current nameservers
Completed the CloudFlare onboarding, during which I imported the above zone file
Updated the authoritative DNS servers for my domain to the *.ns.cloudflare.com name servers:
Tested the site out, fixed a CSS link that was loaded over HTTP
Forced HTTPS in CloudFlare:

… and that was it. I finished this in part of an afternoon.

Shortcomings

There is one major shortcoming with this setup: there is no certificate validation between CloudFlare and GitHub (CloudFlare supports fetching from an origin without validating certificates, which is the option I’ve chosen - ‘strict’ HTTPS can be enabled for most use cases).

As we mentioned before, the GitHub cert is valid for *.github.io, and we’re using my custom domain, which is mdjnewman.me.

If we switched off the custom domain on GitHub, and did some smarts in CloudFront to rewrite requests so that the request to GitHub was using mdjnewman.github.io, then we’d get HTTPS all the way to GitHub servers.

CloudFlare does support rewriting HTTP Host headers , but it’s an enterprise feature.

I could switch to using CloudFront with an AWS Certificate Manager cert, which would meet all the above constraints except for ‘no cost’ (admittedly, my tiny blog doesn’t get much traffic, so the cost would be minimal).

Given that most of the shenanigans with injecting content into web sites happens at the last leg of a connection (I’m looking at you, dodgy internet cafe), I’m happy that the new setup for this blog mitigates that problem and am willing to accept the cost/security trade-off. While it’s possible for someone to perform a man in the middle attack and impersonate GitHub, given my site has no sensitive information I’m not too worried about this threat model (Troy Hunt also wrote about this trade-off).

When should I force push after rebasing?

2017-07-05T00:00:00+00:00

tl;dr: assuming it’s your own branch, immediately.

On the last few projects I have been a part of, I have been the defacto Git problem solver.

(obligatory xkcd)

On more than one occasion, I was presented with something like the following:

$ git l master
* 232e985 (origin/master, origin/HEAD, master) Feature 4
* 94989d0 Feature 3
* 296d0b3 Feature 2
* 4b856cd Feature 1

$ git l feature-5
* 5c31f6d (HEAD -> feature-5) Feature 4
* 420f83b Feature 3
* 724236a (origin/feature-5) Feature 5
* 296d0b3 Feature 2
* 4b856cd Feature 1

In the above, the commits adding features 3 and 4 on each branch are logically the same, but on the feature-5 branch they’ve somehow ended up after the commit adding feature 5.

The team was using feature branches, and the author of feature 5 above was trying to rebase their changes onto master, but somehow ended up inserting their commit between commits on master. At this stage, you’re in a pretty bad place, as you’ve diverged from master.

How did this happen? I’m guessing it was the following series of events:

Developer branches from master to create feature-5
Features 3 & 4 are pushed to master
Feature 5 is committed to the feature-5 branch
Developer runs git pull --rebase origin master or similar
… some time passes …
Developer runs git pull --rebase without really thinking about it

After the first few steps above, we have something like the following:

So far, so good. We want to rebase our changes onto master, so that we can test and push our code. After git pull --rebase origin master:

Still looking good. At this stage, we could git push --force origin feature-5 and all would be well in the world.

But what happens if we go for a tea and forget what we were doing (or we use an overzealous git GUI tool), and we try to rebase onto origin/feature-5?

What we see above is the result of Git rebasing commits 94989d0, 232e985 and 44e1c44 on top of origin/feature-5. As the commit ID is computed by hashing of the contents of a commit and its parent, the same logical commits from master now exist on our branch with different IDs.

This could have been avoided if we followed this rule of thumb:

If you’re working on your own branch, always push immediately after rebasing.

Some people on the team were seeing a message like the following:

Your branch and 'origin/feature-5' have diverged,
and have 3 and 1 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

and assuming that they should git pull --rebase, which in this case is exactly what you don’t want.

If you’re looking for further reading Getting Started - Git Basics covers the fundamentals and LearnGitBranching is a great interactive tool.

Diagrams were created using Gitgraph.js, and this script creates an example repository.

Monad transformers

2016-07-31T00:00:00+00:00

I’ve been spending some time working through Haskell Programming From First Principles recently which is a very comprehensive introduction to functional programming and Haskell.

One of the exercises to implement a simplified command line version of Morra, which involves keeping track of scores and reading user input.

My initial method to play a round looked something like this:

playRound :: Config -> Scores -> IO (Bool, Scores)
  playRound config scores = do
  putStr "P1: "
  p1HandMaybe <- getHumanHand
  case p1HandMaybe of
    Nothing -> return (True, scores)               -- error case 1
    Just x -> do
      putStr "P2: "
      p2HandMaybe <- getHumanHand
      case p2HandMaybe of
        Nothing -> return (True, scores)           -- error case 2
        Just y -> do
          let evenWins = (x + y) `mod` 2 == 0
          return (False, updateScores config evenWins scores)

playRound takes the configuration for the game and the current score, and returns a side effecting computation that will return a tuple with the new scores and a boolean indicating if the game is finished.

The method getHumanHand used above returns a IO (Maybe Int), which can be interpreted as a side effecting action that might return an integer (in this case, the side effect is reading from the console and we can’t trust the user to enter an integer, hence the Maybe).

The problem then is that we’re then manually unpacking these Maybe Int values, which leads to the ugly nesting and case statements. However, we can see on the lines marked ‘error case’ above that the handling for both cases is the same - we assume that if the user has entered something other than an Int that they want to end the game.

I recently learned about Monad transformers, which allow you to compose monads. In this case, we want to compose the Maybe monad with the IO monad, so we will use MaybeT.

Rewriting getHumanHand to return a MaybeT IO Int and rewriting playRound results in the following:

playRound' :: Config -> Scores -> IO (Bool, Scores)
playRound' config scores = do

  newScores <- runMaybeT $ do
    liftIO $ putStr "P1: "
    p1Hand <- getHumanHand'
    liftIO $ putStr "P2: "
    p2Hand <- getHumanHand'
    let evenWins = (p1Hand + p2Hand) `mod` 2 == 0
    return $ updateScores config evenWins scores

  return $ case newScores of
    Nothing -> (True, scores)
    Just x -> (False, x)

The nice thing about this implementation is that we’ve avoided the need for pattern matching, as the do block above where we’re dealing with the potentially failing computations will immediately short circuit and return a Nothing if either user fails to provide a legitimate value.

This is the first time I’ve actually used a Monad transformer, and it was good to see how it cleans up the implementation of playRound.

BFPG talk - Functional Programming in Scala Chapters 7 & 8

2016-06-19T00:00:00+00:00

In May, I presented a talk based on Chapters 7 & 8 of Functional Programming in Scala at the Brisbane Functional Programming Group.

The talk explores the design and implementation of two functional libraries, one for parallelism and one for property based testing.

The video of the talk (also embedded below) is now available online at the BFGP talks site, along with a link to the slides.

See http://talks.bfpg.org/past.html for other past BFPG presentations.

-Xlint:unchecked and -Werror for Java projects with Gradle

2016-05-04T00:00:00+00:00

I was recently working in a codebase where there were chunks of Java code that used generic interfaces from the Apache Commons Collections project as well as java.util Collections in a type unsafe way, even though the interfaces/classes that were being used supported generics.

Our gradle build was emitting the following warning:

:compileJavaNote: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

This warning should be an error in my books. It’s possible to get this behaviour by adding the following to your build.gradle file:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:unchecked" << "-Werror"
}

This way, when our code is built everyone is aware when they’re using unsafe operations, and can either fix the issue or supress the warning with @SupressWarnings("unchecked") if it’s intentional.

Pedantic console for JavaScript tests

2016-01-30T00:00:00+00:00

When running JavaScript tests that fire up a browser, it can be useful to ensure that console.warn and console.error aren’t being called, and to have your automated build fail if this starts happening.

If you’re using a framework like Jasmine, you can place the following file in your helpers directory to ensure that an exception will be thrown whenever a call is made to console.warn or console.error:

console.error = (function () {

  var originalConsole = console.error;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.error called, failing test');
  }

  return myError;
})();


console.warn = (function () {

  var originalConsole = console.warn;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.warn called, failing test');
  }

  return myError;
})();

I recently used this to locate all the places that a deprecated Moment.js feature was being used in a codebase.

Haskell stack, Yesod and Docker

2016-01-01T00:00:00+00:00

The why

Over the break I’ve been working on a web app to replace a fairly old MS Access Database that I built for my Dad to use in 2009 (he has a mobile vet business).

This seemed like a good chance to try out Yesod, a web framework for Haskell. The Yesod philosophy is to leverage the Haskell type system wherever possible. For example, in the Hamlet templating language everything from generating URLs to including static files and generating forms is checked at compile time.

stack is a cross platform build tool for developing Haskell programs which (when combined with Stackage makes Haskell development much more enjoyable.

I’d recommend stack for any new Haskell project, and hopefully this post can point someone else in the right direction for a stack/Yesod/Docker project.

I built a very basic site with Yesod and stack locally on OS X, based off the yesod-postgres template (stack new mysite yesod-postgres). The yesod dev server makes this all very easy, just running stack exec -- yesod devel will keep Yesod running and recompiling the app whenever any changes are made.

Soon it was time to make the app available somewhere in order to start getting some feedback. At this stage, the most basic requirement for the hosting was that it be low cost. Digital Ocean came to mind (use this link for $10 credit), and I’m now using their one-click Docker droplet at USD $5 per month. This host can run both the Yesod frontend and the PostgreSQL database.

Binaries built locally on OS X won’t run on a Linux-based Digital Ocean droplet, and it was a bit early to spend $$ on something like Snap or Travis, so I needed a VM or container to run the build in.

The how

Set up

stack has support for Docker to run the build and package the application, but it isn’t currently supported when using boot2docker (see these issues), so I used a simple Vagrantfile to start up a beefy VM:

Vagrant.configure(2) do |config|

  config.vm.box = "puppetlabs/centos-6.6-64-nocm"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4096"
    vb.cpus = 8
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
  end

end

Centos 6.6 might seem like an odd choice for a development environment, but the Vagrant box was already on my laptop, so using it saved me the download time (bandwidth is a precious commodity in semi-rural Australia).

I didn’t bother setting up provisioning properly, but here are the commands that are needed to get stack working:

# Add FP Complete repo & install stack
curl -sSL https://s3.amazonaws.com/download.fpcomplete.com/centos/6/fpco.repo | sudo tee /etc/yum.repos.d/fpco.repo
sudo yum -y install stack

# Add EPEL repo and install Docker
sudo rpm -iUvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo yum -y install docker-io

# stack requires the Docker client to be able to connect
# to the daemon without root, so add vagrant user to dockerroot group
sudo gpasswd -a ${USER} dockerroot

# At this point, you'll need to log out and back in for
# the group change to take effect

# Setup iptables to allow Docker links to work
sudo iptables -t filter -A DOCKER -d 172.17.0.0/16 -i docker0 -j ACCEPT

Adding the vagrant user to dockerroot comes with all the usual security issues, but I’m not too worried about that here. There are some other suggestions in the stack documentation for how to handle this.

Running the build

So, stack and Docker are now both installed. Enabling stack’s Docker integration on the VM is as simple as modifying the ~/.stack/config file:

[vagrant@localhost ~]$ cat ~/.stack/config.yaml
docker:
  enable: true

This gives the advantage of using Docker to run the builds on the VM, while not impacting the yesod devel workflow on OS X.

By default stack will use the fpco/stack-build repository to obtain an image to run the build in. This suits me, as there is also fpco/stack-run for running the binaries once they’re compiled. The build image includes everything that’s necessary for building the whole of Stackage, whereas stack-run is more trimmed down.

Running stack build kicked off the build in the Docker container, which completed successfully as all required dependencies (e.g. libpq and pg_config) were already in the image. Thanks to FP Complete for setting up these Docker containers for everyone to use!

Packaging Docker image

stack’s support for creating container images is a relatively recent addition, and currently isn’t covered well in the documentation. There is a reference to the image configuration section here, which was enough for my needs. After added the relevant bits, my stack.yml looks something like:

resolver: lts-3.20

packages:
- '.'

extra-deps: []

flags:
  mysite:
    library-only: false
    dev: false

extra-package-dbs: []

# This isn't required as it's set in the user config on the VM
# docker:
#     enable: true

image:
  container:
    name: mdjnewman/mysite
    base: fpco/stack-run
    add:
      config: /app/config
      static: /app/static

With that in place, running stack image container produced a Docker container ready to be pushed to Docker Hub:

[vagrant@localhost vagrant]$ stack image container
Sending build context to Docker daemon 34.25 MB
Sending build context to Docker daemon
Step 0 : FROM fpco/stack-run
 ---> db9b2a858ef5
Step 1 : ADD ./ /
 ---> e682c572d7ed
Removing intermediate container 1ce35ed1ccea
Successfully built e682c572d7ed

Running a regular docker push published the image to the registry.

Deploying to Digital Ocean

After all these steps, the final package was in Docker Hub and it was just a matter of running the following commands on the droplet, and the site was live:

docker run --name=postgres -e POSTGRES_PASSWORD=$(uuidgen) -e POSTGRES_USER=postgres -d postgres:9.3

# Yes, I know UUIDs don't make the best passwords :)

docker run              \
    -d                  \
    -w /app             \
    --link postgres:postgres         \
    -p 3000:3000        \
    mdjnewman/mysite    \
    /bin/bash -c 'PGHOST=$POSTGRES_PORT_5432_TCP_ADDR PGPASS=$POSTGRES_ENV_POSTGRES_PASSWORD /usr/local/bin/mysite'

The last command is running the Yesod server, with environment variables set to the values provided by Docker (see here for info about setting configuration variables in Yesod).

The results

I’m very happy with the Yesod workflow, and stack’s Docker integration makes deploying a lot easier as I don’t have to worry about what packages are in my build and test environments. Seems like a good compromise between manually copying files around and having a full CD pipeline.

Using stack build with a LTS resolver and an isolated container is also the holy grail of repeatable builds!

There is still a lot that would need to be done (improving security, backups, logging etc) before this was closer to a production environment, but it’s good enough as a development region.

This whole process (including writing this post) from having something that I wanted to deploy to being able to view the live site took less than a day, and I was learning a lot along the way. Deciding where was the best place to host the site and waiting for Docker to pull the fpco/stack-build image took up most of the time!

Simon Peyton Jones on maintenance with strong types

2014-06-26T00:00:00+00:00

People don’t like fixing type errors, but they are a lot easier to fix than runtime errors… The single biggest thing going for static typing is not so much that it helps you write your program in the first place, but it helps you maintain your program.

GHC itself is an example – it’s now 150,000 lines of Haskell & twenty years old and I regularly refactor it in pretty large scale ways. I just wouldn’t dare do that if I didn’t have static typing to keep me sane. It’s this enormous code base, chunks of which I’ve forgotten about, and yet I can confidently make systemic changes to it because I know the type checker is going to catch all the places that change needs to go.

Simon Peyton Jones, speaking on Functional Geekery Episode 11.

The part about GHC is anecdotal evidence, to be sure, but it makes the point nicely.

Why Functional Programming Matters

2014-06-13T00:00:00+00:00

The following is the conclusion from a paper entitled ‘Why Functional Programming Matters’:

In this paper, we’ve argued that modularity is the key to successful programming. Languages that aim to improve productivity must support modular programming well. But new scope rules and mechanisms for separate compilation are not enough — modularity means more than modules. Our ability to decompose a problem into parts depends directly on our ability to glue solutions together. To support modular programming, a language must provide good glue. Functional programming languages provide two new kinds of glue — higher-order functions and lazy evaluation. Using these glues one can modularize programs in new and useful ways, and we’ve shown several examples of this. Smaller and more general modules can be reused more widely, easing subsequent programming. This explains why functional programs are so much smaller and easier to write than conventional ones. It also provides a target for functional programmers to aim at. If any part of a program is messy or complicated, the programmer should attempt to modularize it and to generalize the parts. He or she should expect to use higher-order functions and lazy evaluation as the tools for doing this.

Of course, we are not the first to point out the power and elegance of higher-order functions and lazy evaluation. For example, Turner shows how both can be used to great advantage in a program for generating chemical structures. Abelson and Sussman stress that streams (lazy lists) are a powerful tool for structuring programs. Henderson has used streams to structure functional operating systems. But perhaps we place more stress on functional programs’ modularity than previous authors.

This paper is also relevant to the present controversy over lazy evaluation. Some believe that functional languages should be lazy; others believe they should not. Some compromise and provide only lazy lists, with a special syntax for constructing them (as, for example, in Scheme). This paper provides further evidence that lazy evaluation is too important to be relegated to second-class citizenship. It is perhaps the most powerful glue functional programmers possess. One should not obstruct access to such a vital tool.

J. Hughes, “Why Functional Programming Matters,” Comput. J., vol. 32, pp. 98–107, 1989.

This paper was written twenty years ago – the more things change, the more they stay the same. I’d urge any programmer (whether currently interested in FP or otherwise) to read this paper if you haven’t already.

ThoughtWorks LevelUp EXP Brisbane – Take 2

2014-05-05T00:00:00+00:00

LevelUp EXP Brisbane Attendees (thanks Alexandra Tran for the photo!)

ThoughtWorks ran another great express LevelUp event on Saturday at their Brisbane office. LevelUp events aim to help students bridge the gap between university and their first full-time job. LevelUp EXP is a mini-conference with a number of talks and hands on sessions, as well as lots of opportunities to mingle with the ThoughtWorks employees and other attendees. One theme that recurred through almost every talk of the day was focussing on the user at every stage of the development process – if you’re not building something that your users can use, then you’re wasting time. This does mean users, not customers. You can be building a product for a client, but it’s ultimately the user’s experience that will determine success.

Real Project Example: Domino’s HTML5 in Brisbane

First up was Mark Ryall (@markryall) discussing the recent consolidation of three separate websites into a single HTML5 site for the Dominos pizza franchise in Australia, New Zealand and the Netherlands. The new site was to replace separate Flash, mobile and accessibility-focussed sites, and was a greenfield project. From a software process perspective, the project kicked off with a two-week inception where BAs, developers, designers and testers worked together for two weeks to get initial requirements worked out. Mark pointed out that this is an effective alternative to the business struggling to work out requirements on their own for six months before getting anyone else involved. It reminded me of the quote that’s often attributed to Henry Ford:

If I’d asked people what they wanted, they would have said a faster horse.

The point is that the business doesn’t always know what they want, and even if they do, they may not be able to articulate that in such a way that both parties have the same vision. The rest of the project ran in two-week iterations, with showcases at the end of every iteration. Mark mentioned that the biggest challenges (from a technical perspective) in this project were internationalisation (I18N) and localisation. As Mark explained it, internationalisation ensures everything is available in the correct languages for the correct region, whereas localisation focusses on differences such as different address formats (another example of what this site in particular has to handle is that Domino’s in NL doesn’t provide catering, like it does in AU). There was a heavy emphasis on test automation in this project, and Selenium WebDriver was used for automated browser testing. WebDriver allows single tests to be written to target more than one browser, and are more robust than trying to guess what the browser is doing and interspersing your tests with loads of sleep() calls. In order to make tests more readable for everyone on the project (BAs/the business included), Specflow was used. Specflow is a .NET alternative to Cucumber, and uses a similar declarative style to define specifications & expected behaviour. Tests are then generated from these easy to read specifications, targeting whichever testing framework you’re using (NUnit in this case). Given this site was replacing an existing site with a focus on accessibility, Domino’s required this site to be just as accessible for people using screen readers and keyboard only navigation. Accessibility is something that is easy to overlook, but every developer should make the effort to do these things correctly (using alternative text, ARIA tags, skip links for navigation etc). If you want to know more about how important accessibility is, Scott Hanselman recently did an interesting interview with Katherine Moss, a blind software technician.

The Lego Game

The Lego Game is a game designed to give you a taste of the Agile software development process. Our task was building a Lego ‘animal’ with a client in mini sprints lasting a few minutes each. We worked through everything from estimation to a retrospective in less than ten minutes, and completed three iterations with a client who wasn’t afraid to change their mind. An interesting resource that was mentioned by the ThoughtWorks employees running the Lego game was the Retrospective Prime Directive by Norman Keith:

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

I thought this was a good guideline for running retrospectives, as it ensures that you focus on what went well and what can be improved, and not on trying to assign blame. There are plenty of pages describing the agile Lego game on the net if you want to read more.

Agile 101 with a Business Analyst

Agile 101 with a BA by Henry Lawson was the first of the twenty-minute lightning talks for the day. In twenty minutes, Henry managed to cover a brief history of Agile, the Agile manifesto and different Agile methodologies. There was more about Agile on the cards but the twenty-minute timeline had to be adhered to! The following paragraphs sum up the main points of the talk.

Agile manifesto: the Agile manifesto is what gave the movement towards iterative, lightweight processes a name. Written in 2001 by several prominent figures in the software industry, the core of the Agile manifesto is the following four points:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The differences between traditional Waterfall methodologies and Agile: The waterfall model is a sequential design process, where each stage is completed and locked-down before the next is started.

Waterfall model of system development – avoid it! (you want the cycle above to take two weeks, not twelve months) (source)

While waterfall might work for other engineering projects (like building a bridge), it’s not really suitable for software development (too many software projects have elements of a wicked problem). I won’t go into the benefits of Agile or Agile processes in too much detail here, as there is plenty more information available.

Different agile methodologies: Examples of agile methodologies include Scrum, Kanban, Lean, XP and Crystal Clear. The main difference between these methodologies is the practices, such as daily Scrum meetings or pair programming in XP. The general processes are similar, and there is no reason you can’t mix and match.

Functional Programming

Hugo Firth (@hugo_firth) was giving the next lightning talk, this time with a quick intro to functional programming concepts. There are a number of programming paradigms, such as procedural and object-oriented (OO). Procedural code (e.g. C/BASIC code) doesn’t provide many ways to create abstractions and manage state. OO, such as you’ve probably encountered in Java/C#/C++ allows abstractions to hide global state. Functional programming on the other hand is all about removing state where possible, and programming with pure functions. Hugo went through some common functions you’ll meet in functional programming, such as map & filter by providing some simple examples in Clojure. If you want to see a less rushed presentation by Hugo, his recent talk at the Brisbane Functional Programming Group is available on Vimeo.

BFPG is a great community if you’re looking to learn more about functional programming – I’ve always found everyone there to be extremely willing to help.

Working in different environments… from Heaven to Hell

The last lightning talk in the first session was by Claudia Ferreira (@claudia_onfire), providing some practical advice for dealing with difficult work environments. You shouldn’t assume that the end of uni is the end of the hard work. Starting a new job, meeting new people and adapting to company culture is just as hard. Also, you don’t want to fall into the trap of thinking that you know everything and that you can stop learning. There is an almost endless variety of work environments, some good and some not so great. Claudia showed the following clip from The Matrix and told us about a time she was in this exact position.

Claudia’s boss basically said you’re here to work, not think, and there are hundreds of people who can fill this spot if you aren’t happy with that. Hopefully you won’t find yourself in a situation this extreme, but chances are you’ll have a boss you’re not overly excited about working with at some point in your career. If you find yourself in this situation, don’t just bottle it up and say nothing. Your team is there to help, you should be able to trust them and speak to them about issues. It’s important to have a good relationship with your team, as they’re the people that you spend the majority of your day with. So don’t stress if you’re confronted with a difficult situation, just take a deep breath, speak up if you really don’t agree and be the bigger person.

The Way of the Consultant

Sarbashrestha Panda has been a consultant for 5-6 years, working as an analyst and project manager on many projects. There are many types of consulting, from technical to business strategy. Technical consulting still has a strong focus on the business needs, no matter what area you’re in you are helping solve a problem the business is having. As Panda explained it, the role of the consultant is to always ask the difficult questions and have the difficult conversations. You want to make a problem visible, make people think and help to implement solutions. As a consultant, your knowledge should drive change.

Most problems are people problems - recognise that everyone has a different perspective (source)

If you’re working as a consultant, embrace the opportunity to work on many different projects – you can’t develop a broad skill set without experience. You can be a consultant in any language with any technology, and after a few years should develop the skills to become a consultant for a specific domain. Don’t let yourself get cornered into a specific technology – technologies change far too fast.

UI Dev. What and Why?

Eru Penkman gave the next talk, which was originally entitled ‘UI Dev. What and Why?’, but seemed to have transformed into a talk about programmer culture and day-to-day life. Eru pointed out a bunch of things that a lot of programmers/developers have in common. Most of the people in the room had read XKCD at some point, everyone had used Stack Overflow, lots of people read similar blogs etc. Having all these elements in common helps when working in a team, which as Eru pointed out is pretty much what development is about. It’s very rare to have a project where you don’t interact with a number of people throughout the day. There are plenty of resources for developers, as everyone uses Stack Overflow it means that SO can answer almost any question, and lots of problems you might want to solve have already been solved by one open source project or another. However, don’t just consume these resources – make accounts on SO/GitHub and start contributing!

What they didn’t tell you about Testing!

Do not assume anything. Ever.

The final lightning talk for the day was by Leonor Salazar & Brian Mason, covering testing and quality. The first point that Leo & Brian drilled home is that quality is subjective. Everyone has different ideas of quality, and you can’t do much meaningful testing without knowing what quality means for your project. Quality as defined by the users of the product is the most important. Everyone on a project should be thinking about quality, it’s not something that one person can ‘add in’ at the end. There is also a simple and compelling economic argument for having quality on everyone’s mind: it’s up to 1000x times more expensive to fix a bug in production than it is in analysis. Don’t let that cripple you or force you into a big-design-up-front type approach, but keep quality at the front of your mind.

Development is solving problems, testing is asking questions.

You need to ask yourself four questions:

Are we doing the right things?
Are we doing the things right?
Are we getting them done well?
Are we getting the benefits?

There is no point obsessing over whether you’re doing things right if you’re not doing the right things! Testers also have the opportunity to improve process, and take the product in new directions.

User Testing what you Build – Hands On Session

Carrying on with the theme of user first was Pete Chemsripong, with a hands on session where each team had to develop a ‘product’ that was then tested by a user. For example, a media remote control for a user with boxing gloves on or a lift control panel for someone carrying 15 bags of groceries. To design a good user experience, you need to ask why you’re building anything in the first place. You need to understand why people would use this product. Ultimately the answer to a question like ‘What makes Google better than Bing?’ is always that it offers a better user experience and satisfies the users needs. It doesn’t matter what you build or how you build it if people don’t enjoy using it. User testing also doesn’t have to be expensive. You can even grab a dev from another team who hasn’t touched your project. Chances are they’re going to find a lot of the same issues as a ‘real’ user of the software, allowing you to pick the low hanging fruit for minimal cost. Pete showed a video from a recent ThoughtWorks experiment, where an in-store innovation lab at a Woolworths store produced loads of great feedback in a very short time. The other ThoughtWorkers present mentioned Usability Testing with Morae as a good read for more information about inexpensive usability testing.

TDD – from woe to go in 10 minutes – Hands On Session

As professional developers, the burden of proving a system works lies with us.

Steve Morris held the floor for the final session of the day, and did a great job showing everyone how simple test driven development can be. TDD provides the following benefits:

Shows the code does what you say it does
Improves quality by making code more testable
Allows you to figure out what to test only once – no more writing a monolithic class/function and then spending time working through what needs to be tested over and over
At any time, you can see what is broken and whether your code is shippable (as there is never code that isn’t covered by tests)

TDD is coding with intent – don’t code by accident.

The ‘test’ in TDD could just as easily be replaced by ‘expectation’, ‘behaviour’, ‘specification’ or ‘intent’. It’s as much about helping to define expected behaviour and a minimal specification as it is about testing actual code. It doesn’t matter if you’re a developer or not, TDD provides the same benefits to BAs, managers and QA testers. I couldn’t write this post without including the TDD mantra, lest Steve think we all learnt nothing at all from the talk. The TDD mantra is as follows

Red – write a minimal failing test
Green – write the minimum amount of code to make the test pass
Refactor – incrementally refactor the code (both code under test and the tests themselves – test code is first class code!)

Writing tests before code forces you to think about naming, architecture, APIs and design without considering implementation details. This helps with design and forces us to progress with intent, to avoid building unnecessary features. The refactor step encourages design improvements in incremental steps, rather than writing a bunch of code and then struggling through a ‘big bang’ refactor later in the project. Failing tests are not a failure of any kind, rather, they are statements of intent. The minimal failing test doesn’t even have to compile. A test that doesn’t compile signifies an intent to write the code to make it compile. In this way tests are documentation of future intent as well as progress to date. The hands on component of this talk included getting testing environments for C# and Node.js set up and writing our first code using TDD. Set-up for C# is as simple as installation of a testing framework like NUnit, which in a newer version of Visual Studio is as easy as running

Install-Package Nunit

in the package manager console. The set up for Node.js was just as simple, with npm installed simply run

npm install karma
npm install phantomjs
npm install jasmine

from your favourite shell. In the talk, it was possible to get this environment set up and have our first test running in under ten minutes for each environment. There really isn’t any excuse for not giving TDD a try! Steve also pointed out that TDD (like most concepts in software development) isn’t as new as you might think.

The second LevelUp EXP event in Brisbane was eye-opening, and I think the attendees learnt a lot to complement what is taught at university. If you attended/presented and I’ve omitted anything, please let me know and I’ll fix it up. My post about the first Brisbane LevelUp event is here, and more about ThoughtWorks LevelUp can be found on Twitter (@TW_LevelUp) and Facebook.

Working on Haskell code in vim

2014-04-17T00:00:00+00:00

I decided to get a fresh vim install properly set up for some Haskell development, and went down the road of using hdevtools, vim-hdevtools and Syntastic. Below are notes on some issues I encountered setting up vim-hdevtools.

The instructions on the GitHub page for hdevtools & vim-hdevtools linked above are simple enough, but I ran into a couple of issues. I’m using the sandbox feature added in Cabal 1.18, and by default hdevtools doesn’t know to look for a .cabal-sandbox folder. So, when I tried to use any of the hdevtools commands, I received an error message similar to the following:

Error loading targets
(file):
Could not find module `(module)'
Use -v to see a list of the files searched for.

This is an issue that a few people have run into, and I found a link to this solution in the archives of the #Haskell IRC channel. Adding it to my .vimrc seems to have fixed this error, and I can now use all the hdevtool commands in vim. I’m still having issues with *.cabal files that specify default-extensions (vim-hdevtools doesn’t seem to pick these up).

Also, if you get an error message similar to the following:

E492: Not an editor command: ...

make sure that hdevtools is on your path and that you have plugin files for specific file types enabled by adding

filetype plugin on

to your .vimrc.

Installing self signed certificates on Android (Jelly Bean)

2013-11-06T00:00:00+00:00

I was pretty happy when I found out Android now supports installing self signed certificates, but had some trouble getting the certificate I use for administering this blog installed.

So, a quick note for anyone who is attempting to install a self signed SSL certificate on Jelly Bean Android devices: you can only install a root CA certificate (i.e. a certificate with “Subject Type=CA”).

Trying to install a certificate for a single site (i.e. a certificate with “Subject Type=End Entity”) isn’t possible as far as I can tell. I don’t have any references for this, it’s just based from my experience, so let me know if you’ve experienced otherwise.

Two-factor authentication and ‘trusted’ devices

2013-10-30T00:00:00+00:00

“I challenged hackers to investigate me and what they found out is chilling” is an interesting article – it’s the story of a couple who challenged a hacking team to pen-test their digital lives. To give some context to the following quote, their laptop has been compromised and user ID and password for their on-line banking account at Chase has been stolen by the ethical hacking team:

Chase.com uses a two-step verification system, which momentarily stymied SpiderLabs’ hackers. Every time she or I logs on from an IP address that Chase doesn’t recognize, it offers to send us an activation code via text to our mobile phones. But a search of Charlotte’s hard drive revealed Chase cookies, which the team copied and used to convince Chase that she was logging in from home.

You could argue that if your PC has been this thoroughly compromised then you have bigger problems, but it still points out how limited some two-factor authentication systems can be if you allow devices to be remembered/trusted.

ThoughtWorks LevelUp EXP Brisbane

2013-10-29T00:00:00+00:00

On Saturday I was lucky enough to attend ThoughtWorks LevelUp EXP at the ThoughtWorks Brisbane office.

As well as plenty of opportunities to chat with the ThoughtWorkers (and Chad!) who kindly gave up their Saturday, there were six presentations throughout the day:

Opening Keynote: Design Thinking

Ben Melbourne (@benmelb) set out to teach us about design thinking, and how it’s important for any project. The main points that I took away from his talk were:

Visual design is only a small part of design.
UX research is not strictly academic or scientific – iterate quickly and rely on feedback to get the best results in the shortest time.
Embrace constraints to get the most out of a situation. Design is harder when you have a green field to work in – if you have constraints, don’t fight against them, use them for inspiration.
Everyone is a designer – if you’ve ever been on a project without a designer, it was you.
Always ask if the current approach is the best use of business capability.

Functional Programming for Fun and Profit

Next up was James Ottoway (@jms_) with a good intro to functional programming. He started out by explaining declarative languages and reactive programming, as well as DSLs and closures and how they can be used to cut complexity.

Declarative languages focus on expressing the logic of a computation, not how to carry out that computation. Regular expressions, Chef/Puppet configuration files, and SQL were all used as examples of declarative languages. In James’s words:

We’re expressing what, not how. And that’s a good thing.

Expressing what you want to do (and not how it must be done) frees up the compiler/interpreter to make decisions about implementation at runtime, and eases the burden on you as a programmer.

When talking about reactive programming, James used Excel (everyone’s favourite spreadsheet application) as an example of a reactive program. This is a really simple analogy which I think helped everyone grasp the idea of reactive programming quickly. He also pointed out that the windows GitHub client is built using reactive programming (using ReactiveUI from what I can see).

As someone who’s trying to learn Haskell at the moment, this section about of this talk about closures was interesting to me. Closures aren’t something that I was seeing in Haskell, perhaps because I’m still battling with unfamiliar syntax (turns out they’re pretty much everywhere). A closure is just a function together with a referencing environment, and they’re pretty essential for functional programming.

Business Analysis

After lunch we heard about the life of a BA from Cathie Hagen (@catherist) and Jiangmei Kang. BAs are an integral part of every stage of a project, and use their place as a bridge between the client and the developers to ensure that value is delivered to the client. The following is a photo I took of the board used to explain the many aspects of work as a BA, I think it says more than I can:

(Sorry about the poor quality – the projector turned on for the next talk just as I was taking this picture)

User Testing

Why build something that people don’t want to use? It’s 2013, doing that is a waste of humanities potential.

Pete Chemsripong gave a great talk about the importance of user testing. Speed is important for user testing and ideally should be a daily thing. Every feature that you’re working on in a project should be validated by a user (this agrees with what Ben said in the design talk).

Helping you get your first job

The last presentation before the keynote was by Jean D’amore (@jeandamore) with an entertaining talk on how to land your first job in IT. The talk introduced job hunting as a game where you can’t control everything, but what you can control can make all the difference. He split the journey into seven steps, each accompanied by a great movie reference & quote. Hopefully the slides will be made available as I didn’t get everything down, but what follows is my summary:

Yourself (The Last Samurai)
- Improve the environment by working on yourself.
- Do: know your boundaries, be versatile, stretch yourself, maintain balance, listen, commit to projects, be willing to sacrifice, learn/do new things
- Don’t: underestimate yourself, thing you’re alone, deny the challenge, limit the horizon
The Search (An Unexpected Journey)
- The search will change you
- Do: know what you seek, choose a domain, know the employer, ask about graduate programs/training, keep your LinkedIn (and internet presence in general) up to date, favour simple job ads (i.e. not ads that list every skill under then sun – you often need to be a specialist before you can generalise)
- Don’t: dismiss too quickly, use recruitment agencies, work on projects you don’t agree with (everything you do has an impact and will impact you), waste too much time searching on-line (do spend more time talking to people)
Your CV (The Colour of Money)
- Do: PDF it!, make it unique, know what’s on it, web design it (use web design principles – e.g. readability, don’t cram everything in), make it one page, get friends to read it, start with skills over education
- Don’t: lie/exaggerate, rush it, use ‘bullshit bingo’ words (except in a few cases where it’s in the ad – in this case, always explain and give examples)
Applying (Close Encounters of the Third Kind)
- You need to understand and be understood
- Do: strike with precision, add a cover letter, apply to people (not computers), customise your CV, wait, follow-up, be organised
- Don’t: bulk apply (do apply to at most 5 organisations at once – it becomes too hard to keep track of what you’re doing otherwise)
Interviews (Gattaca)
- Little things are often the most important. For example, be good to everyone at the company.
- Do: look clean and tidy, research the company & their interview process, arrive early, know who you are and why you’re there, know what you’ll be asked, listen, ask relevant questions, be humble and curious, know your stuff (especially your résumé)
- Don’t: be afraid to ask for clarification, lower your guard until you’re clear of the building (i.e. do treat every moment you’re in/near their offices as part of the interview), ask how you did at the interview, ask about time/money when you’re after your first job, leave your phone on, have alcohol (even if offered), go if you’re sick (you need to be 100%, and the interviewers will understand)
Hearing Yes/No (The Hurricane)
- Don’t be afraid of failing and keep bouncing back
- Do: take your chance to explore/re-evaluate, take risks, trust your instinct, understand that this is one step is a career
- Don’t: burn bridges, take it personally, rush to accept, say you’re waiting on another offer
First Day (City Hall)
- Make a difference and work where you’re satisfied. Focus on employers that give you the upper parts of Maslow’s hierarchy of needs
- Do: learn names, go to social events, pair, do what you’re told, be helpful, learn the ropes step by step, leave if you don’t like it after a month of so
- Don’t: panic, try to change things quickly, cold shoulder anyone

Closing Keynote: Who are you, where are you going, and how will you get there?

The closing keynote by Chad Renando (@ChadRenando) followed on nicely from the presentation from Jean, and was really a step back, focussing on making the most of who you are, being clear about where you are going, and being intentional about how you can get there.

Every man invents a story for himself – which he often, and with great cost to himself – takes to be his life.

Chad used this quote from Max Frisch as a way of explaining that you create your own story, but you should never think that it can’t be changed or even completely rewritten.

You need to understand yourself to understand how you will work with others – are you more practical or creative? Do you approach things in an analytical way or listen to your beliefs? Are you structured or flexible? Learn what your signature strengths are and use those to create a mission statement, as your strengths are the things that make you feel alive. At the same time, look out for the strengths that you rank poorly in.

Create goals (what is your 9-day plan? your 120-day plan?) that align with your strengths, but don’t think that these goals are set in stone. Use goals as a way to direct yourself, not as measures of success. This allows you to iterate on your goals and improve them as you learn more about yourself.

Chad finished the main part of his talk with his axioms from life in a digital studio:

Ask questions
Focus on resolving process conflict, not relationship conflict
There are things that are understandable, but not acceptable – don’t get the two confused
Everyone is generally doing the best they can
There is no fault, only cause
Every conversation we have is with ourselves, and sometimes they involve others
If you want to lead others, lead yourself

The day covered a range of topics and gave me a lot to think about. One point that appeared a couple of times throughout the day (and which Chad reiterated at the end of his keynote) is how important it is to get involved with your community. It allows you to grow your network, get references and learn. I can’t agree more, I have learnt so much and met lots of interesting people by attending the YOW! conference, YOW! nights, and the BFPG & Brisbane GPU Users meetups (there are tons of similar user groups in Brisbane – I guarantee you won’t regret attending one of their meetups!). Another way to get involved in the broader community while sharpening your skills is volunteering – there are plenty of non-profits that could use help with IT. Or, if you have an idea/are working on a start-up, River City Labs is building an amazing community of like-minded people.

Thanks again to ThoughtWorks for running this mini-conference – you’ve done a great job of inspiring this IT undergrad! More about ThoughtWorks LevelUp can be found on Twitter (@TW_LevelUp) and Facebook.

Kevlin Henney on code comments

2013-10-16T00:00:00+00:00

A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments.

Kevlin Henney (original tweet here).

There are some interesting responses, pointing out that comments are great for references and in-line diagrams, and sometimes necessary for code that has been restructured to improve performance (although creating difficult to read code solely for performance reasons is a separate issue).

John Carmack on type systems

2013-10-04T00:00:00+00:00

It’s just amazing how many mistakes [are made] and how bad programmers can be. Everything that is syntactically legal, that the compiler will accept, will eventually wind up in your code base. And that is why I think static typing is so valuable.

John Carmack (full talk here – note this link jumps to this quote – part one of the keynote is here).

Pretty much sums up how I feel about the static v. dynamic debate for anything other than very short-lived prototypes.

Don’t fear the Monad

2013-10-03T00:00:00+00:00

I recently watched Brian Beckman’s Don’t fear the monad talk (YouTube version here) and thought it was a really good explanation of functions, monoids & monads and why they’re a useful way to master complexity.

I recommend it to anyone who’s starting to look at functional programming. Here are some of the key points I took away from the talk:

Functions are data
- A function takes an argument(/s) and returns a value, and always returns the same result for the same inputs
- Think of a function as a table lookup – the table is the data
Functional programming is a style that can be used in almost any language
- But some languages make it easier than others
- First class functions (or at least lamdbas) are a must have
Complexity is killing software developers – we need to control it
- The best way to control complexity is to put functions into groups where things are disciplined, and where they can be composed in arbitrary ways
- Build complexity out of simplicity by composing things
A monad adds some data to a value, and the implementation of the bind operator defines how that data interacts
If you want to add a feature, you shouldn’t even have to know what you’re going to interact with
- Using monads means the interaction is already built into the structure
You don’t need to know what a monad does
- So long as it follows the rules, it can have arbitrary side effects

Another interesting part was Brian’s reasoning about why functional programming is a ‘hard sell’:

Immediately upon the foundation of computing, the world split into two camps – the bottom up camp and the top down camp.

The bottom up camp is typified by Turing machines – “let’s start with the hardware and add abstraction as we need it to get close to the mathematics, but never at the expense of performance”. That gave rise to FORTRAN, C, C++, Pascal, Java, C# … all of those languages are in that tradition.

The top down crowd says “let’s start with the mathematics and subtract abstractions to get down to the machine, sometimes at the expense of performance”. That tradition starts with lambda calculus … and only subtracts abstraction when it is convenient to do so. That’s the tradition that gave rise to ALGOL, Lisp, Smalltalk, ML, Haskell and all these languages that have reputations of being slow.

As most programmers are familiar with the bottom up way of doing things and the two approaches come from very different origins, it can be hard to convince a bottom up programmer of the benefits of the ‘top down’ approach. It can be even harder to pick up functional programming once you’ve resigned to giving it a try.

[The top-down crowd] are saying “don’t go to the hardware so early, you’ll lock yourself into irreversible design choices”. [The bottom-up crowd] are saying “don’t give me all this math, I can’t handle garbage collection, or function calls or whatever it is that’s getting in the way of my performance, I need every ounce of performance the hardware can give me. And I’m willing to write software into the wee hours of the morning to get it”.

The last sentence here is what stands out to me – as software gets more complex, it seems that bottom up programming is going to require more and more time working in the wee hours of the morning to get the desired performance and functionality.

Maybe Haskell …

2013-09-14T00:00:00+00:00

After reading Learn You a Haskell on and off over the past fortnight and watching a couple of Erik Meijer’s Functional Programming Fundamentals, I finally got around to installing the Haskell platform and firing up GHCi.

After a couple of hours of implementing the simple functions scattered throughout LYAH, I was up to implementing quick sort. Given the number of times I’ve seen this example, I’m not going to reproduce it here, but it’s pretty amazing ¹.

One of the main things I noticed was that after two hours of writing code in a language I’d never used, I’d produced zero run time errors. There were plenty of compile errors, but that’s to be expected – especially given I was using an editor that was inserting tabs in a language with significant white space (which, by the way, makes for fun times when unfamiliar with Haskell compiler errors). I know what kind of error I would prefer to deal with.

So far I’m liking Haskell, and I think it’s a huge shame that there is only one functional programming course at my university (COMP1024), which isn’t available to most dual degree students.

Edit: COMP1024 hasn’t been offered since 2010.

1. I know this isn’t ‘true’, in-place, high performance quick sort, but in how many other languages can you write a readable, generic sorting algorithm in 6 lines of code?

VirtualBox + Dell laptop VT-x issues

2013-08-15T00:00:00+00:00

I recently had to use a VM for a uni project, and needed to have access on my laptop & desktop. I haven’t used a VM on my laptop in a while, but didn’t expect to see the following error message from VirtualBox (4.2.16):

VT-x features locked or unavailable in MSR.

I’ve used x64 hosts before with no problems, but I guess my BIOS settings were reverted for some reason (using a Dell laptop with Intel PM45 chipset).

So thinking to myself that this should be an easy fix, I rebooted, enabled VT-x and restarted, but I received the same error message. No amount of changing BIOS/VirtualBox settings, rebooting or otherwise reconfiguring would make this message go away.

I eventually read that it’s sometimes necessary to power down and physically disconnect the machine from power to properly enable VT-x. This solved my problem – I decided to make a note of it here to hopefully help someone else in the same situation.

Ubuntu 12.10 – Transmission fails to connect to HTTPS trackers

2013-01-10T00:00:00+00:00

Transmission, the BitTorrent client installed by default in Ubuntu seems to have an issue with connecting to trackers using HTTPS at the moment.

After Transmission failed to connect to a tracker using HTTPS, I went searching and found this bug report.

The steps there are pretty simple and worked well for me:

Close Transmission
Download the .deb for updated libgnutls26 from here (bottom of page – choose your architecture then mirror)
Install using
```
 dpkg -i (package name).deb
```
Restart Transmission

UPDATE: It looks like this is now fixed, so running

sudo apt-get update
sudo apt-get upgrade

should get it working again for you.