BFPG talk - Functional Programming in Scala Chapters 7 & 8

In May, I presented a talk based on Chapters 7 & 8 of Functional Programming in Scala at the Brisbane Functional Programming Group.

The talk explores the design and implementation of two functional libraries, one for parallelism and one for property based testing.

The video of the talk (also embedded below) is now available online at the BFGP talks site, along with a link to the slides.

See http://talks.bfpg.org/past.html for other past BFPG presentations.

-Xlint:unchecked and -Werror for Java projects with Gradle

I was recently working in a codebase where there were chunks of Java code that used generic interfaces from the Apache Commons Collections project as well as java.util Collections in a type unsafe way, even though the interfaces/classes that were being used supported generics.

Our gradle build was emitting the following warning:

:compileJavaNote: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

This warning should be an error in my books. It’s possible to get this behaviour by adding the following to your build.gradle file:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:unchecked" << "-Werror"
}

This way, when our code is built everyone is aware when they’re using unsafe operations, and can either fix the issue or supress the warning with @SupressWarnings("unchecked") if it’s intentional.

Johannesburg & the ThoughtWorks Pan Africa Away Day 2016

I’m back in Australia after a quick trip to South Africa to attend the ThoughtWorks Pan Africa Away Day 2016.

I left on Tuesday morning, and was early to the airport in Brisbane so I got on an earlier flight to Sydney before flying to Johannesburg. The flight from SYD-JNB was good, it was the first time I’ve flown with Qantas internationally and I was pretty impressed. I was following the sun so by the time it got dark here I’d been in daylight for about 20 hrs which was a very long day!

ThoughtWorks SA had organised a pickup from the airport and the hotel, so a driver picked me up from the airport. On the way to the hotel he told me about how different areas had different races living in them during apartheid. Some nicer looking houses have very serious security, but apparently home invasions are becoming less of a problem. There were still some questionable characters in parts though so I’m sure home invasions still happen, it’s interesting how much the crime rate varies depending on who you talk to.

I spent Wednesday in the ThoughtWorks office. It’s a nice office and much bigger than Brisbane despite having a similar number of people working out of it. I met some of the locals and another traveller here for the away day from Brazil. All up there were 32 international visitors, some there doing talks, some attending like I was and a lot from ThouthWorks leadership/management. The guests were from Brazil, China, Ecuador, Germany, India, Kenya, Singapore, Turkey, UK and the US.

On Thursday we did a tour of Soweto, an area in Johannesburg. First we walked through some of Motsoaledi, an ‘informal settlement’ (a.k.a. shantytown/favela). There are around 8,000 people living there - you should be able to pick it out on Google Maps. One thing I found pretty crazy in all these areas is how much Coca Cola sponsors things - most people there don’t have running water, but there are Coke signs everywhere!

From there we went to Vilakazi Street, which is the only street in the world to have two Nobel Peace Prize winners living there - Nelson Mandela and Desmond Tutu. Mandela’s house is a typical government built house from that era, a very basic four bedroom place. It’s now a (small) museum:

After reading some of what he and his family put up with, it’s pretty amazing he was as forgiving as he was when was released from jail!

We also went to the Hector Pieterson Museum which is all about the anti-Afrikaans Soweto Uprising - in 1976 some politician decided that all schools (where students were already struggling with English) would now run lessons in Afrikaans. A whole load of students refused to go to school, and lead a march and the police reacted pretty brutally - 13-year-old Hector Pieterson was one of the first to be shot dead.

On Friday we left to Khaya iBhubesi, the venue for the Away Day. Despite the name translating to ‘home of the lion’ there wasn’t a lot of wildlife around, though there were some lions there in an enclosure.

Friday and Saturday were mostly talks from ThoughtWorks Global and Africa leadership. There were events on Friday and Saturday night too, so I didn’t get a whole load of sleep (though I think that made jetlag easier to handle when I got back; I slept the whole flight back and then the whole night on Monday too!).

On Sunday we drove back to Johannesburg and then I went with two people from the Ecuador office to the Apartheid Museum. It’s crazy how arbitrary the laws were in that time!

After the museum I was off to the airport.

It was a really good trip even if it was a bit quick. It would have been good if I had more notice and could plan to visit other countries in Africa, but I guess that will have to wait until next time!

Pedantic console for JavaScript tests

When running JavaScript tests that fire up a browser, it can be useful to ensure that console.warn and console.error aren’t being called, and to have your automated build fail if this starts happening.

If you’re using a framework like Jasmine, you can place the following file in your helpers directory to ensure that an exception will be thrown whenever a call is made to console.warn or console.error:

console.error = (function () {

  var originalConsole = console.error;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.error called, failing test');
  }

  return myError;
})();


console.warn = (function () {

  var originalConsole = console.warn;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.warn called, failing test');
  }

  return myError;
})();

I recently used this to locate all the places that a deprecated Moment.js feature was being used in a codebase.

Haskell stack, Yesod and Docker

The why

Over the break I’ve been working on a web app to replace a fairly old MS Access Database that I built for my Dad to use in 2009 (he has a mobile vet business).

This seemed like a good chance to try out Yesod, a web framework for Haskell. The Yesod philosophy is to leverage the Haskell type system wherever possible. For example, in the Hamlet templating language everything from generating URLs to including static files and generating forms is checked at compile time.

stack is a cross platform build tool for developing Haskell programs which (when combined with Stackage makes Haskell development much more enjoyable.

I’d recommend stack for any new Haskell project, and hopefully this post can point someone else in the right direction for a stack/Yesod/Docker project.

I built a very basic site with Yesod and stack locally on OS X, based off the yesod-postgres template (stack new mysite yesod-postgres). The yesod dev server makes this all very easy, just running stack exec -- yesod devel will keep Yesod running and recompiling the app whenever any changes are made.

Soon it was time to make the app available somewhere in order to start getting some feedback. At this stage, the most basic requirement for the hosting was that it be low cost. Digital Ocean came to mind (use this link for $10 credit), and I’m now using their one-click Docker droplet at USD $5 per month. This host can run both the Yesod frontend and the PostgreSQL database.

Binaries built locally on OS X won’t run on a Linux-based Digital Ocean droplet, and it was a bit early to spend $$ on something like Snap or Travis, so I needed a VM or container to run the build in.

The how

Set up

stack has support for Docker to run the build and package the application, but it isn’t currently supported when using boot2docker (see these issues), so I used a simple Vagrantfile to start up a beefy VM:

Vagrant.configure(2) do |config|

  config.vm.box = "puppetlabs/centos-6.6-64-nocm"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4096"
    vb.cpus = 8
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
  end

end

Centos 6.6 might seem like an odd choice for a development environment, but the Vagrant box was already on my laptop, so using it saved me the download time (bandwidth is a precious commodity in semi-rural Australia).

I didn’t bother setting up provisioning properly, but here are the commands that are needed to get stack working:

# Add FP Complete repo & install stack
curl -sSL https://s3.amazonaws.com/download.fpcomplete.com/centos/6/fpco.repo | sudo tee /etc/yum.repos.d/fpco.repo
sudo yum -y install stack

# Add EPEL repo and install Docker
sudo rpm -iUvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo yum -y install docker-io

# stack requires the Docker client to be able to connect
# to the daemon without root, so add vagrant user to dockerroot group
sudo gpasswd -a ${USER} dockerroot

# At this point, you'll need to log out and back in for
# the group change to take effect

# Setup iptables to allow Docker links to work
sudo iptables -t filter -A DOCKER -d 172.17.0.0/16 -i docker0 -j ACCEPT

Adding the vagrant user to dockerroot comes with all the usual security issues, but I’m not too worried about that here. There are some other suggestions in the stack documentation for how to handle this.

Running the build

So, stack and Docker are now both installed. Enabling stack’s Docker integration on the VM is as simple as modifying the ~/.stack/config file:

[vagrant@localhost ~]$ cat ~/.stack/config.yaml
docker:
  enable: true

This gives the advantage of using Docker to run the builds on the VM, while not impacting the yesod devel workflow on OS X.

By default stack will use the fpco/stack-build repository to obtain an image to run the build in. This suits me, as there is also fpco/stack-run for running the binaries once they’re compiled. The build image includes everything that’s necessary for building the whole of Stackage, whereas stack-run is more trimmed down.

Running stack build kicked off the build in the Docker container, which completed successfully as all required dependencies (e.g. libpq and pg_config) were already in the image. Thanks to FP Complete for setting up these Docker containers for everyone to use!

Packaging Docker image

stack’s support for creating container images is a relatively recent addition, and currently isn’t covered well in the documentation. There is a reference to the image configuration section here, which was enough for my needs. After added the relevant bits, my stack.yml looks something like:

resolver: lts-3.20

packages:
- '.'

extra-deps: []

flags:
  mysite:
    library-only: false
    dev: false

extra-package-dbs: []

# This isn't required as it's set in the user config on the VM
# docker:
#     enable: true

image:
  container:
    name: mdjnewman/mysite
    base: fpco/stack-run
    add:
      config: /app/config
      static: /app/static

With that in place, running stack image container produced a Docker container ready to be pushed to Docker Hub:

[vagrant@localhost vagrant]$ stack image container
Sending build context to Docker daemon 34.25 MB
Sending build context to Docker daemon
Step 0 : FROM fpco/stack-run
 ---> db9b2a858ef5
Step 1 : ADD ./ /
 ---> e682c572d7ed
Removing intermediate container 1ce35ed1ccea
Successfully built e682c572d7ed

Running a regular docker push published the image to the registry.

Deploying to Digital Ocean

After all these steps, the final package was in Docker Hub and it was just a matter of running the following commands on the droplet, and the site was live:

docker run --name=postgres -e POSTGRES_PASSWORD=$(uuidgen) -e POSTGRES_USER=postgres -d postgres:9.3

# Yes, I know UUIDs don't make the best passwords :)

docker run              \
    -d                  \
    -w /app             \
    --link postgres:postgres         \
    -p 3000:3000        \
    mdjnewman/mysite    \
    /bin/bash -c 'PGHOST=$POSTGRES_PORT_5432_TCP_ADDR PGPASS=$POSTGRES_ENV_POSTGRES_PASSWORD /usr/local/bin/mysite'

The last command is running the Yesod server, with environment variables set to the values provided by Docker (see here for info about setting configuration variables in Yesod).

The results

I’m very happy with the Yesod workflow, and stack’s Docker integration makes deploying a lot easier as I don’t have to worry about what packages are in my build and test environments. Seems like a good compromise between manually copying files around and having a full CD pipeline.

Using stack build with a LTS resolver and an isolated container is also the holy grail of repeatable builds!

There is still a lot that would need to be done (improving security, backups, logging etc) before this was closer to a production environment, but it’s good enough as a development region.

This whole process (including writing this post) from having something that I wanted to deploy to being able to view the live site took less than a day, and I was learning a lot along the way. Deciding where was the best place to host the site and waiting for Docker to pull the fpco/stack-build image took up most of the time!