BFPG talk - Functional Programming in Scala Chapters 7 & 8

In May, I presented a talk based on Chapters 7 & 8 of Functional Programming in Scala at the Brisbane Functional Programming Group.

The talk explores the design and implementation of two functional libraries, one for parallelism and one for property based testing.

The video of the talk (also embedded below) is now available online at the BFGP talks site, along with a link to the slides.

See http://talks.bfpg.org/past.html for other past BFPG presentations.

-Xlint:unchecked and -Werror for Java projects with Gradle

I was recently working in a codebase where there were chunks of Java code that used generic interfaces from the Apache Commons Collections project as well as java.util Collections in a type unsafe way, even though the interfaces/classes that were being used supported generics.

Our gradle build was emitting the following warning:

:compileJavaNote: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

This warning should be an error in my books. It’s possible to get this behaviour by adding the following to your build.gradle file:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:unchecked" << "-Werror"
}

This way, when our code is built everyone is aware when they’re using unsafe operations, and can either fix the issue or supress the warning with @SupressWarnings("unchecked") if it’s intentional.

Pedantic console for JavaScript tests

When running JavaScript tests that fire up a browser, it can be useful to ensure that console.warn and console.error aren’t being called, and to have your automated build fail if this starts happening.

If you’re using a framework like Jasmine, you can place the following file in your helpers directory to ensure that an exception will be thrown whenever a call is made to console.warn or console.error:

console.error = (function () {

  var originalConsole = console.error;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.error called, failing test');
  }

  return myError;
})();


console.warn = (function () {

  var originalConsole = console.warn;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.warn called, failing test');
  }

  return myError;
})();

I recently used this to locate all the places that a deprecated Moment.js feature was being used in a codebase.

Haskell stack, Yesod and Docker

The why

Over the break I’ve been working on a web app to replace a fairly old MS Access Database that I built for my Dad to use in 2009 (he has a mobile vet business).

This seemed like a good chance to try out Yesod, a web framework for Haskell. The Yesod philosophy is to leverage the Haskell type system wherever possible. For example, in the Hamlet templating language everything from generating URLs to including static files and generating forms is checked at compile time.

stack is a cross platform build tool for developing Haskell programs which (when combined with Stackage makes Haskell development much more enjoyable.

I’d recommend stack for any new Haskell project, and hopefully this post can point someone else in the right direction for a stack/Yesod/Docker project.

I built a very basic site with Yesod and stack locally on OS X, based off the yesod-postgres template (stack new mysite yesod-postgres). The yesod dev server makes this all very easy, just running stack exec -- yesod devel will keep Yesod running and recompiling the app whenever any changes are made.

Soon it was time to make the app available somewhere in order to start getting some feedback. At this stage, the most basic requirement for the hosting was that it be low cost. Digital Ocean came to mind (use this link for $10 credit), and I’m now using their one-click Docker droplet at USD $5 per month. This host can run both the Yesod frontend and the PostgreSQL database.

Binaries built locally on OS X won’t run on a Linux-based Digital Ocean droplet, and it was a bit early to spend $$ on something like Snap or Travis, so I needed a VM or container to run the build in.

The how

Set up

stack has support for Docker to run the build and package the application, but it isn’t currently supported when using boot2docker (see these issues), so I used a simple Vagrantfile to start up a beefy VM:

Vagrant.configure(2) do |config|

  config.vm.box = "puppetlabs/centos-6.6-64-nocm"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4096"
    vb.cpus = 8
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
  end

end

Centos 6.6 might seem like an odd choice for a development environment, but the Vagrant box was already on my laptop, so using it saved me the download time (bandwidth is a precious commodity in semi-rural Australia).

I didn’t bother setting up provisioning properly, but here are the commands that are needed to get stack working:

# Add FP Complete repo & install stack
curl -sSL https://s3.amazonaws.com/download.fpcomplete.com/centos/6/fpco.repo | sudo tee /etc/yum.repos.d/fpco.repo
sudo yum -y install stack

# Add EPEL repo and install Docker
sudo rpm -iUvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo yum -y install docker-io

# stack requires the Docker client to be able to connect
# to the daemon without root, so add vagrant user to dockerroot group
sudo gpasswd -a ${USER} dockerroot

# At this point, you'll need to log out and back in for
# the group change to take effect

# Setup iptables to allow Docker links to work
sudo iptables -t filter -A DOCKER -d 172.17.0.0/16 -i docker0 -j ACCEPT

Adding the vagrant user to dockerroot comes with all the usual security issues, but I’m not too worried about that here. There are some other suggestions in the stack documentation for how to handle this.

Running the build

So, stack and Docker are now both installed. Enabling stack’s Docker integration on the VM is as simple as modifying the ~/.stack/config file:

[vagrant@localhost ~]$ cat ~/.stack/config.yaml
docker:
  enable: true

This gives the advantage of using Docker to run the builds on the VM, while not impacting the yesod devel workflow on OS X.

By default stack will use the fpco/stack-build repository to obtain an image to run the build in. This suits me, as there is also fpco/stack-run for running the binaries once they’re compiled. The build image includes everything that’s necessary for building the whole of Stackage, whereas stack-run is more trimmed down.

Running stack build kicked off the build in the Docker container, which completed successfully as all required dependencies (e.g. libpq and pg_config) were already in the image. Thanks to FP Complete for setting up these Docker containers for everyone to use!

Packaging Docker image

stack’s support for creating container images is a relatively recent addition, and currently isn’t covered well in the documentation. There is a reference to the image configuration section here, which was enough for my needs. After added the relevant bits, my stack.yml looks something like:

resolver: lts-3.20

packages:
- '.'

extra-deps: []

flags:
  mysite:
    library-only: false
    dev: false

extra-package-dbs: []

# This isn't required as it's set in the user config on the VM
# docker:
#     enable: true

image:
  container:
    name: mdjnewman/mysite
    base: fpco/stack-run
    add:
      config: /app/config
      static: /app/static

With that in place, running stack image container produced a Docker container ready to be pushed to Docker Hub:

[vagrant@localhost vagrant]$ stack image container
Sending build context to Docker daemon 34.25 MB
Sending build context to Docker daemon
Step 0 : FROM fpco/stack-run
 ---> db9b2a858ef5
Step 1 : ADD ./ /
 ---> e682c572d7ed
Removing intermediate container 1ce35ed1ccea
Successfully built e682c572d7ed

Running a regular docker push published the image to the registry.

Deploying to Digital Ocean

After all these steps, the final package was in Docker Hub and it was just a matter of running the following commands on the droplet, and the site was live:

docker run --name=postgres -e POSTGRES_PASSWORD=$(uuidgen) -e POSTGRES_USER=postgres -d postgres:9.3

# Yes, I know UUIDs don't make the best passwords :)

docker run              \
    -d                  \
    -w /app             \
    --link postgres:postgres         \
    -p 3000:3000        \
    mdjnewman/mysite    \
    /bin/bash -c 'PGHOST=$POSTGRES_PORT_5432_TCP_ADDR PGPASS=$POSTGRES_ENV_POSTGRES_PASSWORD /usr/local/bin/mysite'

The last command is running the Yesod server, with environment variables set to the values provided by Docker (see here for info about setting configuration variables in Yesod).

The results

I’m very happy with the Yesod workflow, and stack’s Docker integration makes deploying a lot easier as I don’t have to worry about what packages are in my build and test environments. Seems like a good compromise between manually copying files around and having a full CD pipeline.

Using stack build with a LTS resolver and an isolated container is also the holy grail of repeatable builds!

There is still a lot that would need to be done (improving security, backups, logging etc) before this was closer to a production environment, but it’s good enough as a development region.

This whole process (including writing this post) from having something that I wanted to deploy to being able to view the live site took less than a day, and I was learning a lot along the way. Deciding where was the best place to host the site and waiting for Docker to pull the fpco/stack-build image took up most of the time!

Simon Peyton Jones on maintenance with strong types

People don’t like fixing type errors, but they are a lot easier to fix than runtime errors… The single biggest thing going for static typing is not so much that it helps you write your program in the first place, but it helps you maintain your program.

GHC itself is an example – it’s now 150,000 lines of Haskell & twenty years old and I regularly refactor it in pretty large scale ways. I just wouldn’t dare do that if I didn’t have static typing to keep me sane. It’s this enormous code base, chunks of which I’ve forgotten about, and yet I can confidently make systemic changes to it because I know the type checker is going to catch all the places that change needs to go.

Simon Peyton Jones, speaking on Functional Geekery Episode 11.

The part about GHC is anecdotal evidence, to be sure, but it makes the point nicely.