Matt Newman

Jan/Feb/Mar 2019 travel

2019-04-30T00:00:00+00:00

After the stay in California, I didn’t do a load of travel in the first three months of 2019, mostly due to being staffed in Chicago & spending some time there. Also, there isn’t a lot of easily accessible hiking in winter!

This post covers what I was up to for the first three months of the year, and is mostly about Chicago.

Chicago

Early weather

Chicago in winter is a level of cold that I’m not well acquainted with 🥶

I drove up from Cincinnati, and was blessed with a snow storm the night before my drive:

I guess it was a sign of things to come - a fortnight later was the 2019 polar vortex, which shut down most of the city for a couple of days. It was the only time I saw the river completely frozen over:

I was staying a very short distance from work and decided to walk during the cold weather, which was an experience. My face was completely covered, except for my eyes, which watered. Of course the water from my struggling eyes froze, along with the tiny bit of hair that was exposed and water condensing in my nose (producing a satisfying crunch when I pinched my nostrils).

Field Museum

I went full tourist in Chicago and purchased a CityPASS, good for entry into 5 attractions. The first I visited was the Field Museum. For some reason I don’t have loads of photos here, but the main things I remember are

the huge collection of taxidermied animals
SUE the T. rex
the Underground Adventure

Art Institute of Chicago

I’m not normally one for art museums, but The Art Institute has a great collection, including a number of paintings by van Gogh, Picasso, Pollock, Warhol and more. The Deering Family Galleries of Medieval and Renaissance Art, Arms, and Armor (an ongoing exhibit) was another highlight, and I was amazed at the detail in the 68 Thorne Miniature Rooms.

Skydeck Chicago

I’ve been told by locals that under no circumstances am I to refer to the Sears Tower by it’s new name, The Willis Tower. Regardless of the name, the views from the The Skydeck are spectacular:

Museum of Science and Industry

I think the Museum of Science and Industry in Chicago is my new favourite museum, with the Deutsches Museum in Munich a close second.

It’s absolutely huge, I could easily spend far more than a day there. A few highlights, in no particular order:

Avalanche simulation in the Science Storms exhibit.

Mirror maze in Numbers in Nature exhibit.

The mirror maze was oddly peaceful for the few seconds I was in there alone! For the rest of the time I was just trying to avoid running into the kids that were barrelling though.

Tornado simulation in the Science Storms exhibit.

U-505 submarine, captured during World War II.

St Patrick’s Day

St Patrick’s Day in Chicago was an experience … I knew it was a huge event, but I didn’t expect the streets to be packed from 7:30 AM.

The Chicago River is dyed a vibrant green on St Patrick's day.

… Spring?

I’ve been warned that March warm weather in Chicago only lulls you into a false sense of security. As I’m writing this it’s closer to the end of April & I’m hopeful the worst of the cold weather is finished. As a reference, here’s a recent photo of Maggie Daley Park & Grant Park - it looks a little more inviting than the photo earlier in this post!

LA & Spartan Race

I visited LA briefly in December on a trip to go hiking in Joshua Tree National Park, but most of the time in LA itself was spent sitting in traffic.

I went back in January to do the Spartan SoCal Sprint just outside of LA, and had a chance to see a little more of LA itself.

Griffith Observatory is definitely worth a visit, for the exhibits as well as the views of LA.

On Saturday we spent some time in Venice Beach area, including the Venice Canals and a ride along the boardwalk and ocean front walk. It was a nice change from the winter in Chicago!

Seattle

I had planned a trip to Chicago the weekend after the polar vortex hit Chicago, but my flight out was cancelled due to the weather. I still made it out of Chicago the next day, but that made for a fairly short trip, so I changed my plans to spend the time in Seattle itself and ditched the hiking I was planning in Mt Rainier National Park.

I stayed right near Pike Place Market which turned out to be a great spot.

Some notable places:

Piroshky Piroshky bakery: I may have returned here more than once, fodder was delicious but not entirely compatible with my desire to be somewhat healthy 🤣
The Gum Wall: one of the most germ ridden attractions in existence

The original Starbucks
Underground tour: it was interesting hearing about how Seattle was built, it is honestly a miracle they didn’t just give up. Wikipedia has some info on the underground.
Seattle Center: I’d like to go back here, there is loads to do. It was the site of the 1962 World’s Fair. The Space Needle, MoPOP and the Chihuly Garden and Glass are apparently all worth a visit, but I just walked around the area without going in.

South Lake Tahoe

My first trip in March was to South Lake Tahoe for a weekend of skiing. There was a little more snow there than I’ve seen in Australia before 😂.

The base of the resort is at 1,907 m and the top is 3,068 m, so it isn’t super high but its location near the lake means lots of snow (the peak of Mount Kosciuszko is 2,228 m, so most of the Thredbo resort in Australia is less than 2,000 m).

Toronto

The second trip for March was to Toronto, again a short trip so I really only had time to explore part of Old Toronto near where I was staying:

St. Lawrence Market
Distillery District
Yonge-Dundas Square
CF Toronto Eaton Centre
Kensington Market

California

2019-01-01T00:00:00+00:00

I’ve been fortunate to spend some time in The Golden State in November and December. I haven’t made it over here much, as it’s a long way from where I have been based in the east of the states.

Part 1 - San Francisco

After working most of Friday from the ThoughtWorks office in the SoMa (South of Market) area, we took a walk through Chinatown to Coit Tower on Telegraph Hill, where I got my first views of the bay.

Despite being very hilly, SF is a walkable city. We continued down to Fisherman’s Wharf and on to the Palace of Fine Arts in the Marine District. I didn’t get many photos, but it’s an interesting place originally constructed for the 1915 Panama-Pacific Exposition.

We rented bikes on Saturday, and rode from North Beach across the Golden Gate Bridge, back to Sutro Heights and through the Presidio on the way back. Lots of great views, the photos below don’t really do it justice!

I think this might have been the first time I’ve seen the sun setting over an ocean … I’ve always lived near eastern coasts so sunrise over the ocean feels more normal! 😂

Part 2 - Los Angeles and Joshua Tree National Park

In early December I was in LA for a couple of days before flying back to Australia for the Christmas break. We did a bit of hiking, and saw a little of LA as well.

The first hike was the Mount Zion Loop - Sturtevant to Winter Creek Trail

For the second day we drove to Joshua Tree National Park and did the Lost Horse Mine Loop Trail.

I managed to sneak in a visit to The Getty:

And to Griffith Observatory to get the obligatory selfie with the Hollywood sign before flying out on Sunday:

Part 3 - Drive from LA → SF

After landing in LA on Boxing Day morning, I took a couple of days to drive up to San Francisco, mostly along California State Route 1 (a.k.a. the Pacific Coast Highway). It’s a lovely drive, a few highlights include:

Beautiful beaches along the way:

Elephant seals:

The Buzzards Roost trail in Big Sur State Park:

Carmel-by-the-Sea:

And finally Monterey:

Part 4 - San Francisco (again)

I spent the last four days of 2018 running, walking and hiking my way primarily through SF, in an effort to work of combined Thanksgiving/holiday party/Christmas calories 😂

The only visits outside SF were to Sausalito and a hike in the Claremont Canyon Regional Preserve in Berkeley. Some select photos:

Pawleys Island

2018-11-26T00:00:00+00:00

For Thanksgiving this year I stayed with some friends at a house in Pawleys Island, a town and island in Georgetown County, South Carolina.

It was a relaxing time, with lots of walks on the beach and way too much food.

We were also there for a king tide, which produced some minor flooring and had us worried for out rental car. It was amazing to me how many of these places are so close to the ocean, you could see the dunes getting washed away during the higher tides.

Acadia National Park

2018-10-22T00:00:00+00:00

My fourth trip in what has turned into a very busy October was to Acadia National Park in Maine, the easternmost state of the US. I flew into Boston, drove up to Acadia & stayed in Airbnbs nearby.

Acadia National Park is beautiful, and is the first coastal park I’ve visited (other than the Everglades NP, it’s nice in a different way).

The first hike I did there was up Cadillac Mountain along the South Ridge Trail. Cadillac Mountain is the highest point along the North Atlantic seaboard (though it’s still not very high at 466 meters).

The weather wasn’t great to start, but the views were still worth the effort.

Some pretty strong winds had the clouds blown away in no time!

The last part of the loop went down part of the A. Murray Young Path which had amazing fall colours on display.

The second hike for the day was The Beehive Loop Trail, plus a little of the Champlain North Ridge Trail.

The clouds had been well and truly blown away by this point!

Sunset at the The Bass Harbor Head Lighthouse was a great finish to the day.

Salt Lake City

2018-10-15T00:00:00+00:00

Last weekend I was able to do some hiking near Salt Lake City in Utah. There were some stunning views of Great Salt Lake from the flight. The Lucin Cutoff causes differences in salinity in three parts of the lake. The corresponding difference in algae growth is visible in the photo below.

The first hike we did on Friday was the Mount Olympus trail:

It was a long, steep climb but the views from the top were worth it!

The group I was with continued on to Lake Tahoe on Saturday, but I stayed in the area to do a couple more hikes. First up was the Lake Blanche Trail:

After Lake Blanche was Bells Canyon Trail:

This trail started out looking very different to the snow that was at Lake Blanche, but there was lots of snow on the trail by the top.

I’m happy anywhere aspen leaves are shaking in the wind!

Alaska

2018-10-02T00:00:00+00:00

I flew into Anchorage after work on Thursday night, landing at a reasonable time thanks to Alaska being four hours behind Eastern Time. We stayed in Anchorage on Thursday, did some work from there in the morning and drove up towards Denali National Park on Friday afternoon.

Before leaving Alaska, we stopped at an outdoor store to buy some bear deterrent spray. This National Park Service video convinced us, though it was unlikely we’d run into a grizzly on the hikes we had planned. Neither of us fancied running into a male bear given they can tip the scales at 680 kg and stand 3 m tall.

The drive was nice, we were lucky with the weather for the whole weekend. We did a couple of short hikes on the way, at Thunderbird Falls and Little Coal Creek before arriving at our accomodation for Friday and Saturday nights. On Friday I saw a little of the northern lights; our host woke us up as they were out. When he woke me up I raced outside in my pajamas with a jacket on top which was inadequate given the weather! In hindsight I wish I’d gotten dressed and stayed out there longer. It was a bit bright where I was standing so didn’t get any good photos.

We were staying about 35 minutes south of the park entrance, so we still had a short way to go on Saturday morning. We ended up driving further north in search of coffee, and ended up at the nearest ‘large town’ of Healy (population around 1,000) where we found somewhere that was still open. We drove through a few small communities that were all boarded up for the winter which was bizarre to see. A lot of the businesses shut down completely and the people working there move south to somewhere warmer for the winter!

The drive along the eastern edge of Denali National Park featured some breathtaking views of the mountains. Denali Mountain was almost always visible in the distance. It’s the highest mountain peak in North America, with a summit of 6,190 m (for reference, the highest mountain in Australian peaks at 2,228 m). The upper half is permanently snowy.

The first hike was the Savage Alpine Trail, the first photo I took on the hike was of a ‘bears frequent this area’ warning sign so we were happy to have the spray. It was super windy in some parts, here’s a video which tries to capture it (turn your volume down before opening it). After the hike we drove into the park until reaching the point where the road is closed for the winter at the Teklanika River.

On the way out we got our wish of seeing a bear from a safe distance, I’m glad we got to see one in the wild. It’s a fair distance in this video, but it gives you some idea of the size. A group of people assembled to watch and as you can see about half way through the video it must have got a bit spooked! We didn’t do loads more on Saturday, but did head out again late at night to try and see the northern lights.

Sunday was a late start, but we did a full on hike at Bison Gulch (a ‘gulch’ is the name for a valley formed by erosion). I recorded the hike on Strava and plotted it on Google Maps:

Photos from the hike start here in the album. It was pretty brutal, we climbed 1,259 m and a lot of that towards the end was over very loose shale. Here’s one of us up near the top:

It’s lucky we went when we did; a couple of weeks later and it would have been rather chilly! I think the snow there is from some early falls a week before we arrived. We were ridiculously fortunate with the weather. It was relatively warm, with clear skies. The whole place seemed so remote, the air was super fresh and the stars were all out. Other than the bear, we also saw a couple of moose and a pair of mountain goats.

All the photos from the trip are in an album on Google Photos (some of the links above should jump to certain spots in the album).

Philadelphia

2018-09-23T00:00:00+00:00

This weekend I visited Philadelphia, Pennsylvania’s largest city. It has plenty of historic sites from the American Revolution, as well as a fairly famous prison and the Philadelphia Museum of Art which is potentially better known for “Rocky” than the art.

I didn’t have big plans (I was looking for a relatively quiet weekend after the week tripping around CO/UT), but still had a busy Saturday. The day started with a train to the Wissahickon Valley Park. From there I did some hiking/walking through Wissahickon Valley and on to Boathouse Row, along the Schuylkill River. That leg (recorded on Strava) came to 18.5 km, and the walk downtown came to roughly another 12 km, so I definitely got my steps in for the day! Here’s the route I took through the city centre:

That took me past Boathouse Row, Philadelphia Museum of Art, Eastern State Penitentiary, Washington Monument Fountain, Rodin Museum, Parkway Central Library, Logan Square, Love Park, Reading Terminal Market, Sonny’s Famous Steaks for a Philly cheesesteak, and a load of spots in Independence National Historical Park.

The walk from Wissahickon Valley also included lots of historic buildings and bridges. A few highlights of the day follow.

Statue of Tedyuscung

At a point along the trail in Wissahickon Valley, there is a statue positioned fairly high up the valley, showing a Native American looking out over the valley. It’s a fairly huge statue, and I didn’t expect to come across it, but I’ve done some reading about it since. The history of the sculpture is here (the marble version I saw was dedicated in 1902), though apparently it’s a bit contentions with one local calling it “a monument to ignorance”.

Eastern State Penitentiary

Eastern State Penitentiary was the first major prison based on the principle of keeping prisoners in solitary confinement. The idea was the inmates would practice penance through silent reflection upon their crimes and behaviour. The whole prison is designed around this idea. Inmates never saw anyone for the entire duration of their sentence. The walls are solid and at least a foot thick. Each prisoner has their own exercise yard. The guards even wore socks over their shoes so the silence wasn’t disturbed. This understandably went fairly terribly, with lots of inmates mental health taking a hit. The prison operated using solitary confinement from 1829 until 1913, and it closed in 1971.

The city council intended to redevelop it, but the site was abandoned for around a decade and fell to ruin. It was ‘stabilised’ before opening to the public as a museum in 1994, but large parts of the prison are as they were after the period of abandonment:

There was also an exhibit at the end of the tour about prisons today and mass incarceration, which was pretty sobering.

More photos from the day are in an album on Google Photos.

Colorado & Utah hiking

2018-09-16T00:00:00+00:00

I attended Explore DDD in Denver again this year, after enjoying the conference and Denver itself so much last year.

This year I took a couple of days off work before the conference, and planned a road trip to do some hiking in the area. I made it to one of the parks near Boulder, five national parks in the area, and a quick hike near Colorado Springs on the way back to Denver.

It was a fun week of hiking (89 km) and driving (2,261 km), with a little planning required:

AllTrails worked really well for organising hikes. I paid for a lifetime Pro membership, which meant I could download trails for offline use (very handy in the more remote parks)
Airbnb rooms near the hiking spots meant I could get up early and beat the crowds & heat
An America the Beautiful Pass covered the entrance fees at all the national parks
A 3L Osprey Raptor 14 pack kept me hydrated
KEEN Targhee III Low WP hiking shoes ensured that my feet stayed dry and had decent support

Boulder Open Space

First up was Chautauqua Park in Boulder, CO. A couple of colleagues based in Denver/Boulder recommended hiking here, and I’m glad I listened.

The route below is a combination of the Royal Arch Trail, First and Second Flatiron Loop and the Green Mountain Loop.

The hike had amazing views of the Flatirons & Royal Arch:

Rocky Mountain National Park

I visited the Rocky Mountains National Park last year, but as you can see from the earlier post linked above, the weather was a little different this time.

I was lucky enough to drive past a fairly huge elk chilling very close to the road.

Most of the hiking I did was in the Bear Lake area, on the Emerald Lake and Sky Pond via Glacier Gorge trails.

The mountain lakes & aspens were spectacular:

There was a very friendly marmot up near Sky Pond, though he was probably just looking for food:

Arches National Park

Sunday saw me driving from Boulder to Arches National Park in Utah, through Grand Valley.

The Devils Garden trail was amazing, with the 88 meter Landscape Arch a highlight.

Canyonlands National Park

Next up was Canyonlands National Park. I stayed near La Sal on Sunday night, which made the Needles section of the park a better option than heading back north to Island in the Sky.

The Chesler Park Loop was one of the highlights of the whole trip. The landscape looks other-wordly at times, and it was a lightly trafficked trail.

Mesa Verde National Park

After staying in Cortez on Monday night, I visited Mesa Verde National Park, a UNESCO World Heritage Site. The park is best known for the cliff dwellings built by the Ancestral Puebloans.

I managed to get in early and score a ticket to a guided tour of Cliff Palace, a dwelling built around year 1200.

Other than the Cliff Palace tour, the only hike I did here was Petroglyph Point, which passes a large petroglyph panel. The carvings were likely made around the same time as the cliff dwellings were built.

Great Sand Dunes National Park

The final national park I visited was Great Sand Dunes. This was a rushed visit, but I wanted to make a stop in the park to see the contrast of sand dunes and mountains. In my mind dunes belong in deserts or at the coast, not at 8,200 feet elevation in the foothills of mountains.

Zion National Park and Las Vegas

2018-06-23T00:00:00+00:00

Last weekend a I took a couple of days off to see some of the western United States, with a visit to Zion National Park and Las Vegas.

Zion National Park is absolutely stunning. We did a couple of hikes, the first was to the peak of Angels Landing, a 454 m high rock formation (total elevation 1,760 m). The trail was built in 1926 and is cut into solid rock most of the way. It’s only 3.9 km to the top, but it felt much longer given the elevation and the temperature (it was about 40 °C).

The photo below was taken from somewhere near the start of the trail, Angels Landing is the peak in the foreground of the photo on the left side.

The view from the top was worth it though!

Given how hot it was on the first day, we set out a little earlier for our second hike to Observation Point. The line for the buses into the Canyon was already quite long, so we took the opportunity to walk the Pa’rus Trail to the second bus stop.

I’m glad the buses were full, as the Pa’rus Trail provided some amazing views along a fairly flat and safe path. It was nice to have a casual stroll and look around without needing to worry about veering off the track!

The walk up to Observation Point runs through Echo Canyon Passage, which was a nice change of landscape and the canyon walls also offered some protection from the sun.

Observation Point itself made the hike worthwhile, it’s a great vantage point for almost all of Zion Canyon.

We were all flying out of Las Vegas, so on Sunday afternoon we drove back to Las Vegas and explored there a bit. It was roughly what I expected - lots of bright lights and tacky touristy things 🤣. The Fountains of Bellagio were pretty impressive though!

Travel posts

2018-06-16T00:00:00+00:00

Sometime in the next couple of days I’m going to publish a bunch of back dated posts about travel I’ve done recently (or in some cases not so recently). A lot of these posts are taken straight from emails I sent to family and friends at the time I did the actual travel, I decided to post them here as email is where keystrokes go to die.

This is a bit of a departure from what has been the norm on this blog so far, and for those following via RSS I’ve introduced categories and separate feeds for the Software/Tech category and the Travel category in addition to the existing feed containing all posts. If you’d rather not hear about travel posts, please update your subscription :)

Technically, the only interesting part of this is handling the images I wanted to include in the posts. I’ve included a few images in the past, but never in a large quantity.

I use Google Photos/Drive for all my photos, and originally tried hotlinking the photos from there. If this was supported it would have been the best solution, as I wouldn’t have to duplicate the photos and it would be easy to include them (though some of them are quite large). Unfortunately (though unsurprisingly) Google Photos/Drive doesn’t support hotlinking, and any direct links to images expire after a while.

The next option was using the Google Drive ‘embed’ functionality, which involves inserting some HTML into the post which will load an IFrame. This worked, but:

it looked a little clunky on desktop and didn’t scale well on mobile
a Google IFrame means Google tracking cookies
performance was poor - loading all the images on one of the travel posts using this method resulted in a whole load of extra requests:

I ended up committing them into the Git repository that backs this static site, but that does mean the repo size is going to grow a lot more than it should. As it’s just me working on this repo though, I can always use git filter-branch as a last resort if I do want to move the images elsewhere. To avoid bloating the repo too much and to keep loading times low I did fairly aggressively reduce the file size with ImageMagick:

mogrify -sampling-factor 4:2:0 \
  -strip \
  -quality 85 \
  -interlace JPEG \
  -colorspace sRGB \
  -resize 50% \
  -auto-orient
  *

The final change was adding lazy loading using Lozad.js. Thankfully I include almost all the images on my site using a custom Jekyll include so I just had to include the library and change one file to get lazy loading across the board.

Brown County State Park & Nashville, IN

2018-05-28T00:00:00+00:00

Last week was my final week staffed in Cincinnati, so I finally got my act together, booked an Airbnb and drove across to Brown County State Park in Indiana. I’ve been meaning to go there for a while, as according to MTB Project and Singletracks the state park and surrounding areas have some of the best trails in Indiana.

I wasn’t disappointed - the state park is amazing. Miles of well maintained trails (gravel, downhill flow, technical sections/rock gardens) coupled with amazing views made the trip worthwhile.

Washington DC

2018-05-13T00:00:00+00:00

I’m at the airport about to head back to Cinci after a couple of days in the US capital.

It was a good trip, though I almost missed my flight thanks to leaving a bag at security and only realising when boarding had already started. The 1 km run to security and back while wearing boots and wondering if I was going to make it got my heart rate up!

The moment I left the airport in DC a motorcade stopped all the traffic which I guess is a fairly regular occurrence here.

On Friday night I walked about 8 km from my hotel past the Marine Corps memorial, around Arlington national cemetery (you’ve probably seen this in movies rows and rows of white tombstones), across into DC, to the Lincoln Memorial and Washington Monument, north to the White House, past the treasury and to a local pub for a nightcap. It’s crazy how close all these attractions are. While I was outside the White House a guy next to me said to his mate, not too quietly, “I think I see Donald standing at the window in his underwear, crying” at which point I burst out laughing.

The big green area between the Lincoln Memorial and the Washington monument is called the National Mall extends even further east to the US capitol building. Its surrounded by museums, monuments, memorials and other federal buildings.

On Saturday I caught the Metro and started at the Smithsonian National Air and Space Museum which someone from work recommended, it was filled with really cool stuff. There was also a fighter jet simulator there which would roll upside down and everything, it was pretty great! It was a short run but I think I’d have made myself sick if it was too much longer.

The national art gallery was on the other side of national mall, so I went there next. There was lots of … art. I appreciated a few things but I can only handle so many portraits of stuffy old white dudes. The oldest painting I saw was from 1247, and there were loads of pretty old sculptures. There was a painting by da Vinci and more recent names I know like Andy Warhol.

From there it was a short walk to the United States Capitol, which is the most impressive building around. I was able to go inside the rotunda, but the full tours were all booked out.

The Library of Congress is reachable from the Capitol via a tunnel, so I went there next - another super impressive building. Jefferson was apparently a big reader and his collection of books was there, they weren’t all dated but some were super old.

Another Metro ride home and a run closed out Saturday. Managed 9km at just over 12 km/h (with a few stops for stretching and photos) despite the temperature sitting at around 30°C. Theodore Roosevelt Island was really nice.

New York City

2018-05-09T00:00:00+00:00

I spent the last weekend in New York City, doing the tourist thing. I went with very few plans and met up with another guy from work who has spent a year in NYC. Despite a several hour delay and change of airline & airport on the flight there, it was a successful trip!

Saturday played out as a busy day! In order:

Run in Central Park

Grand Central Station

The High Line - a park that’s been built on decommissioned overhead train tacks
Oculus - a $4 billion transit station

9/11 Memorial - pretty sobering to think there were two massive buildings full of people where the monuments now stand
One World Observatory, 102 floors up

Stone Street - a street in the historic district with lots of old bars, it was absolutely packed though because of Cinco de Mayo

The Dead Rabbit Grocery and Grog, which won a few ‘worlds best bar’ awards in 2016
Wall St
Trinity Church
Staten island ferry

Times Square
Dinner at Ipanema
A Housewarming in Brooklyn, where we ended the day

Sunday was a quieter day. We spent most of our time in Dumbo (Down Under the Manhattan Bridge Overpass), an area in Brooklyn with pretty great views of the city. We went to Grimaldi’s pizza, which is one of the old, amazing pizza places in NYC. A walk across the Brooklyn Bridge closed out the weekend:

Miami, Florida Keys & Shark Valley in the Everglades

2018-04-23T00:00:00+00:00

I spent last weekend in Miami! South Beach/South Pointe Pier was the first stop, and the weather played along nicely.

The nearby Art Deco Historic District had some classic Miami colours:

I spent Friday evening wandering around Wynwood. The urban art there is amazing, the below is just a tiny sample.

On Saturday I drove the Overseas Highway from my hotel to Key West.

The drive down was beautiful, though it’s a little bizarre being on some of the really long bridges.

I had to take the obligatory selfie at the Southernmost Point of the Continental US in Key West.

There was loads more to do in Key West, but I didn’t have loads of time given I was doing the drive back to Miami the same day - I definitely have to visit again!

On Sunday I went to Shark Valley in the Everglades where I hired a bike and rode a 24 km loop. There were so many alligators!

The tower at the half way point of the loop provided a good view, and was my last stop for the day before heading to MIA and back to CVG.

Dallas, Trinity River/Bishop Arts/Downtown

2018-04-01T00:00:00+00:00

Yesterday I went for a walk through some of Dallas that I haven’t explored yet, despite technically living here for about 9 months (I say technically, as a lot of that time was spent travelling for work). I made a rough map of my walk using On the Go Map.

Following are a few notable points on the trip.

Trinity River and the Margaret Hunt Hill Bridge:

Lockhart Smokehouse - a reasonably famous BBQ place, where the ‘Texan Vegetarian’ section comprises of chicken/turkey, instead of beef 🤣:

The delicious taster flight at Bishop Cider Co:

A 30-ft eyeball in downtown Dallas, which is quite hard to miss:

And finally, this sign, which I feel captures so much of what it is like walking in a US city other than New York:

I never did spot the pavement in that area, but at least I know not to expect one to appear now that it’s ‘ended’.

The lack of people walking here continues to blow my mind. I walked an hour from the Bishop Arts District to downtown, and didn’t walk past a single person. A group of young guys went past in a car and one of them yelled “ain’t y’all heard of Uber?”, further affirming my belief that everyone thought I was crazy for walking. I guess it’s not that surprising given a couple of times the pavement I was walking on came to a major road and just stopped, leaving me to backtrack and find somewhere else to go.

Vetted - tweaking importer performance

2018-03-03T00:00:00+00:00

Since the last post, I’ve started working on an importer to load data from the existing Access database. Work to date is on GitHub.

In the current domain model, there is a single aggregate root, the Client. The importer is written as a command line application which interacts directly with the domain, assuming an empty database (I might get to incremental imports in the future). At a high level, the importer currently:

Creates the clients
Adds any existing ‘notes’ about the client
- Notes are freeform text about a client, unrelated to any particular patient or transaction
- The existing application is a little limited in what can be entered into the main form, so notes have been used to make up the slack (e.g. in the existing data, there are numerous clients which have an email address or fax number in the notes field, as there is no first class input for these values)
Adds home and mobile phone numbers
Adds the ‘most common travel distance’ as a note

These steps are visible in the implementation of the importer:

override fun run(vararg args: String?) {
    val rows = accessDb.clientTableRows

    val newClientIds: Map<String, UUID> = createClients(
        rows,
        accessDb::postCodeFor,
        accessDb::stateFor,
        commandGateway
    )

    allOf(
        addClientNotes(rows, newClientIds, commandGateway),
        addPhoneNumbers(rows, newClientIds, commandGateway),
        addMostCommonDistance(rows, newClientIds, commandGateway)
    ).get()
}

First, clients are created, producing a Map of the old client ID to the new client ID. Once all clients have been created, all the other updates are applied (potentially concurrently).

Without giving away too much information about the existing data, the order of magnitude of the existing number of clients is 3, and the total number of events generated with the current importer implementation is at most 5x the number of clients (one ClientMigratedEvent, up to two ClientNoteAddedEvents and up to two ClientPhoneNumberAddedEvents).

My first pass at the importer was taking around 80 seconds to import everything into a PostgreSQL database. I know that premature optimization is the root of all evil, and that I don’t have anything resembling a working product at the moment, but this seemed far too high. Also, it was impacting my ability to iterate quickly with ‘production’ data, which is enough of a reason to look for improvements.

After looking at the generated schema and doing some sampling with VisualVM, I decided there were three options to investigate:

Asynchronous processing of commands
Serialisation format changes
Generated schema changes

In order to compare a full run of the importer pre and post optimisations, I want to be able to toggle the optimisations on/off from the command line. The following script has the toggle properties in place, and in the sections below I will use Spring config management to read these properties.

PASSWORD=$(uuidgen)

docker stop vetted-postgres ; docker rm vetted-postgres

docker run \
    --publish 5432:5432/tcp \
    --name vetted-postgres \
    --env POSTGRES_PASSWORD=$PASSWORD \
    --detach \
    postgres

./gradlew build

java \
    -jar importer/build/libs/vetted-importer-0.0.1-SNAPSHOT.jar \
    --axon.use-async-command-bus=false \
    --axon.use-cbor-serializer=false \
    --spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL95Dialect \
    --spring.datasource.password=$PASSWORD

The script above will allow me to evaluate the impact of any changes I make in a repeatable fashion.

Option 1 - Asynchronous processing of commands

I’m using the Axon framework, which handles a lot of the plumbing of building an application based on DDD & CQRS principles. By default when using the Spring auto-configuration, a SimpleCommandBus is used which processes commands on the calling thread.

I added some configuration to use a AsynchronousCommandBus with a configurable number of threads:

@Bean
@ConditionalOnProperty(
    value = ["axon.use-async-command-bus"],
    matchIfMissing = true
)
fun bus(
    transactionManager: TransactionManager,
    @Value("\${axon.command-bus.executor.pool-size}") poolSize: Int
): CommandBus {
    val bus = AsynchronousCommandBus(
        Executors.newFixedThreadPool(poolSize)
    )

    val tmi = TransactionManagingInterceptor(transactionManager)
    bus.registerHandlerInterceptor(tmi)

    return bus
}

I initially tried this configuration out with a pool size of 10. This reduced the time for the import to around 30 seconds, which is an improvement from 80 seconds but short of an order of magnitude improvement which should be possible. This led me to believe that there was either contention somewhere else, or that some of the constant factors are just too high at the moment.

Option 2 - Serialisation format changes

By default, Axon will use XStream to serialise events, which uses an XML representation. XML is quite verbose, and the Axon documentation even suggests using a different serializer.

Overriding the serializer is thankfully quite easy:

@Primary
@Bean
@ConditionalOnProperty(
    value = ["axon.use-cbor-serializer"],
    matchIfMissing = true
)
fun serializer(): Serializer {
    val objectMapper = ObjectMapper(CBORFactory())
    objectMapper.findAndRegisterModules()
    objectMapper.setSerializationInclusion(NON_ABSENT)
    return JacksonSerializer(objectMapper)
}

I opted for using Jackson with a ‘Concise Binary Object Representation’ (CBOR) JsonFactory. This resulted in a ~70% reduction in size for the serialized payload for most events. With XML:

postgres=# select avg(length(loread(lo_open(payload::int, x'40000'::int), x'40000'::int))) from domain_event_entry;
     avg
--------------
 433.69003053

and with CBOR:

     avg
--------------
 111.54379774

This didn’t have a huge impact on the run time of the importer, but is still a worthwhile optimisation.

Option 3 - Generated schema changes

You may have noticed in the SQL statments above that the current schema is using the PostgreSQL large objects functionality. From the PostgreSQL docs:

PostgreSQL has a large object facility, which provides stream-style access to user data that is stored in a special large-object structure. Streaming access is useful when working with data values that are too large to manipulate conveniently as a whole.

If we inspect the schema that’s being generated:

postgres=# \d domain_event_entry
     Table "public.domain_event_entry"
      Column      | Type | Nullable | Default
------------------+------+----------+---------
 meta_data        | oid  |          |
 payload          | oid  | not null |
 ...

The oid type here is an object identifier - a reference to a large object which is stored externally from the table. The events we’re writing are small enough that the overhead of reading them as separate streams is hurting performance rather than helping.

At least two people have had the same issue when using Axon with PostgreSQL, as evidenced by the questions on Google Groups and StackOverflow. The suggestion to customise the PostgreSQL dialect used by Hibernate seems to work, and further reduced the runtime to around 8 seconds.

Conclusion

Based on my very rough benchmarking, the three changes above have reduced the run time of the importer from around 80 seconds to 8 seconds. The code is all at the link above, and the optimisations are on by default.

There is surely more that can be done to improve performance, but that’s fast enough for now!

Vetted - importing data into an event sourced system

2018-01-30T00:00:00+00:00

See Vetted - a new project for some background.

A task I’m going to have to tackle sooner or later is importing data from the existing Access database. As I’m going to try my hand at event sourcing, this raises an interesting question:

Given an existing application with data stored in an relational database & no notion of events, how do you go about importing data into an event sourced system?

Options

At a high level, the initial options seem to be:

Try to reverse engineer/map the existing state to events
Have some sort of migration event event in your domain model (e.g. FooMigrated) which acts as a snapshot
Run everything through your new API as commands, and allow the API implementation to take care of creating the relevant events like normal

‘Recreating’ domain events

Option one above would be nice, but seems impractical at best and is more likely impossible. For every domain object (client, patient, payment, invoice, vaccination etc) I’d need to try and reverse engineer the real-world happenings that occurred to transition the object into its current state.

A ‘migration event’ as a snapshot

Originally when a colleague suggested this it conflicted with my understanding of the term ‘snapshot’. To me a ‘snapshot’ has always been about collapsing an event stream into a single event for performance reasons. When using this kind of snapshot, the original stream of events is still available.

The second kind of snapshot (which I didn’t see immediately) is a snapshot which is used as base data. When using a snapshot as base data, the collapsed state of the aggregate at the time the snapshot is all the information you have about the history of the aggregate.

It could also be argued that the migration is a meaningful domain event in its own right, and should be captured explicitly. A CustomerMigratedEvent could result in the creation of a new customer aggregate root in the same way that a CustomerRegisteredEvent does.

Run all existing data through the new API

It should be possible to write a script that reads data from the existing database, creates commands and posts those to the appropriate API. The relevant events would ultimately be created off the back of processing the commands, so all ‘legacy’ data should look exactly the same as anything created going forward.

The outcome is probably close to option one above, but with less manual work.

Next steps

So far I’ve been spending a lot of time on the technical concepts & design of an event sourced system, without doing much on the implementation side.

It’s hard to build a useful conceptual domain model without considering implementation issues, so I think it’s time I stopped debating concepts and wrote some code.

I’m planning to explore a little and gain an understanding of how building and executing commands would differ in practice from the ‘import event’ option above.

Acknowledgements

Thanks to Roman Safronov, Chris Rowe, Martin Fowler, Mariano Giuffrida, Jim Barritt and Nouman Memon for taking the time to reply and/or chat about event sourcing! Any good ideas are theirs, and errors are mine.

Vetted - choosing an appropriate license

2017-11-05T00:00:00+00:00

See Vetted - a new project for some background.

I’m hoping to start pushing some code soon. Before I do, it’s a good opportunity to do some reading into a topic that I am more ignorant of than I should be: software licensing. The following is what I’ve learnt so far.

Disclaimer: I am not a lawyer and this may be totally incorrect, and as such should not be used as the basis for any decision ever.

Once code is released as open source software, most common OSS licenses (GPL, BSD, MIT, Apache etc) do not allow for the revocation of rights granted under the license. This is a good thing. Imagine having to be prepared for any open source library/framework you’re currently using become proprietary software with no warning. Such uncertainty would severely limit the utility of open source software.

It is possible for the current copyright owners to relicense their creations. So, theoretically, any OSS software can be relicensed if all copyright owners agree. The important part is that this relicensing does not revoke the rights assigned under the previous license. So if I’ve released some software as open source software, I can decide a year later to relicense it and create a commercial version, with the following caveats:

I still own the copyright for the entire project
- I need to have been the sole contributor to the project, or to have ensured that contributors have assigned copyright to me for their work
The existing rights assigned under the OSS license remain in place
- If the license permits, anyone can fork the project at this point and develop/use their own version

Given what I’ve learned above, I’m planning to license the project under an OSS license, but I won’t accept any contributions until I’ve got some kind of Contributor License Agreement (CLA) in place.

This is likely the first of many posts where it might seem I’m researching a topic and deliberating a little excessively, given I have no working software or even a particularly interesting idea. It’s fairly premature to assume that there are going to be any contributors to this project other than myself. I don’t believe veterinary practice management software is so exciting that I am going to be swamped with contributions. However, as I mentioned earlier, this whole project is mainly a learning opportunity for me.

Further resources:

Vetted - a new project

2017-11-05T00:00:00+00:00

When I was in high school, I created a fairly basic application for managing a small veterinary practice. It’s written in Microsoft Access and is used by my parents to manage their mobile veterinary business.

I’m toying with rebuilding it as a web application. For its current users, the main benefits of this would be:

client contact details could be made available on a mobile device;
there will be no more (or at least fewer) issues with concurrent modification (merging two access database files that have been modified independently on different computers because Dropbox didn’t sync is no fun); and
it will be accessible anywhere with internet access, so that my dad could do accounts when he is away from home and has some downtime.

For me, it would mainly be a learning opportunity.

Naming things is not my strong suit, so I’m going with ‘Vetted’ for now.

I’m going to try to do the following throughout the project:

Apply domain driven design rigorously
Apply functional programming principles
Document failures and successes
Document my design heuristics
Develop in the open
Deploy continuously to ~~production~~ somewhere
Focus on adding the most valuable parts first (e.g. make phone numbers available online) & delivering vertical slices

I’m thinking that the basic architecture for now will be:

Single page application
Elm frontend
Kotlin backend
Maybe some kind of event sourced data store, as I’d like to see how badly I can shoot my foot off

While I want to build something functional, I also want to learn about a few techniques/patterns/tools that would be applicable on larger projects, so I might be making some choices which seems strange. I’ll try to call these out as they happen. I need to remember that I am not Google.

I’m hoping to publish posts more regularly about this project, so keep an eye out if you’re interested!

Notes

Design Heuristics

I attended the excellent Explore DDD conference this year and one of my favourite talks was Cultivating Your Design Heuristics by Rebecca Wirfs-Brock.

As defined in the talk, a heuristic is:

anything that provides a plausible aid (not a guaranteed aid) or direction in defining a solution but is ultimately irrelevant to the final product

Rebecca encourages everyone to consciously document and cultivate heuristics, learn others heuristics & discuss them and ultimately adapt (or wholesale replace) your own heuristics when appropriate.

It’s a great talk, and I’m going to try to document the heuristics that I find myself using while working on this project.

Develop in the open

I’m going to be writing about what I’m building, and the code will be available on GitHub. However, I haven’t yet determined how to license the project. My basic requirement is that I retain copyright and can relicense the project if that’s ever required.

Look out for a post on this in the near future.

Debugging slow bash startup files

2017-10-31T00:00:00+00:00

Recently I found that opening a new bash session (e.g. when opening a new terminal window) was getting a bit slow on my machine. I take reasonable care to make sure my dotfiles don’t get too crufty, and I keep them all in version control.

The following is a walk through of how I went about debugging the issue.

So, how does one go about profiling what bash is doing when starting a login shell/interactive shell?

My initial thought was to use some kind of system call tracing to see what files were being opened/executed. dtrace exists on OS X, so let’s try that:

sudo dtruss -ef bash

Sadly, the output isn’t overly useful due to System Integrity Protection. I don’t want to boot into recovery mode, so what are our options?

I regularly add set -o xtrace to my bash scripts … would the same thing work for my .bashrc? I added the line, and executed bash:

+ source /Users/mnewman/.bash_profile
++ export PATH=/Users/mnewman/bin:/Users/mnewman/perl5/bin:/Users/mnewman/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mnewman/.rvm/bin
++ PATH=/Users/mnewman/bin:/Users/mnewman/perl5/bin:/Users/mnewman/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/mnewman/.rvm/bin
++ for file in ~/.{path,bash_prompt,exports,aliases,functions,extra}
++ '[' -r /Users/mnewman/.path ']'
++ for file in ~/.{path,bash_prompt,exports,aliases,functions,extra}
++ '[' -r /Users/mnewman/.bash_prompt ']'
...

It looks like that works (the above is showing the start of my .bash_profile, which is sourced from .bashrc). There is a lot of output there though, and we still don’t have any timing information. A little searching for variants of bash add timestamp to each line led me to an SO answer recommending ts. Looking at the manual page for ts:

$ man ts

NAME
       ts - timestamp input

SYNOPSIS
       ts [-r] [-i | -s] [format]

DESCRIPTION
       ts adds a timestamp to the beginning of each line of input.

       The optional format parameter controls how the timestamp is formatted, as used by strftime(3). The default format is "%b %d %H:%M:%S". In addition to the regular strftime
       conversion specifications, "%.S" and "%.s" are like "%S" and "%s", but provide subsecond resolution (ie, "30.00001" and "1301682593.00001").

       If the -r switch is passed, it instead converts existing timestamps in the input to relative times, such as "15m5s ago". Many common timestamp formats are supported. Note that
       the Time::Duration and Date::Parse perl modules are required for this mode to work. Currently, converting localized dates is not supported.

       If both -r and a format is passed, the existing timestamps are converted to the specified format.

       If the -i or -s switch is passed, ts timestamps incrementally instead. In case of -i, every timestamp will be the time elapsed since the last timestamp. In case of -s, the time
       elapsed since start of the program is used.  The default format changes to "%H:%M:%S", and "%.S" and "%.s" can be used as well.

So far so good, it looks like we could use ts -i and get the duration of every command! I’d like to try this out, but how can we redirect the xtrace output to ts?

Some further Googling led me to this SO answer, which suggests using the BASH_XTRACEFD variable to tell bash where to write its xtrace output. After some trial and error, I added a few lines to my .bashrc:

# open file descriptor 5 such that anything written to /dev/fd/5
# is piped through ts and then to /tmp/timestamps
exec 5> >(ts -i "%.s" >> /tmp/timestamps)

# https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html
export BASH_XTRACEFD="5"

# Enable tracing
set -x

# Source my .bash_profile script, as usual
[ -n "$PS1" ] && source ~/.bash_profile;

Upon restarting bash, this produces (a lot of) output in /tmp/timestamps, and each line contains an incremental timestamp, like so:

000046 ++ which brew
003437 +++ brew --prefix
025518 ++ '[' -f /usr/local/share/bash-completion/bash_completion ']'
000741 +++ brew --prefix

These particular lines tell me that a brew --prefix command executed and took 20ms.

With output like the above, I had enough info to track down a couple of slow loading scripts (like sourcing nvm.sh) and remove them from my .bashrc/.bash_profile.

Denver and the Rocky Mountains

2017-09-26T00:00:00+00:00

Last week I went to Explore DDD (which I would highly recommend) in Denver on Thursday and Friday, joined a tour to the Rocky Mountains on Saturday, and explored Denver on Sunday.

Content of the conference aside, even the view from the venue was pretty amazing:

Sadly the weather was pretty poor on Saturday for the Roucky Mountains visit, and there wasn’t lots to see out of the lookout 12,000 ft (~3600m) up:

Denver is a cool city, figuratively and literally. It’s quite high, almost exactly one mile above sea level. The Colorado capital building is in Denver; the stairs out the front have a plaque to show which one is exactly one mile above sea level, but measurements kept getting more and more accurate, so there are actually three plaques!

A few random facts from the visit:

The trees with yellow leaves in the picture above are aspens, and they actually grow in clumps all connected via a common root system
Denver convention center is the 13th largest building in the world
The Broncos (the Denver NFL team) have sold out every home game for the last 23 years
If attacked by a black bear, fight back
Up high, where the trees can only grow for a few months a year, a tree thats 2-3 inches thick might be 80 years old
A building that’s part of the Denver art gallery contains no right angles
The eagle atop the Denver local government building was made by someone who hated the mayor who planned the construction of the whole building. Apparently you only see eagles with their wings in that position when they’re a) about to take off, or b) defacating
In Denver, 1% of every public construction project must be set aside for a public art installation nearby the building
The Denver mint makes coins for China
This art piece was meant to be for a primary school and was trying to make a statement about the perspective of children (the horse atop the chair is actually a full size horse). However, the school didn’t like it and it ended up at the library. Another installation at the convention center pokes a bit of fun at this piece.
‘Iseewhatyoumean’ is the name of a huge blue bear who’s always trying too see what’s happening in the Denver convention center, and is consistently voted the favorite public are installation

Despite the weather it was a fun weekend, I’m looking forward to going to Explore DDD again in 2018.

Migrating a GitHub Pages blog with a custom domain to HTTPS

2017-07-06T00:00:00+00:00

At the time of writing, this blog is hosted on GitHub Pages, which has been working well since I set it up a few years back.

The only thing that has bugged me for a while now is that the whole site was served over HTTP, rather than HTTPS.

I wanted to move to move this blog to HTTPS, but with some constraints:

Continue using GitHub pages (it’s free and easy, I don’t want to manage a server)
No certificate renewal (smart me plans for stupid me, who would surely forget to renew a cert)
Continue using my domain (mdjnewman.me)
No cost

GitHub pages doesn’t support HTTPS for custom domain names, as it uses a certificate with a wildcard SAN of *.github.io.

CloudFlare offers HTTPS on a free plan, which Troy Hunt has written about before.

It looks like this will meet my constraints above - I get to keep using GitHub Pages, I don’t have to manage a cert (CloudFlare takes care of this), and I can keep using my custom domain.

The steps I followed to do this were relatively simple:

Exported a zone file from current nameservers
Completed the CloudFlare onboarding, during which I imported the above zone file
Updated the authoritative DNS servers for my domain to the *.ns.cloudflare.com name servers:
Tested the site out, fixed a CSS link that was loaded over HTTP
Forced HTTPS in CloudFlare:

… and that was it. I finished this in part of an afternoon.

Shortcomings

There is one major shortcoming with this setup: there is no certificate validation between CloudFlare and GitHub (CloudFlare supports fetching from an origin without validating certificates, which is the option I’ve chosen - ‘strict’ HTTPS can be enabled for most use cases).

As we mentioned before, the GitHub cert is valid for *.github.io, and we’re using my custom domain, which is mdjnewman.me.

If we switched off the custom domain on GitHub, and did some smarts in CloudFront to rewrite requests so that the request to GitHub was using mdjnewman.github.io, then we’d get HTTPS all the way to GitHub servers.

CloudFlare does support rewriting HTTP Host headers , but it’s an enterprise feature.

I could switch to using CloudFront with an AWS Certificate Manager cert, which would meet all the above constraints except for ‘no cost’ (admittedly, my tiny blog doesn’t get much traffic, so the cost would be minimal).

Given that most of the shenanigans with injecting content into web sites happens at the last leg of a connection (I’m looking at you, dodgy internet cafe), I’m happy that the new setup for this blog mitigates that problem and am willing to accept the cost/security trade-off. While it’s possible for someone to perform a man in the middle attack and impersonate GitHub, given my site has no sensitive information I’m not too worried about this threat model (Troy Hunt also wrote about this trade-off).

When should I force push after rebasing?

2017-07-05T00:00:00+00:00

tl;dr: assuming it’s your own branch, immediately.

On the last few projects I have been a part of, I have been the defacto Git problem solver.

(obligatory xkcd)

On more than one occasion, I was presented with something like the following:

$ git l master
* 232e985 (origin/master, origin/HEAD, master) Feature 4
* 94989d0 Feature 3
* 296d0b3 Feature 2
* 4b856cd Feature 1

$ git l feature-5
* 5c31f6d (HEAD -> feature-5) Feature 4
* 420f83b Feature 3
* 724236a (origin/feature-5) Feature 5
* 296d0b3 Feature 2
* 4b856cd Feature 1

In the above, the commits adding features 3 and 4 on each branch are logically the same, but on the feature-5 branch they’ve somehow ended up after the commit adding feature 5.

The team was using feature branches, and the author of feature 5 above was trying to rebase their changes onto master, but somehow ended up inserting their commit between commits on master. At this stage, you’re in a pretty bad place, as you’ve diverged from master.

How did this happen? I’m guessing it was the following series of events:

Developer branches from master to create feature-5
Features 3 & 4 are pushed to master
Feature 5 is committed to the feature-5 branch
Developer runs git pull --rebase origin master or similar
… some time passes …
Developer runs git pull --rebase without really thinking about it

After the first few steps above, we have something like the following:

So far, so good. We want to rebase our changes onto master, so that we can test and push our code. After git pull --rebase origin master:

Still looking good. At this stage, we could git push --force origin feature-5 and all would be well in the world.

But what happens if we go for a tea and forget what we were doing (or we use an overzealous git GUI tool), and we try to rebase onto origin/feature-5?

What we see above is the result of Git rebasing commits 94989d0, 232e985 and 44e1c44 on top of origin/feature-5. As the commit ID is computed by hashing of the contents of a commit and its parent, the same logical commits from master now exist on our branch with different IDs.

This could have been avoided if we followed this rule of thumb:

If you’re working on your own branch, always push immediately after rebasing.

Some people on the team were seeing a message like the following:

Your branch and 'origin/feature-5' have diverged,
and have 3 and 1 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

and assuming that they should git pull --rebase, which in this case is exactly what you don’t want.

If you’re looking for further reading Getting Started - Git Basics covers the fundamentals and LearnGitBranching is a great interactive tool.

Diagrams were created using Gitgraph.js, and this script creates an example repository.

Monad transformers

2016-07-31T00:00:00+00:00

I’ve been spending some time working through Haskell Programming From First Principles recently which is a very comprehensive introduction to functional programming and Haskell.

One of the exercises to implement a simplified command line version of Morra, which involves keeping track of scores and reading user input.

My initial method to play a round looked something like this:

playRound :: Config -> Scores -> IO (Bool, Scores)
  playRound config scores = do
  putStr "P1: "
  p1HandMaybe <- getHumanHand
  case p1HandMaybe of
    Nothing -> return (True, scores)               -- error case 1
    Just x -> do
      putStr "P2: "
      p2HandMaybe <- getHumanHand
      case p2HandMaybe of
        Nothing -> return (True, scores)           -- error case 2
        Just y -> do
          let evenWins = (x + y) `mod` 2 == 0
          return (False, updateScores config evenWins scores)

playRound takes the configuration for the game and the current score, and returns a side effecting computation that will return a tuple with the new scores and a boolean indicating if the game is finished.

The method getHumanHand used above returns a IO (Maybe Int), which can be interpreted as a side effecting action that might return an integer (in this case, the side effect is reading from the console and we can’t trust the user to enter an integer, hence the Maybe).

The problem then is that we’re then manually unpacking these Maybe Int values, which leads to the ugly nesting and case statements. However, we can see on the lines marked ‘error case’ above that the handling for both cases is the same - we assume that if the user has entered something other than an Int that they want to end the game.

I recently learned about Monad transformers, which allow you to compose monads. In this case, we want to compose the Maybe monad with the IO monad, so we will use MaybeT.

Rewriting getHumanHand to return a MaybeT IO Int and rewriting playRound results in the following:

playRound' :: Config -> Scores -> IO (Bool, Scores)
playRound' config scores = do

  newScores <- runMaybeT $ do
    liftIO $ putStr "P1: "
    p1Hand <- getHumanHand'
    liftIO $ putStr "P2: "
    p2Hand <- getHumanHand'
    let evenWins = (p1Hand + p2Hand) `mod` 2 == 0
    return $ updateScores config evenWins scores

  return $ case newScores of
    Nothing -> (True, scores)
    Just x -> (False, x)

The nice thing about this implementation is that we’ve avoided the need for pattern matching, as the do block above where we’re dealing with the potentially failing computations will immediately short circuit and return a Nothing if either user fails to provide a legitimate value.

This is the first time I’ve actually used a Monad transformer, and it was good to see how it cleans up the implementation of playRound.

Europe

2016-07-21T00:00:00+00:00

The post below is a concatenation of multiple emails I wrote during the trip, apologies if it seems a litte disjointed.

I’m writing the first part of this as I sit on the train on the way to Salzburg, there’s about an hour left of a trip that takes a little under two hours.

I had three very busy days in Munich, managing to see

Deutsches Museum
Marienplatz
BMW welt
BMW museum
Olympic park
Tollwood SommerFestival
Residenz (external and garden)
Englischer Garten
Schloss Nymphenburg, Gardens & Outbuildings
Dachau Concentration Camp Memorial

The museum was awesome, I kept thinking how much I would have loved a museum like that when I was little. It was primarily a technical museum - the sections were mining, marine navigation, oceanography, foundries, machine tools, metals, power machinery, marine navigation, electric power, new technologies, physics, energy technologies, pharmaceutics, historic aviation, ceramics and glass production, paper, musical instruments, astronomy, measures and weights, computers, mathematics, microelectronics (I geeked out a bit in these last three), chronometery and geodesy. It is a really big place, you could easily spend upwards of an hour in each of those sections. I spent about 6 hours in the museum and easily could have spent twice that, though maybe over two days; there is only so much museum-ing I can handle.

First up on Thursday morning I went to the area of the city where the BMW headquarters and factory are located.

There is also building called BMW Welt (‘welt’ -> ‘world’) which was like a showroom on steroids. They had some Rolls Royces there too because BMW, Mini Coopers and Rolls Royces are all either owned by one company or they have some partnership going on. I planned to do a tour of the BMW factory but it was sold out (until August!) so I settled for the museum.

The BMW museum had all sorts of stuff from the life of the company over the last 100 years and talked about how much it’s changed, from manufacturing military parts to bikes, cars, airplane engines and even pots and pans at the end of the war. They used POW labour at one point during the war which I didn’t expect given a company like BMW, but this was during the war when they had a lot less control over their operations.

Once I was done at the museum I made my way into Olympiapark where the 1972 Olympics were held. This whole area has been turned into a public space and the stadium and halls are still used for events. While I was there an event that’s on this weekend was being setup, and there was wake boarding and some slopestlye mountain biking going on. They were still completing the mountain biking track so no one was doing full runs but there were pretty massive jumps in there. I got a video of one guy casually doing backflips for the crowd!

I also spotted an event called the Tollwood Sommerfestival from a lookout so I went and had a peek there. It had a cool vibe and would have been good fun later in the evening, but it was a long way from where I planned to finish the day.

From the Olympic Park I made my way back to Munich Residenz where the Bavarian monarchs used to live but didn’t have time to go inside. It’s a museum now but I just checked out the externals and went through the courtyard on my way to the Englischer Garten.

The English Gardens are massive, I only walked through half of them in then end despite plans to go from one end to the other. There is and really cool spot at the start of the gardens where the water flows in (not sure if it’s flowing through or recycled through the canal system in the park - maybe both), there were people surfing there! They’ve set something up that makes an artificial wave across the canal.

In the middle of the gardens is this relic of a bit of a China craze that went through Munich a while back. There is this pagoda type thing, and all around is the most German food/drinks setup ever. So many 1L mugs of beer! I had my dinner of sausages and sourkraut there along with a wheat beer before making my way home.

First up on Friday I went to Nymphenburg Schloss which is a huge palace some Bavarian rules used to live in. Construction started in 1664 and there were still buildings being added and pretty big modifications being made to the grounds until 1825. Water is a very big feature throughout the grounds and is provided by a 2km long channel that brings water from a nearby river. There was a really run down looking building called the Magdalenenklause in the gardens, but it was actually built to look that way as an escape from the prim and proper life at court.

After I’d wandered all around the gardens and reached appropriate levels of sunburn, I made my way to Dachau, where there was a concentration camp. This wasn’t a camp where mass executions were carried out in gas chambers, though there was a chamber built there. The people running Dachau instead really believed in ‘extermination through labour’, whereby they overworked people until they pretty much died of exhaustion. The gates to the camp said “work shall set you free”, though freedom in this case normally meant death. There were 32 huts each originally built to house 250 people, but apparently they each held upwards of 1000 people towards the end of the war. Over 40,000 people died/were executed there in the twelve years it was operating. Sobering place.

After Dachau it was time to head to Salzburg which was very easy to figure out. The public transport in Munich is amazing, I had no trouble getting around despite not knowing the city and not understanding what anyone was saying or what anything said.

This is yet another instalment that’s being written while on the train, this time from Salzburg to Vienna. I have internet on this train, and it’s probably faster than what 99% of Australian ISPs provide (thanks Tony). It made me laugh when I connected to WiFi and saw this immediately:

… ‘this is of course free’. In Aus this would cost an arm and a leg and take four hours to set up.

During my two days in Salzburg I visited:

Festung Hohensalzburg - a fortress
The Domquartier - a museum combining the church and residence of the ‘Prince-Archbishop’
Kapuzinerberg - a mountain in the middle of town, not as heavily fortified as Festung Hohensalzburg
Schloss Mirabell - a palace built around 1606
Augustiner Brauhaus
A few other places around Salzburg, though only the exteriors:
- Mozarts geburtshaus and wohnhaus (where he was born and lived, respectively)
- Christian Doppler’s geburtshaus
- About 1000 churches

Festung Hohensalzburg was probably the hightlight for me. There is evidence that there have been people on Festungsberg since year 0180, and construction started on Festung Hohensalzburg in 1077. It’s no wonder people have been using it as a stronghold for so long, it is an amazing place to put up a strong defence (it’s actually never been taken by force, but was surrendered once during the Napoleonic War of the Second Coalition in 1800 when the ruler at the time bailed to Vienna). I think the only people to actually attack it were some peasants in 1525 who were promptly repelled. The fortress was commissioned by the ‘Prince-Archbishops’ of Salzburg. These guys were the supreme rulers, as they’re non-secular princes as well as archbishops of the church. One of the really interesting things there is the ‘Salzburger Bull’, which is a very loud mechanical organ that was used to signal various things to the town (photo here). The fortress was used as a military outpost until 1861.

Next up was the Domquartier, a museum that combines a palace where the Prince-Archbishops used to live and do the princely thing and the cathedral where they used to be pray and carry out their duties as archbishops. The residence building was initially built in 1604 and the cathedral in 1614. The two buildings have been joined on the second level since one of the Prince-Archbishops decided they didn’t like getting their feet wet when walking between the two. The museum is setup so that you follow the path that a visitor would have followed when seeing the Prince-Archbishop, which mainly involved going through several extravagant rooms. The first room is huge and not as fancy, but the further along you get the smaller and more expensive-feeling the rooms get, this is apparently because the more important you were the further into the palace you could get. The rooms all have portraits of Alexander the Great painted on the ceilings, which were all indirectly drawing comparisons between Alexander and the Prince-Archbishops you were about to see. Sadly I don’t have any photos of the interior of the palace as they weren’t allowed.

It was a bit of a rainy afternoon so I started making my way back to the hotel and decided to check out Augustiner Brauhaus on the way. It’s massive - 5,000 square meters internally and with a beer garden that can sit 1,400 people. The beer is rolled out in massive barrels from which people fill up their steins.

On Wednesday I went to Mozarts geburtshaus and wohnhaus but decided I’d had enough of museums and instead went for a walk up the Kapuzinerberg. The Kapuzinerberg is a mountain in the middle of town, not as heavily fortified as Festung Hohensalzburg. There is one castle up there (Franziskischlössl) and a massive wall that goes all the way back towards the river to another stronghold. This second stronghold could be used to defend the eastern side of what was at the time the only bridge across the river (Hettwer-Bastion). I walked the wall back from the Franziskischlössl to the Hettwer-Bastion, going past some signs that looked vaguely like they were warning me about something. Building this wall now would be a pretty impressive feat, let alone building it in three years from 1629-1632. I think it would take the Brisbane City Council about 100 years and several billion of machinery. The Franziskischlössl and the wall were built such that there are no angles were attackers could hide without being under fire. The main drawbridge into Franziskischlössl is protected on three sides, and there were watchtowers built within shouting distance of each other all the way back to Hettwer-Bastion.

From the end of the wall I made my way to Schloss Mirabell - a palace built around 1606, also for the Prince-Archbishops. The palace building itself is still used by the local council and events and concerts are held there, but the gardens are open for all to see. One of the features of the gardens are 14 marble dwarfs which were put there in 1710 as part of an outdoor theatre. Several scenes from The Sound of Music were shot in the gardens.

Ahoy! (which is apparently a common greeting used by people in Bratislava as well as by pirates)

In keeping with the train theme, I’m starting this update on the train back to Vienna.

Since arriving in Vienna on Sunday night, I have checked into our Airbnb, done some exploring of Vienna, and spent the day in Bratislava, the capital of Slovakia. Bratislava just on the Slovakia side of the Austria-Solvakia border and only an hour in the train from Vienna.

We went for a walk along part of Donauinsel on Monday morning. Donauinsel is a roughly 20km long island in Vienna on the Danube River. From there we walked back towards our apartment, hoping to walk through a local university but somehow walking past a horse racing track and a huge construction site. After a lunch at home with groceries from the local supermarket and Kirstens arrival we headed off to explore some of Vienna.

From the bridge to Donauinsel, we’d seen a very large church nearby so we had a look at that first. The church was ‘Heiliger Franz von Assisi’, or the St. Francis of Assisi Church. It was built much more recently than most of the churches around here, construction only started in 1898! The interior was quite plain compared to how it looked from the outside:

After checking out the local church, we caught the U-bahn (Untergrundbahn, or undergroud railway in English) to Schönbrunn Palace. The first part of the mansion there was built in 1548 and was normally occupied by members of the Habsburg Monarchy, including Franz Joseph, the longest-reigning emperor of Austria. It’s been a museum since the fall of the Austro-Hungarian Empire in 1918. It’s a pretty striking place:

The final stop on Monday was Karlskirche (or St. Charles’s Church) which was built between 1716 & 1737. The columns outside this church are pretty unusual:

I will send another update about Bratislava, but it’s Wednesday morning now so I’m off to explore some of Vienna :)

I’m not writing on a train this time, but having a relaxed day in Vienna as my feet feel like they’re about to fall off. I will probably head out and go to a museum or something this afternoon.

As I said in my last email that we went to Bratislava on Tuesday which was a really good trip. Most of our time there was spent on a free walking tour of the city (‘free’ but tips expected - our guide was really good though). I made a map of the path we took.

There are a lot of statues in Bratislava, including one of Hans Christian Andersen (author of “The Emperor’s New Clothes”, “The Little Mermaid”, “The Nightingale”, “The Snow Queen”, “The Ugly Duckling”, “Thumbelina” …) which we would have walked straight past if our guide had not pointed it out. The back of the statue has lots of things from his stories. Another one is the ‘man at work’:

Courtesy of Frettie on Wikimedia Commons

This statue doesn’t really have a name, but a sign warning saying ‘Man at work’ was put up after people kept tripping over him. According to the locals his name is Cumil and he is the quintessential Bratislavian man at work, as he just chills out all day and perves on the tourists. The executioner below has been present for quite some time in one form or another:

This is at the entrance to the lane where the executioner used to live (his house is now a massage parlour!). The lane where the executioner lived was next to Michael’s Gate, part of the fortifications for the city build in the mid 14th century. A second gate was added along with a 90° corner in the 17th century, so that enemies had to slow down even if they breached both gates.

From Michael’s Gate we walked on to the Slovak National Uprising Square. This square, as well as another in the middle of the city, were important sites in the Velvet Revolution which in 1989 ended the communist rule in then Czechoslovakia (the split into the Czech Republic and Slovakia happened in 1993).

After the SNP Square we visited The Church of St. Elizabeth/The Blue Church/The Smurf Church which was built in 1908 and is very, very blue:

After a final short walk we were sitting at the entrance to a building when our guide was telling us more about the Soviet invasion of Czechoslovakia in 1968. She handed around a physical copy of a rather controversial photograph. I haven’t included it here, as the image isn’t in the public domain, but after looking at it we realised we were sitting on the stairs of the building in the background!

The tour guide also told us some interesting stories about traditions in Slovakia. For men, easter apparently involves pouring water on the women in your life and lightly hitting them with willow sticks. Christmas means it’s time to catch a carp and keep it in the bathtub for a few days before eating it for a Christmas meal.

I did some exploring of Vienna yesterday, but I will send another email about that!

I’ve finally got around to writing up what I got up to in Vienna, while I’m on the train of course! We’re currently on the long ride from Vienna to Mainz. The good thing is the train is currently going at 170 km/h. I’ll split this into a few emails, the first one is about Wednesday when I explored the inner area of Vienna.

I started out at the Belvedere, a complex consisting of two palaces, a botanical garden and even a small zoo. The large palace now houses a museum (or an art gallery, I can’t remember - there are so many galleries and museums in Vienna). The palace was built was built as a summer residence for Prince Eugene of Savoy sometime in the early 1700’s. Savoy was a big fan of animals and had a small zoo setup there, though that’s all gone now.

On my way to the centre of old Vienna I walked past Heldendenkmal der Roten Armee (the Heroes’ Monument of the Red Army), a Soviet War Memorial built in 1945 (that is at times a little unpopular with the locals, as the Soviets caused quite a bit of damage when they took the city in a battle from 2 to 13 April 1945 and in the weeks following the battle).

From there I walked north to the Ringstrasse (Ring Road) which forms a loop around most of the old buildings and museums. There was lots to see, but I didn’t actually go into many places as they all had a roughly €10 entrance fee and there wasn’t time to do them justice on the Wednesday. I walked past a bunch of places inside the Ringstrasse, including

The ‘Haus der Musik’ was my first stop inside the Ringstrasse, it is kind of like a museum with ways for people to explore sound and how sounds are generated. Entry was a bit expensive and I wasn’t sure if I wanted to go in, but there was a choir performing in the lobby. When I heard the person organising the choir speak after one of their songs, I thought they had a bit of an Australian accent. Sure enough, it was a choir from Adelaide called Young Adelaide Voices.
A bunch of embassies
Wiener Staatsoper (Vienna State Opera), an opera house that opened in 1869
Albertina, a museum/gallery built in 1805 with approximately 65,000 drawings and approximately 1 million old master prints
Hofburg Palace, a massive complex including Österreichische Nationalbibliothek (Austrian National Library), Michaelerplatz (an archaeological site), Joseph Square, Heldenplatz, Kaiserappartements (see the image below)
Austrian Parliament Building, completed in 1883 & still used today
Rathaus (town hall) also opened in 1883. There were a bunch of massive screens setup for people to watch the Euro Cup, we came here on Thursday night to watch the Germany/France game
Burgtheater, a theatre built in 1888

Google Maps view of a few of the places I visited

The final stop for the day was Stephansdom (St Stephens Cathedral), one of the larger churches in Vienna (the bell there is the largest in the country and weighs just over 20 metric tonnes). The original church on the site was consecrated in 1147 and the cathedral in its current form was finished in 1359 (some parts are still from the old church).

The one place I did pay to go into on Wednesday was the katakomben (crypts) at the cathedral. The crypts have been there since the 13th century, and everyone who died in the town used to be buried down there. There are remains from 11,000 people (400 people per room) buried under the church over 30 years until 1783 until the smell grew too strong to run masses and they also ran out of space. When they ran out of space they had prisoners empty out full rooms, clean the bones and stack them neatly in ‘bone houses’ so they used less space. We weren’t allowed to take pictures down there, but there were rooms stacked floor to ceiling with all sorts of bones, though skulls & femurs were the most recognisable. Duke Rudolph IV is in the crypt, who died in 1365 (his coffin was replaced in the 18th century as the old one was falling apart).

The final interesting fact from the crypt is that when a member of the Habsburg family (who used to run the show here, the same family I’ve mentioned earlier) died, there used to be disagreement about where their remains should be placed, as there were three locations that were of particular significance to the family. What ended up happening is that each location got a piece - their heart was sealed in a chalice type thing and sent to one location, their other organs where sealed in another larger container and placed in the crypt at St Stephens, and their bodies were put into a coffin and stored at the palace (I think, I might be wrong on the location). Some of the containers that the organs are in are larger than then others, the guide told us this is because some of them sprung a leak! The leaky container just gets resealed in a larger one whenever it happens. Apparently it happened a few years back and they had to close the crypts for a week because of the smell.

On Thursday afternoon I visited Kaiserliche Schatzkammer (the Imperial Treasury).

The treasury was split into two sections, secular and ecclesiastical. There were a bunch of interesting things in there. In no particular order:

a ‘unicorn horn’ from the 1500’s, which was actually the tusk from a narwhal
a part of the ‘True Cross’, though I swear if you added up every piece of the cross you’d have a forest
the point from a ‘holy lance’ in 750 - ‘a guarantee of divine assistance of victory in battle’ (after all, what’s more holy than winning wars?)
artefacts from the ‘Order of the Golden Fleece’, a secular order of chivalry from 1430. I don’t know what is particularly chivalrous about a sheep skin, and the chain their members wore has a golden but droopy looking sheep on it.
the basin the royalty were baptised in - solid gold!
a 2.68 carat emerald vessel which an emperor tried to use as security for a loan in the 17th century, but was too priceless to be valued
a swordbelt for the imperial sword, from the 12th or 13th century
John the Baptist’s tooth (apparently)

I also walked past this store on my way home:

I guess marijuana is legal in Austria!

On Friday I went to Seegrotte Hinterbrühl. “Hinterbrühl” was an underground gypsum mine until 1912 when a blasting operation in went awry and 20 million litres of water flooded the mine. It was opened as a tourist attraction in 1930 and has been open to tourists since then, with the exception of World War II. During WWII the mine was used as a factory for Heinkel He 162 jet fighters during, there were 2,000 people working down there as the underground site couldn’t be bombed.

Currently 50,000 L of water have to be pumped out every day to keep the water level constant. The water is super clean and apparently drinkable, though very mineral heavy. It was also surprisingly cold down there at only 9 °C. The guide was doing his thing in German, but I heard someone else translating for their friend and asked if I could listen in.

Burg Lichtenstein was also nearby. It was originally built during the 12th century, destroyed by the Ottomans in 1529 and 1683, and remained in ruins until 1884, when it was rebuilt. I only realised while doing some reading to write this email that the family who established Liechtenstein the country (or principality?) took their family name from the castle, and the country is named after the family. The castle wasn’t open for tours, so I just had a walk around there and the surrounding forest. There was a amphitheater built near the castle around 1800 which is now all overgrown, it was surreal walking through it.

You can see almost the whole structure in the background of the image below.

I got chatting to the two people I was eavesdropping on at the caves and they recommended going to Schneeberg mountain south of Vienna, so I decided to do that on Saturday.

Getting there was a bit of a debacle - I was on the train which was meant to be going all the way to my destination and we’d been stopped at a train station for a while, which was odd. Soon enough a ticket inspector wandered through the train and said something to me in German which I didn’t understand. After I made it clear I didn’t speak German, he just said the destination station I was going to, and I nodded. He then pointed at a train on the other side of the station and said one of the only German words I know: “schnell, schnell!”. “Schnell” is roughly “quick”, which was enough instruction for me. I galloped and hopped on the other train just before it left.

Eventually I made it to Puchberg am Schneeberg, which is at the foot of the mountain. I’d been planning on maybe hiking up the mountain but I didn’t have enough time so I caught the ‘Schneebergbahn’ train up to the top. The track up is a cog railway that was built from 1895-1897.

There was still a decent walk up to from where the train station is to the peak, and I didn’t have loads of time, so I powered from Berghaus Hochschneeberg (1800m) to Klosterwappenhohle (2076m) and Kaiserstein (2061m) and then back:

(the map above is from OpenStreetMap, which is an open source alternative to Google Maps and often has more detailed walking/hiking/biking tracks).

I was a little underprepared for how windy and cold it would be up there (there is still some snow that hasn’t melted yet). Here is me with my hands firmy in the pockets of my inadequate jacket:

The cold was worth it though, the views were amazing:

BFPG talk - Functional Programming in Scala Chapters 7 & 8

2016-06-19T00:00:00+00:00

In May, I presented a talk based on Chapters 7 & 8 of Functional Programming in Scala at the Brisbane Functional Programming Group.

The talk explores the design and implementation of two functional libraries, one for parallelism and one for property based testing.

The video of the talk (also embedded below) is now available online at the BFGP talks site, along with a link to the slides.

See http://talks.bfpg.org/past.html for other past BFPG presentations.

-Xlint:unchecked and -Werror for Java projects with Gradle

2016-05-04T00:00:00+00:00

I was recently working in a codebase where there were chunks of Java code that used generic interfaces from the Apache Commons Collections project as well as java.util Collections in a type unsafe way, even though the interfaces/classes that were being used supported generics.

Our gradle build was emitting the following warning:

:compileJavaNote: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

This warning should be an error in my books. It’s possible to get this behaviour by adding the following to your build.gradle file:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:unchecked" << "-Werror"
}

This way, when our code is built everyone is aware when they’re using unsafe operations, and can either fix the issue or supress the warning with @SupressWarnings("unchecked") if it’s intentional.

Johannesburg & the ThoughtWorks Pan Africa Away Day 2016

2016-02-29T00:00:00+00:00

I’m back in Australia after a quick trip to South Africa to attend the ThoughtWorks Pan Africa Away Day 2016.

I left on Tuesday morning, and was early to the airport in Brisbane so I got on an earlier flight to Sydney before flying to Johannesburg. The flight from SYD-JNB was good, it was the first time I’ve flown with Qantas internationally and I was pretty impressed. I was following the sun so by the time it got dark here I’d been in daylight for about 20 hrs which was a very long day!

ThoughtWorks SA had organised a pickup from the airport and the hotel, so a driver picked me up from the airport. On the way to the hotel he told me about how different areas had different races living in them during apartheid. Some nicer looking houses have very serious security, but apparently home invasions are becoming less of a problem. There were still some questionable characters in parts though so I’m sure home invasions still happen, it’s interesting how much the crime rate varies depending on who you talk to.

I spent Wednesday in the ThoughtWorks office. It’s a nice office and much bigger than Brisbane despite having a similar number of people working out of it. I met some of the locals and another traveller here for the away day from Brazil. All up there were 32 international visitors, some there doing talks, some attending like I was and a lot from ThouthWorks leadership/management. The guests were from Brazil, China, Ecuador, Germany, India, Kenya, Singapore, Turkey, UK and the US.

On Thursday we did a tour of Soweto, an area in Johannesburg. First we walked through some of Motsoaledi, an ‘informal settlement’ (a.k.a. shantytown/favela). There are around 8,000 people living there - you should be able to pick it out on Google Maps. One thing I found pretty crazy in all these areas is how much Coca Cola sponsors things - most people there don’t have running water, but there are Coke signs everywhere!

From there we went to Vilakazi Street, which is the only street in the world to have two Nobel Peace Prize winners living there - Nelson Mandela and Desmond Tutu. Mandela’s house is a typical government built house from that era, a very basic four bedroom place. It’s now a (small) museum:

After reading some of what he and his family put up with, it’s pretty amazing he was as forgiving as he was when was released from jail!

We also went to the Hector Pieterson Museum which is all about the anti-Afrikaans Soweto Uprising - in 1976 some politician decided that all schools (where students were already struggling with English) would now run lessons in Afrikaans. A whole load of students refused to go to school, and lead a march and the police reacted pretty brutally - 13-year-old Hector Pieterson was one of the first to be shot dead.

On Friday we left to Khaya iBhubesi, the venue for the Away Day. Despite the name translating to ‘home of the lion’ there wasn’t a lot of wildlife around, though there were some lions there in an enclosure.

Friday and Saturday were mostly talks from ThoughtWorks Global and Africa leadership. There were events on Friday and Saturday night too, so I didn’t get a whole load of sleep (though I think that made jetlag easier to handle when I got back; I slept the whole flight back and then the whole night on Monday too!).

On Sunday we drove back to Johannesburg and then I went with two people from the Ecuador office to the Apartheid Museum. It’s crazy how arbitrary the laws were in that time!

After the museum I was off to the airport.

It was a really good trip even if it was a bit quick. It would have been good if I had more notice and could plan to visit other countries in Africa, but I guess that will have to wait until next time!

Pedantic console for JavaScript tests

2016-01-30T00:00:00+00:00

When running JavaScript tests that fire up a browser, it can be useful to ensure that console.warn and console.error aren’t being called, and to have your automated build fail if this starts happening.

If you’re using a framework like Jasmine, you can place the following file in your helpers directory to ensure that an exception will be thrown whenever a call is made to console.warn or console.error:

console.error = (function () {

  var originalConsole = console.error;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.error called, failing test');
  }

  return myError;
})();


console.warn = (function () {

  var originalConsole = console.warn;

  function myError () {
    originalConsole.apply( this, arguments );
    throw Error('console.warn called, failing test');
  }

  return myError;
})();

I recently used this to locate all the places that a deprecated Moment.js feature was being used in a codebase.

Haskell stack, Yesod and Docker

2016-01-01T00:00:00+00:00

The why

Over the break I’ve been working on a web app to replace a fairly old MS Access Database that I built for my Dad to use in 2009 (he has a mobile vet business).

This seemed like a good chance to try out Yesod, a web framework for Haskell. The Yesod philosophy is to leverage the Haskell type system wherever possible. For example, in the Hamlet templating language everything from generating URLs to including static files and generating forms is checked at compile time.

stack is a cross platform build tool for developing Haskell programs which (when combined with Stackage makes Haskell development much more enjoyable.

I’d recommend stack for any new Haskell project, and hopefully this post can point someone else in the right direction for a stack/Yesod/Docker project.

I built a very basic site with Yesod and stack locally on OS X, based off the yesod-postgres template (stack new mysite yesod-postgres). The yesod dev server makes this all very easy, just running stack exec -- yesod devel will keep Yesod running and recompiling the app whenever any changes are made.

Soon it was time to make the app available somewhere in order to start getting some feedback. At this stage, the most basic requirement for the hosting was that it be low cost. Digital Ocean came to mind (use this link for $10 credit), and I’m now using their one-click Docker droplet at USD $5 per month. This host can run both the Yesod frontend and the PostgreSQL database.

Binaries built locally on OS X won’t run on a Linux-based Digital Ocean droplet, and it was a bit early to spend $$ on something like Snap or Travis, so I needed a VM or container to run the build in.

The how

Set up

stack has support for Docker to run the build and package the application, but it isn’t currently supported when using boot2docker (see these issues), so I used a simple Vagrantfile to start up a beefy VM:

Vagrant.configure(2) do |config|

  config.vm.box = "puppetlabs/centos-6.6-64-nocm"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4096"
    vb.cpus = 8
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
  end

end

Centos 6.6 might seem like an odd choice for a development environment, but the Vagrant box was already on my laptop, so using it saved me the download time (bandwidth is a precious commodity in semi-rural Australia).

I didn’t bother setting up provisioning properly, but here are the commands that are needed to get stack working:

# Add FP Complete repo & install stack
curl -sSL https://s3.amazonaws.com/download.fpcomplete.com/centos/6/fpco.repo | sudo tee /etc/yum.repos.d/fpco.repo
sudo yum -y install stack

# Add EPEL repo and install Docker
sudo rpm -iUvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo yum -y install docker-io

# stack requires the Docker client to be able to connect
# to the daemon without root, so add vagrant user to dockerroot group
sudo gpasswd -a ${USER} dockerroot

# At this point, you'll need to log out and back in for
# the group change to take effect

# Setup iptables to allow Docker links to work
sudo iptables -t filter -A DOCKER -d 172.17.0.0/16 -i docker0 -j ACCEPT

Adding the vagrant user to dockerroot comes with all the usual security issues, but I’m not too worried about that here. There are some other suggestions in the stack documentation for how to handle this.

Running the build

So, stack and Docker are now both installed. Enabling stack’s Docker integration on the VM is as simple as modifying the ~/.stack/config file:

[vagrant@localhost ~]$ cat ~/.stack/config.yaml
docker:
  enable: true

This gives the advantage of using Docker to run the builds on the VM, while not impacting the yesod devel workflow on OS X.

By default stack will use the fpco/stack-build repository to obtain an image to run the build in. This suits me, as there is also fpco/stack-run for running the binaries once they’re compiled. The build image includes everything that’s necessary for building the whole of Stackage, whereas stack-run is more trimmed down.

Running stack build kicked off the build in the Docker container, which completed successfully as all required dependencies (e.g. libpq and pg_config) were already in the image. Thanks to FP Complete for setting up these Docker containers for everyone to use!

Packaging Docker image

stack’s support for creating container images is a relatively recent addition, and currently isn’t covered well in the documentation. There is a reference to the image configuration section here, which was enough for my needs. After added the relevant bits, my stack.yml looks something like:

resolver: lts-3.20

packages:
- '.'

extra-deps: []

flags:
  mysite:
    library-only: false
    dev: false

extra-package-dbs: []

# This isn't required as it's set in the user config on the VM
# docker:
#     enable: true

image:
  container:
    name: mdjnewman/mysite
    base: fpco/stack-run
    add:
      config: /app/config
      static: /app/static

With that in place, running stack image container produced a Docker container ready to be pushed to Docker Hub:

[vagrant@localhost vagrant]$ stack image container
Sending build context to Docker daemon 34.25 MB
Sending build context to Docker daemon
Step 0 : FROM fpco/stack-run
 ---> db9b2a858ef5
Step 1 : ADD ./ /
 ---> e682c572d7ed
Removing intermediate container 1ce35ed1ccea
Successfully built e682c572d7ed

Running a regular docker push published the image to the registry.

Deploying to Digital Ocean

After all these steps, the final package was in Docker Hub and it was just a matter of running the following commands on the droplet, and the site was live:

docker run --name=postgres -e POSTGRES_PASSWORD=$(uuidgen) -e POSTGRES_USER=postgres -d postgres:9.3

# Yes, I know UUIDs don't make the best passwords :)

docker run              \
    -d                  \
    -w /app             \
    --link postgres:postgres         \
    -p 3000:3000        \
    mdjnewman/mysite    \
    /bin/bash -c 'PGHOST=$POSTGRES_PORT_5432_TCP_ADDR PGPASS=$POSTGRES_ENV_POSTGRES_PASSWORD /usr/local/bin/mysite'

The last command is running the Yesod server, with environment variables set to the values provided by Docker (see here for info about setting configuration variables in Yesod).

The results

I’m very happy with the Yesod workflow, and stack’s Docker integration makes deploying a lot easier as I don’t have to worry about what packages are in my build and test environments. Seems like a good compromise between manually copying files around and having a full CD pipeline.

Using stack build with a LTS resolver and an isolated container is also the holy grail of repeatable builds!

There is still a lot that would need to be done (improving security, backups, logging etc) before this was closer to a production environment, but it’s good enough as a development region.

This whole process (including writing this post) from having something that I wanted to deploy to being able to view the live site took less than a day, and I was learning a lot along the way. Deciding where was the best place to host the site and waiting for Docker to pull the fpco/stack-build image took up most of the time!

Simon Peyton Jones on maintenance with strong types

2014-06-26T00:00:00+00:00

People don’t like fixing type errors, but they are a lot easier to fix than runtime errors… The single biggest thing going for static typing is not so much that it helps you write your program in the first place, but it helps you maintain your program.

GHC itself is an example – it’s now 150,000 lines of Haskell & twenty years old and I regularly refactor it in pretty large scale ways. I just wouldn’t dare do that if I didn’t have static typing to keep me sane. It’s this enormous code base, chunks of which I’ve forgotten about, and yet I can confidently make systemic changes to it because I know the type checker is going to catch all the places that change needs to go.

Simon Peyton Jones, speaking on Functional Geekery Episode 11.

The part about GHC is anecdotal evidence, to be sure, but it makes the point nicely.

Why Functional Programming Matters

2014-06-13T00:00:00+00:00

The following is the conclusion from a paper entitled ‘Why Functional Programming Matters’:

In this paper, we’ve argued that modularity is the key to successful programming. Languages that aim to improve productivity must support modular programming well. But new scope rules and mechanisms for separate compilation are not enough — modularity means more than modules. Our ability to decompose a problem into parts depends directly on our ability to glue solutions together. To support modular programming, a language must provide good glue. Functional programming languages provide two new kinds of glue — higher-order functions and lazy evaluation. Using these glues one can modularize programs in new and useful ways, and we’ve shown several examples of this. Smaller and more general modules can be reused more widely, easing subsequent programming. This explains why functional programs are so much smaller and easier to write than conventional ones. It also provides a target for functional programmers to aim at. If any part of a program is messy or complicated, the programmer should attempt to modularize it and to generalize the parts. He or she should expect to use higher-order functions and lazy evaluation as the tools for doing this.

Of course, we are not the first to point out the power and elegance of higher-order functions and lazy evaluation. For example, Turner shows how both can be used to great advantage in a program for generating chemical structures. Abelson and Sussman stress that streams (lazy lists) are a powerful tool for structuring programs. Henderson has used streams to structure functional operating systems. But perhaps we place more stress on functional programs’ modularity than previous authors.

This paper is also relevant to the present controversy over lazy evaluation. Some believe that functional languages should be lazy; others believe they should not. Some compromise and provide only lazy lists, with a special syntax for constructing them (as, for example, in Scheme). This paper provides further evidence that lazy evaluation is too important to be relegated to second-class citizenship. It is perhaps the most powerful glue functional programmers possess. One should not obstruct access to such a vital tool.

J. Hughes, “Why Functional Programming Matters,” Comput. J., vol. 32, pp. 98–107, 1989.

This paper was written twenty years ago – the more things change, the more they stay the same. I’d urge any programmer (whether currently interested in FP or otherwise) to read this paper if you haven’t already.

ThoughtWorks LevelUp EXP Brisbane – Take 2

2014-05-05T00:00:00+00:00

LevelUp EXP Brisbane Attendees (thanks Alexandra Tran for the photo!)

ThoughtWorks ran another great express LevelUp event on Saturday at their Brisbane office. LevelUp events aim to help students bridge the gap between university and their first full-time job. LevelUp EXP is a mini-conference with a number of talks and hands on sessions, as well as lots of opportunities to mingle with the ThoughtWorks employees and other attendees. One theme that recurred through almost every talk of the day was focussing on the user at every stage of the development process – if you’re not building something that your users can use, then you’re wasting time. This does mean users, not customers. You can be building a product for a client, but it’s ultimately the user’s experience that will determine success.

Real Project Example: Domino’s HTML5 in Brisbane

First up was Mark Ryall (@markryall) discussing the recent consolidation of three separate websites into a single HTML5 site for the Dominos pizza franchise in Australia, New Zealand and the Netherlands. The new site was to replace separate Flash, mobile and accessibility-focussed sites, and was a greenfield project. From a software process perspective, the project kicked off with a two-week inception where BAs, developers, designers and testers worked together for two weeks to get initial requirements worked out. Mark pointed out that this is an effective alternative to the business struggling to work out requirements on their own for six months before getting anyone else involved. It reminded me of the quote that’s often attributed to Henry Ford:

If I’d asked people what they wanted, they would have said a faster horse.

The point is that the business doesn’t always know what they want, and even if they do, they may not be able to articulate that in such a way that both parties have the same vision. The rest of the project ran in two-week iterations, with showcases at the end of every iteration. Mark mentioned that the biggest challenges (from a technical perspective) in this project were internationalisation (I18N) and localisation. As Mark explained it, internationalisation ensures everything is available in the correct languages for the correct region, whereas localisation focusses on differences such as different address formats (another example of what this site in particular has to handle is that Domino’s in NL doesn’t provide catering, like it does in AU). There was a heavy emphasis on test automation in this project, and Selenium WebDriver was used for automated browser testing. WebDriver allows single tests to be written to target more than one browser, and are more robust than trying to guess what the browser is doing and interspersing your tests with loads of sleep() calls. In order to make tests more readable for everyone on the project (BAs/the business included), Specflow was used. Specflow is a .NET alternative to Cucumber, and uses a similar declarative style to define specifications & expected behaviour. Tests are then generated from these easy to read specifications, targeting whichever testing framework you’re using (NUnit in this case). Given this site was replacing an existing site with a focus on accessibility, Domino’s required this site to be just as accessible for people using screen readers and keyboard only navigation. Accessibility is something that is easy to overlook, but every developer should make the effort to do these things correctly (using alternative text, ARIA tags, skip links for navigation etc). If you want to know more about how important accessibility is, Scott Hanselman recently did an interesting interview with Katherine Moss, a blind software technician.

The Lego Game

The Lego Game is a game designed to give you a taste of the Agile software development process. Our task was building a Lego ‘animal’ with a client in mini sprints lasting a few minutes each. We worked through everything from estimation to a retrospective in less than ten minutes, and completed three iterations with a client who wasn’t afraid to change their mind. An interesting resource that was mentioned by the ThoughtWorks employees running the Lego game was the Retrospective Prime Directive by Norman Keith:

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

I thought this was a good guideline for running retrospectives, as it ensures that you focus on what went well and what can be improved, and not on trying to assign blame. There are plenty of pages describing the agile Lego game on the net if you want to read more.

Agile 101 with a Business Analyst

Agile 101 with a BA by Henry Lawson was the first of the twenty-minute lightning talks for the day. In twenty minutes, Henry managed to cover a brief history of Agile, the Agile manifesto and different Agile methodologies. There was more about Agile on the cards but the twenty-minute timeline had to be adhered to! The following paragraphs sum up the main points of the talk.

Agile manifesto: the Agile manifesto is what gave the movement towards iterative, lightweight processes a name. Written in 2001 by several prominent figures in the software industry, the core of the Agile manifesto is the following four points:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The differences between traditional Waterfall methodologies and Agile: The waterfall model is a sequential design process, where each stage is completed and locked-down before the next is started.

Waterfall model of system development – avoid it! (you want the cycle above to take two weeks, not twelve months) (source)

While waterfall might work for other engineering projects (like building a bridge), it’s not really suitable for software development (too many software projects have elements of a wicked problem). I won’t go into the benefits of Agile or Agile processes in too much detail here, as there is plenty more information available.

Different agile methodologies: Examples of agile methodologies include Scrum, Kanban, Lean, XP and Crystal Clear. The main difference between these methodologies is the practices, such as daily Scrum meetings or pair programming in XP. The general processes are similar, and there is no reason you can’t mix and match.

Functional Programming

Hugo Firth (@hugo_firth) was giving the next lightning talk, this time with a quick intro to functional programming concepts. There are a number of programming paradigms, such as procedural and object-oriented (OO). Procedural code (e.g. C/BASIC code) doesn’t provide many ways to create abstractions and manage state. OO, such as you’ve probably encountered in Java/C#/C++ allows abstractions to hide global state. Functional programming on the other hand is all about removing state where possible, and programming with pure functions. Hugo went through some common functions you’ll meet in functional programming, such as map & filter by providing some simple examples in Clojure. If you want to see a less rushed presentation by Hugo, his recent talk at the Brisbane Functional Programming Group is available on Vimeo.

BFPG is a great community if you’re looking to learn more about functional programming – I’ve always found everyone there to be extremely willing to help.

Working in different environments… from Heaven to Hell

The last lightning talk in the first session was by Claudia Ferreira (@claudia_onfire), providing some practical advice for dealing with difficult work environments. You shouldn’t assume that the end of uni is the end of the hard work. Starting a new job, meeting new people and adapting to company culture is just as hard. Also, you don’t want to fall into the trap of thinking that you know everything and that you can stop learning. There is an almost endless variety of work environments, some good and some not so great. Claudia showed the following clip from The Matrix and told us about a time she was in this exact position.

Claudia’s boss basically said you’re here to work, not think, and there are hundreds of people who can fill this spot if you aren’t happy with that. Hopefully you won’t find yourself in a situation this extreme, but chances are you’ll have a boss you’re not overly excited about working with at some point in your career. If you find yourself in this situation, don’t just bottle it up and say nothing. Your team is there to help, you should be able to trust them and speak to them about issues. It’s important to have a good relationship with your team, as they’re the people that you spend the majority of your day with. So don’t stress if you’re confronted with a difficult situation, just take a deep breath, speak up if you really don’t agree and be the bigger person.

The Way of the Consultant

Sarbashrestha Panda has been a consultant for 5-6 years, working as an analyst and project manager on many projects. There are many types of consulting, from technical to business strategy. Technical consulting still has a strong focus on the business needs, no matter what area you’re in you are helping solve a problem the business is having. As Panda explained it, the role of the consultant is to always ask the difficult questions and have the difficult conversations. You want to make a problem visible, make people think and help to implement solutions. As a consultant, your knowledge should drive change.

Most problems are people problems - recognise that everyone has a different perspective (source)

If you’re working as a consultant, embrace the opportunity to work on many different projects – you can’t develop a broad skill set without experience. You can be a consultant in any language with any technology, and after a few years should develop the skills to become a consultant for a specific domain. Don’t let yourself get cornered into a specific technology – technologies change far too fast.

UI Dev. What and Why?

Eru Penkman gave the next talk, which was originally entitled ‘UI Dev. What and Why?’, but seemed to have transformed into a talk about programmer culture and day-to-day life. Eru pointed out a bunch of things that a lot of programmers/developers have in common. Most of the people in the room had read XKCD at some point, everyone had used Stack Overflow, lots of people read similar blogs etc. Having all these elements in common helps when working in a team, which as Eru pointed out is pretty much what development is about. It’s very rare to have a project where you don’t interact with a number of people throughout the day. There are plenty of resources for developers, as everyone uses Stack Overflow it means that SO can answer almost any question, and lots of problems you might want to solve have already been solved by one open source project or another. However, don’t just consume these resources – make accounts on SO/GitHub and start contributing!

What they didn’t tell you about Testing!

Do not assume anything. Ever.

The final lightning talk for the day was by Leonor Salazar & Brian Mason, covering testing and quality. The first point that Leo & Brian drilled home is that quality is subjective. Everyone has different ideas of quality, and you can’t do much meaningful testing without knowing what quality means for your project. Quality as defined by the users of the product is the most important. Everyone on a project should be thinking about quality, it’s not something that one person can ‘add in’ at the end. There is also a simple and compelling economic argument for having quality on everyone’s mind: it’s up to 1000x times more expensive to fix a bug in production than it is in analysis. Don’t let that cripple you or force you into a big-design-up-front type approach, but keep quality at the front of your mind.

Development is solving problems, testing is asking questions.

You need to ask yourself four questions:

Are we doing the right things?
Are we doing the things right?
Are we getting them done well?
Are we getting the benefits?

There is no point obsessing over whether you’re doing things right if you’re not doing the right things! Testers also have the opportunity to improve process, and take the product in new directions.

User Testing what you Build – Hands On Session

Carrying on with the theme of user first was Pete Chemsripong, with a hands on session where each team had to develop a ‘product’ that was then tested by a user. For example, a media remote control for a user with boxing gloves on or a lift control panel for someone carrying 15 bags of groceries. To design a good user experience, you need to ask why you’re building anything in the first place. You need to understand why people would use this product. Ultimately the answer to a question like ‘What makes Google better than Bing?’ is always that it offers a better user experience and satisfies the users needs. It doesn’t matter what you build or how you build it if people don’t enjoy using it. User testing also doesn’t have to be expensive. You can even grab a dev from another team who hasn’t touched your project. Chances are they’re going to find a lot of the same issues as a ‘real’ user of the software, allowing you to pick the low hanging fruit for minimal cost. Pete showed a video from a recent ThoughtWorks experiment, where an in-store innovation lab at a Woolworths store produced loads of great feedback in a very short time. The other ThoughtWorkers present mentioned Usability Testing with Morae as a good read for more information about inexpensive usability testing.

TDD – from woe to go in 10 minutes – Hands On Session

As professional developers, the burden of proving a system works lies with us.

Steve Morris held the floor for the final session of the day, and did a great job showing everyone how simple test driven development can be. TDD provides the following benefits:

Shows the code does what you say it does
Improves quality by making code more testable
Allows you to figure out what to test only once – no more writing a monolithic class/function and then spending time working through what needs to be tested over and over
At any time, you can see what is broken and whether your code is shippable (as there is never code that isn’t covered by tests)

TDD is coding with intent – don’t code by accident.

The ‘test’ in TDD could just as easily be replaced by ‘expectation’, ‘behaviour’, ‘specification’ or ‘intent’. It’s as much about helping to define expected behaviour and a minimal specification as it is about testing actual code. It doesn’t matter if you’re a developer or not, TDD provides the same benefits to BAs, managers and QA testers. I couldn’t write this post without including the TDD mantra, lest Steve think we all learnt nothing at all from the talk. The TDD mantra is as follows

Red – write a minimal failing test
Green – write the minimum amount of code to make the test pass
Refactor – incrementally refactor the code (both code under test and the tests themselves – test code is first class code!)

Writing tests before code forces you to think about naming, architecture, APIs and design without considering implementation details. This helps with design and forces us to progress with intent, to avoid building unnecessary features. The refactor step encourages design improvements in incremental steps, rather than writing a bunch of code and then struggling through a ‘big bang’ refactor later in the project. Failing tests are not a failure of any kind, rather, they are statements of intent. The minimal failing test doesn’t even have to compile. A test that doesn’t compile signifies an intent to write the code to make it compile. In this way tests are documentation of future intent as well as progress to date. The hands on component of this talk included getting testing environments for C# and Node.js set up and writing our first code using TDD. Set-up for C# is as simple as installation of a testing framework like NUnit, which in a newer version of Visual Studio is as easy as running

Install-Package Nunit

in the package manager console. The set up for Node.js was just as simple, with npm installed simply run

npm install karma
npm install phantomjs
npm install jasmine

from your favourite shell. In the talk, it was possible to get this environment set up and have our first test running in under ten minutes for each environment. There really isn’t any excuse for not giving TDD a try! Steve also pointed out that TDD (like most concepts in software development) isn’t as new as you might think.

The second LevelUp EXP event in Brisbane was eye-opening, and I think the attendees learnt a lot to complement what is taught at university. If you attended/presented and I’ve omitted anything, please let me know and I’ll fix it up. My post about the first Brisbane LevelUp event is here, and more about ThoughtWorks LevelUp can be found on Twitter (@TW_LevelUp) and Facebook.

Working on Haskell code in vim

2014-04-17T00:00:00+00:00

I decided to get a fresh vim install properly set up for some Haskell development, and went down the road of using hdevtools, vim-hdevtools and Syntastic. Below are notes on some issues I encountered setting up vim-hdevtools.

The instructions on the GitHub page for hdevtools & vim-hdevtools linked above are simple enough, but I ran into a couple of issues. I’m using the sandbox feature added in Cabal 1.18, and by default hdevtools doesn’t know to look for a .cabal-sandbox folder. So, when I tried to use any of the hdevtools commands, I received an error message similar to the following:

Error loading targets
(file):
Could not find module `(module)'
Use -v to see a list of the files searched for.

This is an issue that a few people have run into, and I found a link to this solution in the archives of the #Haskell IRC channel. Adding it to my .vimrc seems to have fixed this error, and I can now use all the hdevtool commands in vim. I’m still having issues with *.cabal files that specify default-extensions (vim-hdevtools doesn’t seem to pick these up).

Also, if you get an error message similar to the following:

E492: Not an editor command: ...

make sure that hdevtools is on your path and that you have plugin files for specific file types enabled by adding

filetype plugin on

to your .vimrc.

Installing self signed certificates on Android (Jelly Bean)

2013-11-06T00:00:00+00:00

I was pretty happy when I found out Android now supports installing self signed certificates, but had some trouble getting the certificate I use for administering this blog installed.

So, a quick note for anyone who is attempting to install a self signed SSL certificate on Jelly Bean Android devices: you can only install a root CA certificate (i.e. a certificate with “Subject Type=CA”).

Trying to install a certificate for a single site (i.e. a certificate with “Subject Type=End Entity”) isn’t possible as far as I can tell. I don’t have any references for this, it’s just based from my experience, so let me know if you’ve experienced otherwise.

Two-factor authentication and ‘trusted’ devices

2013-10-30T00:00:00+00:00

“I challenged hackers to investigate me and what they found out is chilling” is an interesting article – it’s the story of a couple who challenged a hacking team to pen-test their digital lives. To give some context to the following quote, their laptop has been compromised and user ID and password for their on-line banking account at Chase has been stolen by the ethical hacking team:

Chase.com uses a two-step verification system, which momentarily stymied SpiderLabs’ hackers. Every time she or I logs on from an IP address that Chase doesn’t recognize, it offers to send us an activation code via text to our mobile phones. But a search of Charlotte’s hard drive revealed Chase cookies, which the team copied and used to convince Chase that she was logging in from home.

You could argue that if your PC has been this thoroughly compromised then you have bigger problems, but it still points out how limited some two-factor authentication systems can be if you allow devices to be remembered/trusted.

ThoughtWorks LevelUp EXP Brisbane

2013-10-29T00:00:00+00:00

On Saturday I was lucky enough to attend ThoughtWorks LevelUp EXP at the ThoughtWorks Brisbane office.

As well as plenty of opportunities to chat with the ThoughtWorkers (and Chad!) who kindly gave up their Saturday, there were six presentations throughout the day:

Opening Keynote: Design Thinking

Ben Melbourne (@benmelb) set out to teach us about design thinking, and how it’s important for any project. The main points that I took away from his talk were:

Visual design is only a small part of design.
UX research is not strictly academic or scientific – iterate quickly and rely on feedback to get the best results in the shortest time.
Embrace constraints to get the most out of a situation. Design is harder when you have a green field to work in – if you have constraints, don’t fight against them, use them for inspiration.
Everyone is a designer – if you’ve ever been on a project without a designer, it was you.
Always ask if the current approach is the best use of business capability.

Functional Programming for Fun and Profit

Next up was James Ottoway (@jms_) with a good intro to functional programming. He started out by explaining declarative languages and reactive programming, as well as DSLs and closures and how they can be used to cut complexity.

Declarative languages focus on expressing the logic of a computation, not how to carry out that computation. Regular expressions, Chef/Puppet configuration files, and SQL were all used as examples of declarative languages. In James’s words:

We’re expressing what, not how. And that’s a good thing.

Expressing what you want to do (and not how it must be done) frees up the compiler/interpreter to make decisions about implementation at runtime, and eases the burden on you as a programmer.

When talking about reactive programming, James used Excel (everyone’s favourite spreadsheet application) as an example of a reactive program. This is a really simple analogy which I think helped everyone grasp the idea of reactive programming quickly. He also pointed out that the windows GitHub client is built using reactive programming (using ReactiveUI from what I can see).

As someone who’s trying to learn Haskell at the moment, this section about of this talk about closures was interesting to me. Closures aren’t something that I was seeing in Haskell, perhaps because I’m still battling with unfamiliar syntax (turns out they’re pretty much everywhere). A closure is just a function together with a referencing environment, and they’re pretty essential for functional programming.

Business Analysis

After lunch we heard about the life of a BA from Cathie Hagen (@catherist) and Jiangmei Kang. BAs are an integral part of every stage of a project, and use their place as a bridge between the client and the developers to ensure that value is delivered to the client. The following is a photo I took of the board used to explain the many aspects of work as a BA, I think it says more than I can:

(Sorry about the poor quality – the projector turned on for the next talk just as I was taking this picture)

User Testing

Why build something that people don’t want to use? It’s 2013, doing that is a waste of humanities potential.

Pete Chemsripong gave a great talk about the importance of user testing. Speed is important for user testing and ideally should be a daily thing. Every feature that you’re working on in a project should be validated by a user (this agrees with what Ben said in the design talk).

Helping you get your first job

The last presentation before the keynote was by Jean D’amore (@jeandamore) with an entertaining talk on how to land your first job in IT. The talk introduced job hunting as a game where you can’t control everything, but what you can control can make all the difference. He split the journey into seven steps, each accompanied by a great movie reference & quote. Hopefully the slides will be made available as I didn’t get everything down, but what follows is my summary:

Yourself (The Last Samurai)
- Improve the environment by working on yourself.
- Do: know your boundaries, be versatile, stretch yourself, maintain balance, listen, commit to projects, be willing to sacrifice, learn/do new things
- Don’t: underestimate yourself, thing you’re alone, deny the challenge, limit the horizon
The Search (An Unexpected Journey)
- The search will change you
- Do: know what you seek, choose a domain, know the employer, ask about graduate programs/training, keep your LinkedIn (and internet presence in general) up to date, favour simple job ads (i.e. not ads that list every skill under then sun – you often need to be a specialist before you can generalise)
- Don’t: dismiss too quickly, use recruitment agencies, work on projects you don’t agree with (everything you do has an impact and will impact you), waste too much time searching on-line (do spend more time talking to people)
Your CV (The Colour of Money)
- Do: PDF it!, make it unique, know what’s on it, web design it (use web design principles – e.g. readability, don’t cram everything in), make it one page, get friends to read it, start with skills over education
- Don’t: lie/exaggerate, rush it, use ‘bullshit bingo’ words (except in a few cases where it’s in the ad – in this case, always explain and give examples)
Applying (Close Encounters of the Third Kind)
- You need to understand and be understood
- Do: strike with precision, add a cover letter, apply to people (not computers), customise your CV, wait, follow-up, be organised
- Don’t: bulk apply (do apply to at most 5 organisations at once – it becomes too hard to keep track of what you’re doing otherwise)
Interviews (Gattaca)
- Little things are often the most important. For example, be good to everyone at the company.
- Do: look clean and tidy, research the company & their interview process, arrive early, know who you are and why you’re there, know what you’ll be asked, listen, ask relevant questions, be humble and curious, know your stuff (especially your résumé)
- Don’t: be afraid to ask for clarification, lower your guard until you’re clear of the building (i.e. do treat every moment you’re in/near their offices as part of the interview), ask how you did at the interview, ask about time/money when you’re after your first job, leave your phone on, have alcohol (even if offered), go if you’re sick (you need to be 100%, and the interviewers will understand)
Hearing Yes/No (The Hurricane)
- Don’t be afraid of failing and keep bouncing back
- Do: take your chance to explore/re-evaluate, take risks, trust your instinct, understand that this is one step is a career
- Don’t: burn bridges, take it personally, rush to accept, say you’re waiting on another offer
First Day (City Hall)
- Make a difference and work where you’re satisfied. Focus on employers that give you the upper parts of Maslow’s hierarchy of needs
- Do: learn names, go to social events, pair, do what you’re told, be helpful, learn the ropes step by step, leave if you don’t like it after a month of so
- Don’t: panic, try to change things quickly, cold shoulder anyone

Closing Keynote: Who are you, where are you going, and how will you get there?

The closing keynote by Chad Renando (@ChadRenando) followed on nicely from the presentation from Jean, and was really a step back, focussing on making the most of who you are, being clear about where you are going, and being intentional about how you can get there.

Every man invents a story for himself – which he often, and with great cost to himself – takes to be his life.

Chad used this quote from Max Frisch as a way of explaining that you create your own story, but you should never think that it can’t be changed or even completely rewritten.

You need to understand yourself to understand how you will work with others – are you more practical or creative? Do you approach things in an analytical way or listen to your beliefs? Are you structured or flexible? Learn what your signature strengths are and use those to create a mission statement, as your strengths are the things that make you feel alive. At the same time, look out for the strengths that you rank poorly in.

Create goals (what is your 9-day plan? your 120-day plan?) that align with your strengths, but don’t think that these goals are set in stone. Use goals as a way to direct yourself, not as measures of success. This allows you to iterate on your goals and improve them as you learn more about yourself.

Chad finished the main part of his talk with his axioms from life in a digital studio:

Ask questions
Focus on resolving process conflict, not relationship conflict
There are things that are understandable, but not acceptable – don’t get the two confused
Everyone is generally doing the best they can
There is no fault, only cause
Every conversation we have is with ourselves, and sometimes they involve others
If you want to lead others, lead yourself

The day covered a range of topics and gave me a lot to think about. One point that appeared a couple of times throughout the day (and which Chad reiterated at the end of his keynote) is how important it is to get involved with your community. It allows you to grow your network, get references and learn. I can’t agree more, I have learnt so much and met lots of interesting people by attending the YOW! conference, YOW! nights, and the BFPG & Brisbane GPU Users meetups (there are tons of similar user groups in Brisbane – I guarantee you won’t regret attending one of their meetups!). Another way to get involved in the broader community while sharpening your skills is volunteering – there are plenty of non-profits that could use help with IT. Or, if you have an idea/are working on a start-up, River City Labs is building an amazing community of like-minded people.

Thanks again to ThoughtWorks for running this mini-conference – you’ve done a great job of inspiring this IT undergrad! More about ThoughtWorks LevelUp can be found on Twitter (@TW_LevelUp) and Facebook.

Kevlin Henney on code comments

2013-10-16T00:00:00+00:00

A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments.

Kevlin Henney (original tweet here).

There are some interesting responses, pointing out that comments are great for references and in-line diagrams, and sometimes necessary for code that has been restructured to improve performance (although creating difficult to read code solely for performance reasons is a separate issue).

John Carmack on type systems

2013-10-04T00:00:00+00:00

It’s just amazing how many mistakes [are made] and how bad programmers can be. Everything that is syntactically legal, that the compiler will accept, will eventually wind up in your code base. And that is why I think static typing is so valuable.

John Carmack (full talk here – note this link jumps to this quote – part one of the keynote is here).

Pretty much sums up how I feel about the static v. dynamic debate for anything other than very short-lived prototypes.

Don’t fear the Monad

2013-10-03T00:00:00+00:00

I recently watched Brian Beckman’s Don’t fear the monad talk (YouTube version here) and thought it was a really good explanation of functions, monoids & monads and why they’re a useful way to master complexity.

I recommend it to anyone who’s starting to look at functional programming. Here are some of the key points I took away from the talk:

Functions are data
- A function takes an argument(/s) and returns a value, and always returns the same result for the same inputs
- Think of a function as a table lookup – the table is the data
Functional programming is a style that can be used in almost any language
- But some languages make it easier than others
- First class functions (or at least lamdbas) are a must have
Complexity is killing software developers – we need to control it
- The best way to control complexity is to put functions into groups where things are disciplined, and where they can be composed in arbitrary ways
- Build complexity out of simplicity by composing things
A monad adds some data to a value, and the implementation of the bind operator defines how that data interacts
If you want to add a feature, you shouldn’t even have to know what you’re going to interact with
- Using monads means the interaction is already built into the structure
You don’t need to know what a monad does
- So long as it follows the rules, it can have arbitrary side effects

Another interesting part was Brian’s reasoning about why functional programming is a ‘hard sell’:

Immediately upon the foundation of computing, the world split into two camps – the bottom up camp and the top down camp.

The bottom up camp is typified by Turing machines – “let’s start with the hardware and add abstraction as we need it to get close to the mathematics, but never at the expense of performance”. That gave rise to FORTRAN, C, C++, Pascal, Java, C# … all of those languages are in that tradition.

The top down crowd says “let’s start with the mathematics and subtract abstractions to get down to the machine, sometimes at the expense of performance”. That tradition starts with lambda calculus … and only subtracts abstraction when it is convenient to do so. That’s the tradition that gave rise to ALGOL, Lisp, Smalltalk, ML, Haskell and all these languages that have reputations of being slow.

As most programmers are familiar with the bottom up way of doing things and the two approaches come from very different origins, it can be hard to convince a bottom up programmer of the benefits of the ‘top down’ approach. It can be even harder to pick up functional programming once you’ve resigned to giving it a try.

[The top-down crowd] are saying “don’t go to the hardware so early, you’ll lock yourself into irreversible design choices”. [The bottom-up crowd] are saying “don’t give me all this math, I can’t handle garbage collection, or function calls or whatever it is that’s getting in the way of my performance, I need every ounce of performance the hardware can give me. And I’m willing to write software into the wee hours of the morning to get it”.

The last sentence here is what stands out to me – as software gets more complex, it seems that bottom up programming is going to require more and more time working in the wee hours of the morning to get the desired performance and functionality.

Maybe Haskell …

2013-09-14T00:00:00+00:00

After reading Learn You a Haskell on and off over the past fortnight and watching a couple of Erik Meijer’s Functional Programming Fundamentals, I finally got around to installing the Haskell platform and firing up GHCi.

After a couple of hours of implementing the simple functions scattered throughout LYAH, I was up to implementing quick sort. Given the number of times I’ve seen this example, I’m not going to reproduce it here, but it’s pretty amazing ¹.

One of the main things I noticed was that after two hours of writing code in a language I’d never used, I’d produced zero run time errors. There were plenty of compile errors, but that’s to be expected – especially given I was using an editor that was inserting tabs in a language with significant white space (which, by the way, makes for fun times when unfamiliar with Haskell compiler errors). I know what kind of error I would prefer to deal with.

So far I’m liking Haskell, and I think it’s a huge shame that there is only one functional programming course at my university (COMP1024), which isn’t available to most dual degree students.

Edit: COMP1024 hasn’t been offered since 2010.

1. I know this isn’t ‘true’, in-place, high performance quick sort, but in how many other languages can you write a readable, generic sorting algorithm in 6 lines of code?

VirtualBox + Dell laptop VT-x issues

2013-08-15T00:00:00+00:00

I recently had to use a VM for a uni project, and needed to have access on my laptop & desktop. I haven’t used a VM on my laptop in a while, but didn’t expect to see the following error message from VirtualBox (4.2.16):

VT-x features locked or unavailable in MSR.

I’ve used x64 hosts before with no problems, but I guess my BIOS settings were reverted for some reason (using a Dell laptop with Intel PM45 chipset).

So thinking to myself that this should be an easy fix, I rebooted, enabled VT-x and restarted, but I received the same error message. No amount of changing BIOS/VirtualBox settings, rebooting or otherwise reconfiguring would make this message go away.

I eventually read that it’s sometimes necessary to power down and physically disconnect the machine from power to properly enable VT-x. This solved my problem – I decided to make a note of it here to hopefully help someone else in the same situation.

Ubuntu 12.10 – Transmission fails to connect to HTTPS trackers

2013-01-10T00:00:00+00:00

Transmission, the BitTorrent client installed by default in Ubuntu seems to have an issue with connecting to trackers using HTTPS at the moment.

After Transmission failed to connect to a tracker using HTTPS, I went searching and found this bug report.

The steps there are pretty simple and worked well for me:

Close Transmission
Download the .deb for updated libgnutls26 from here (bottom of page – choose your architecture then mirror)
Install using
```
 dpkg -i (package name).deb
```
Restart Transmission

UPDATE: It looks like this is now fixed, so running

sudo apt-get update
sudo apt-get upgrade

should get it working again for you.