Multi-Container Deployment with Docker Compose

docker_future

Why Docker?

At thinkWhere we always aim to keep pace with latest tech-industry trends which is easier said than done in such a fast paced sector! However one unavoidable technology trend we’re now employing across our application deployment model is Docker. Docker is a software containerisation platform guaranteeing that software will always run the same, regardless of it’s environment. Docker offers many benefits over traditional application deployment, including:

Simplicity – Once an application is Dockerized, you have full control (start, stop, restart, etc) with only half a dozen commands. As these are generic Docker commands, it is easy for anyone unfamiliar with the specifics of an application to get started.

It’s already Dockerized Docker Hub is the central marketplace for Docker images to be shared with other Docker users. Often you find official Docker images for an application already exist or you can find another users efforts to build upon. Docker images we have used include MapFish Print and Nginx. We have also containerised our own flavours of MapProxy and GeoServer.

Blueprint of application configuration – A Dockerfile provides the blueprint or instructions to build an application. This can be stored in source control and refined overtime to improve the build. It also removes any ambiguity of build/configuration differences between various deployments.

suHPcQ

Rapid to deploy – Having this blueprint for each application means that all we need is a server or virtual machine with Docker engine installed. This has drastically reduced the time spent deploying and configuring our applications.

Plays nicely with continuous integration – Amazon Web Services offer services dedicated to deploying and managing containerised applications. We recently constructed a Shippable Pipeline which builds and pushes new images to the Docker Hub repository as changes are merged into a code base. These new images are pulled down by the Amazon Elastic Container Service and deployed seamlessly. The ‘throw away’ nature of Docker lends itself to scalability, therefore services such as this can be scaled up or down with just a few clicks of the mouse.

Why Compose?

These days its rare to deploy applications which exist in a completely standalone context without the need to communicate with at least one other application or service. If all these applications happen to be Dockerized then Docker Compose is a great tool to create a multi-container application. In a nutshell, Compose lets us start/stop/build a cluster of related Docker containers.

The Compose File

In addition to the existing Dockerfile for each image, a single docker-compose.yaml file must be created for a Compose project. It’s here we define how the various containers will behave and communicate with each other.

This example compose file is from a Flask-restful application I wrote to serve GB Postcode GeoJSON from MongoDB. You can see it working and try your own postcode here.

This Compose configuration comprises three containers – a Python web application which talks to a Mongo database all sitting behind an Nginx web server.

web:
  restart: always
  build: ./web
  expose:
    - "8000"
  volumes:
    - /usr/src/app/static
  env_file: .env
  links:
    - db
  command: /usr/local/bin/gunicorn -w 2 -b :8000 app:app

nginx:
  restart: always
  build: ./nginx/
  ports:
    - "80:80"
  volumes:
    - /www/static
  volumes_from:
    - web
  links:
    - web:web

db:
  restart: always
  build: ./mongo/
  ports:
    - "27017:27017"
  volumes:
    - /home/matt/postcode_search/mongo:/data/db

This Compose file is largely self explanatory as you can see the three container configurations (web, db, nginx) are defined separately.

Some of the entries in the example file above will be familiar to Docker users, such as mapping a volume or forwarding ports to the host. The Compose file is best used for setting some of these parameters which may have previously been configured when starting a single container with the ‘docker run’ command. This is because a single command is used to start all the containers within a Compose cluster, rather than starting them individually.

In order to allow communication between containers a ‘links’ entry is used. You can see that the ‘web’ container will be linked to the ‘db’ container. The ‘links’ entry is used in the same way as the ‘–link’ option for ‘docker run’. When the Compose cluster is started the links entries are used to determine the startup order of the containers.

Another unfamiliar entry may be ‘volumes_from’. As you may have guessed this simply mounts all the volumes from another container. In this case the Nginx container needs visibility of the static files from the Python application.

So to bring up the application we simple use the ‘docker compose up’ command. With this single command we can build (if required) and start our three containers. Easy!

d79025b6d689c4eb240b70f84a3c3c94eb1f753378709ce1ab57f0938d8a93a1

 

Analysing Placenames in QGIS

We meet many different people trying to do different things here at thinkWhere and one of our attendees at our QGIS Skills of Analysis & Statistics was particularly interested in looking at place names. Place names are a really interesting way of telling what kind of history a place has had. The Island of Jura is a great example of this deriving from an old Norse word meaning Animal Island. Common Scottish names such as Tarbert and Ben Mor are Gaelic derived place names translating as a narrow strip of land or a big hill respectively.

While we often wonder about the originality of our forbears in how they choose to name things, these are the footprints of history in maps we use every day. They teach us about where we come from and how the peoples who come before us lived with their landscape. It’s very romantic and also a terrific thing for some GIS analysis.

QGIS has some nice ways of helping with this kind of thing, alongside OpenStreetMap which is a rich source of place name data. The Ordnance Survey Open Names dataset would also work well for this kind of analysis.

Duns & Dums

A common word found in Scottish place names is “Dun” meaning a hill fort or roundhouse in the Celtic tradition. This is sometimes rendered as “Dun” (as in Dundee or Dunfermline) or as “Dum” (As in Dumfries or Dumbarton). Less common in Scotland, although still prevalent in Ireland is the prefix “Don” (As in Donegal), however this gets confused in Scotland due to the River Don. Don in Scots Gaelic is more associated geographically with the Celtic river goddess.

In QGIS we can use the expression:

“name” ILIKE ‘%dun%’ OR “name” ILIKE ‘%dum%’

To select or filter for any words that contain those strings of letters.

map_of_dun

The ILIKE function (a case-insensitive LIKE) is a great way of doing these kind of analyses and you can use the “%” wildcard to make selections in almost any context.

With a spot of good symbology and the use of the Print Composer to show multiple maps on the same page. You can show the distribution of a few different place name components. Having consistent maps next to each other on a page like this helps to show distribution.

placenames

A really effective way of showing this kind of data can be to use the heatmap. The last few versions of QGIS have put this into the standard symbology tool which allows you to generate them on the fly without the need to create a whole new raster. This makes it much easier. This is what the distribution of “dun”/”dum” components looks like.

dun_heatmap

The big problem with this kind of analysis is reliance on the data. The OpenStreetMap relies on users adding in data themselves and that makes an analysis of this nature a bit shaky with crowd sourced data.

The OS Open Names dataset would be a great way of doing that, but it is a huge dataset that would need a database set up for it. We do cover database creation in our advanced QGIS course, but it would be a good follow up.

Check out our QGIS training offerings at: thinkWhere.com

Apprenticeship at thinkWhere

My name is Jack, and I’ve been an apprentice at thinkWhere for 8 months. During my time here I’ve got to be part of the whole agile experience, as thinkWhere recently transitioned over from the traditional waterfall development. At first, it was overwhelming with terminology, scrum, sprint, kanban, story etc.

jack_face

But over time I came to realise I’m not playing rugby, and these terms actually make a difference in development. The experience so far has been invaluable, I’ve discovered the monotony fixing syntax errors and satisfaction of something working. If there’s anything I have learned from working here, it’s that just because there is a goalie, doesn’t mean you can’t score, in other words there is always solution, no matter the problem.

The teams I have worked with have always been helpful towards my development, gradually building up my confidence to work with bigger and better projects. As my time comes to a close, my experience at thinkWhere has been a necessary stepping stone in moving out into the world. I think I’ll remember my experience here, it’s been one big push into the working world, and I’ve gained a lot of new skills from my time being here.

Projects:

Directory Storage Checker:

This was for the Support team (who have to keep customer storage statistics in check), in order to check a client’s current storage usage. I used PHP to write it all, developing it on one of our test servers and using PhpMyAdmin to print database results and to produce a history list.

image2

Automated Reporting:

One the later projects that I have worked on. The idea is to create a more sophisticated and automated approach to displaying customer data, in this case it was to create a portal which displayed all the necessary information about a client’s usage statistics.

Still a work in progress, but it will contain necessary information such as how much memory they are using, how much support they are entitled to via the phone etc. This not only speeds up our work efficiency, as there is less processing that needs to be done in the monthly reports, but also allows the customers more visibility of what they are using.

I built this system using PHP, JavaScript, google docs’ charts and PhpMyAdmin. I have enjoyed working on this project because it has a diverse language range that can be used, meaning there is a lot more flexibility in creativity to how the information is displayed.

image3

Server Storage Monitor:

The most well received project was one I did on the storage reports. This application will run a batch file which reports the storage available in each server. This is incredibly useful as there are many different drives across different servers that need to be regularly checked. These are checked in real-time to ensure there is accuracy before making any appropriate decisions. This application will daily save the current storage into a CSV in case they need to be reviewed at any point. There is also a summary which produces an overlay summarising any discrepancies in storage.

This application was built on php and batch scripts. This project required a lot of research in order to gain a better understanding of how to use scripts in conjunction with php.

image4