Counting Points and the Process Modeller in QGIS

Here at thinkWhere we’ve recently been working with the automation tools in QGIS. The processing toolbox is a much underrated feature in QGIS, lacking the jazzy graphics of the ArcGIS equivalent it is, however, just as useful and functional from the point of view of doing multi-stage GIS analysis.

We recently had occasion to create a tool for a client that counted the number of features within a polygon and provide statistics for subsets of the features. Let’s try this again, for the sake of demonstration and talk about trees.

map1

Here we have a series of suburbs arranged as polygons and trees as points. As you can see there are 5 types of tree in the dataset. A Spatialite or PostGIS database would make quick work of this, but you can also use the QGIS processing toolbox to iterate through this and count each aspect of the data.

Effectively from the point dataset we need to extract each of the attributes.

Extract by Attribute from Trees  “Type = Beech” etc.

extract-by-attribute-setup

This creates effectively a subset of the input points dataset which can then be counted via the polygons.

We can then use the count points in polygon to count the number of points that sit inside each of the neighbourhoods.

Count Points in Polygon From Beech Subset of Input Points by Input Polygon, Create field “Beech”

count-points-in-polygon-setup

Using standard QGIS this will create a new dataset each time you run an algorithm, but the great strength of the processing modeller is that these intermediate steps can run in the background as temporary layers, rather than confusing your users with multiple layers.

Of course what needs to happen now is that the next iteration (Birch) will need to be counted using the polygon dataset you created for the counts in the Beech dataset. If you think about it, the geography of that dataset is the same as the Input Polygon (the neighbourhoods). It also carries forward all the attributes for the layer as well. So for each iteration you are not using the original input polygons, but adding the new attribute information on for each count.

In the processing toolbox this looks like this:

modelimage

When you run the model, you get a polygon shapefile with a new set of attributes showing the number of each of the tree types in the area. You can then generate graphs or use the data for other purposes.

treecounts

This is only one of the many ways that the automation tools in QGIS can help you. It is also a really useful system for analysis. QGIS is hugely flexible and has a load of options for this. To learn more and see details of our QGIS training offerings please see our training page.

Infinite QGIS plugin repositories in the cloud

[…we found it useful to make each version of the code available for installation and testing via their own QGIS repository…]

Here at thinkWhere we’ve recently released roadNet, a tool for managing the spatial database of road layouts, roadworks and roadside assets that local authorities use to create local street gazetteers and to plan maintenance and closures.  roadNet runs as a plugin on top of the excellent open source GIS package, QGIS.

During the build of roadNet, we found it useful to make each version of the code available for installation and testing via its own QGIS repository.  This post explains how it works.

RAMP_screenshot
roadNet manages the spatial and non-spatial data required to produce a BS7666-compliant local street gazetteer.  It features automatic line-splitting and updating multiple database tables in response to user edits as well as data validation and export in standard gazetteer data transfer formats e.g. SDTF.

git, GitHub, Shippable, Docker and Amazon S3

The roadNet continuous integration (CI) system makes use of a number of cloud-based services.  We use git and GitHub for version control to allow developers to track changes to the code and use separate branches to develop new features.  GitHub is linked to Shippable, which watches out for new commits to the code base.  This is similar to other CI systems such as Travis.  When the new code is committed Shippable spins into action.

cloud_deployment_all

Shippable is used to check the code and to create the cloud repositories.  The instructions to do this are stored in the shippable.yml file.  It does this inside a docker container, which contains QGIS and all its dependencies already installed and configured.

This stage was a bit tricky to configure because QGIS, as a desktop GIS application, assumes that it is running on a desktop computer with attached display to which it can send windows and message dialogs, when infact it is running on a little bit of memory and a little bit of hard disk on a big computer in a warehouse somewhere i.e. the Amazon Cloud.  The DockerFile contains the instructions to set up a fake display (or X virtual frame buffer) in the container.

Once the code has been tested a Python script pushes it out to the repository.

QGIS plugin repositories in the cloud

A QGIS plugin repository is just a web-facing folder that contains a plugins.xml file that describes the available plugins and a series of zip files containing the code.  Amazon Web Services includes the S3 service, which provides ‘buckets’ for storing files.  These can be configured to be accessible for the web, making them ideal for hosting repositories.

The deploy.py script contains the instructions to zip up the files, prepare plugins.xml and copy the files to S3.  The core of the key function is below:

def update_repos(build_name):
    """
    Create new repo for build, and update latest master/dev repos.
    :param build_name:
    """
    deploy_to_s3(build_name)
    deploy_to_s3('latest_push')

    branch = os.getenv('BRANCH')
    if branch is None:
        return
    elif branch == 'develop':
        deploy_to_s3('latest_develop')
    elif branch == 'master':
        deploy_to_s3('latest_master')

It is so easy to create repositories that we just make lots of them.  Every set of changes gets a build_name.  The first ‘deploy_to_s3’ line creates a repository specifically for that build.  These are all stored indefinitely.  This means that just by connecting to the specific repository, we can run the code as it was at any stage during the development.

The other ‘deploy_to_s3’ lines provide convenience repositories.  These get a copy of the code that was just pushed, meaning developers can connect to the latest_push and see their most recent changes.  thinkWhere’s testers can connect to latest_develop and try out new features as soon as they are merged into the develop branch.  Clients point their QGIS installs at the latest_master branch to ensure that they keep up with the latest stable releases.

Anyone can update to the latest version in their branch with a single click in the QGIS plugin installer.

Conclusion

We have found the automatic deployment of QGIS plugins to be immensely useful, facilitating both rapid development / testing feedback loops, and easy delivery of bug fixes and upgrades to the master branch. Check out roadNet today from the official QGIS plugin repository:

https://plugins.qgis.org/plugins/Roadnet/

If you think this may be useful to you, please also have a look at the code – its all released under GPL v2.

Further reading

You can install the latest stable build of roadNet in your local QGIS machine by adding our latest_master repository to your installation (Plugins > Manage and Install Plugins > Settings):

http://roadnet-builds.s3-website-eu-west-1.amazonaws.com/latest_master/plugins.xml??qgis=2.14

Viewsheds and Visibility Analysis in QGIS

Recently I had the occasion to do a bit of viewshed analysis in QGIS. I have done these before using tools in ArcGIS, but this was the first time I had the pleasure of doing this kind of analysis in QGIS. I was impressed by the simplicity and flexibility of the toolset.

The views from our office at Stirling are fairly legendary, with Stirling Castle, much of the Ochil Hills and the Wallace Monument all visible from our windows. But how to quantify this – what can we see from our office? This is what a viewshed does – using a height model and a position on the map it will tell you what you can (theoretically) see.

As a starting point, we can use the Ordnance Survey’s 50m Digital Elevation Model: Terrain 50.

Scotland DEM

So now we have the heights of all the hills and glens of central Scotland represented in a raster.

The function we are going to use is the Grass r.viewshed function which, rather conveniently, does most of the heavy mathematical lifting for you.

The R.Viewshed function is a GRASS function which integrates nicely into QGIS allowing you to use the algorithm without delving into the GRASS interfaces.

r_viewshed

So here we are using the Elevation Model called Terrain 50, the coordinates identifying the viewing position (thinkWhere’s office in Stirling).

We are on the first floor and without measuring the height of the building and our average employee standing up, we have estimated the height of the viewing point at 5m.

The viewshed analysis allows you to set up a height to offset the target elevations as well. This would be very useful for an analysis of windfarm visibility or  for a radio mast.

A good way to think about it is like this:

viewshed theory1

In this example, the viewer is looking to see what part of the landscape they can see. The height of the person is the viewing position and there is no height offset for heights on the landscape.

In this example, the viewing position is on the top of a hill and looking out at objects of a defined height.

viewshed theory2

So we can run this algorithm in QGIS to compute the theoretical view from our office.

The output looks a bit like this. Now it needs a little bit of processing to get it to something a bit more meaningful.

viewshed output

First of all, we’ll make this a vector for better symbology options. We can use the GDAL “Polygonize” function for this. The way that QGIS integrates these different applications into one place makes life so much easier for the GIS analyst.

polygonize

Bringing the QGIS symbologies into play, we can show the viewshed like this:

viewshed output_backdrop

However, we can have more fun than that.

Projecting a hillshade on to the Digital Elevation Model and bringing in some Ordnance Survey Vector data allows us to produce something far more meaningful.

viewshed_carto

Using the hillshade analysis (that is the way to  create the really funky 3d effect on the landscape) and bringing in some of the Vector Map District from the Ordnance Survey you can bring this map alive. Using only a very small amount of the data and concentrating on the physical features a viewshed analysis can be easily understood.

Using the blend mode stops the viewshed from obscuring the underlying data in the hillshade raster, making the information easy to understand. A few labels from the VMD names dataset helps give a little bit of context.

Viewsheds are very heavily used in renewables, telecommunication, and planning. QGIS, along with Ordnance Survey Open Data allows this to be done by anyone with access to a computer.

 

Analysing Placenames in QGIS

We meet many different people trying to do different things here at thinkWhere and one of our attendees at our QGIS Skills of Analysis & Statistics was particularly interested in looking at place names. Place names are a really interesting way of telling what kind of history a place has had. The Island of Jura is a great example of this deriving from an old Norse word meaning Animal Island. Common Scottish names such as Tarbert and Ben Mor are Gaelic derived place names translating as a narrow strip of land or a big hill respectively.

While we often wonder about the originality of our forbears in how they choose to name things, these are the footprints of history in maps we use every day. They teach us about where we come from and how the peoples who come before us lived with their landscape. It’s very romantic and also a terrific thing for some GIS analysis.

QGIS has some nice ways of helping with this kind of thing, alongside OpenStreetMap which is a rich source of place name data. The Ordnance Survey Open Names dataset would also work well for this kind of analysis.

Duns & Dums

A common word found in Scottish place names is “Dun” meaning a hill fort or roundhouse in the Celtic tradition. This is sometimes rendered as “Dun” (as in Dundee or Dunfermline) or as “Dum” (As in Dumfries or Dumbarton). Less common in Scotland, although still prevalent in Ireland is the prefix “Don” (As in Donegal), however this gets confused in Scotland due to the River Don. Don in Scots Gaelic is more associated geographically with the Celtic river goddess.

In QGIS we can use the expression:

“name” ILIKE ‘%dun%’ OR “name” ILIKE ‘%dum%’

To select or filter for any words that contain those strings of letters.

map_of_dun

The ILIKE function (a case-insensitive LIKE) is a great way of doing these kind of analyses and you can use the “%” wildcard to make selections in almost any context.

With a spot of good symbology and the use of the Print Composer to show multiple maps on the same page. You can show the distribution of a few different place name components. Having consistent maps next to each other on a page like this helps to show distribution.

placenames

A really effective way of showing this kind of data can be to use the heatmap. The last few versions of QGIS have put this into the standard symbology tool which allows you to generate them on the fly without the need to create a whole new raster. This makes it much easier. This is what the distribution of “dun”/”dum” components looks like.

dun_heatmap

The big problem with this kind of analysis is reliance on the data. The OpenStreetMap relies on users adding in data themselves and that makes an analysis of this nature a bit shaky with crowd sourced data.

The OS Open Names dataset would be a great way of doing that, but it is a huge dataset that would need a database set up for it. We do cover database creation in our advanced QGIS course, but it would be a good follow up.

Check out our QGIS training offerings at: thinkWhere.com