.. only:: html

.. _tm_preparing_data:

Preparing Exercise Data
=======================

.. note:: This process is intended for course conveners, or more experienced
QGIS users who wish to create localised sample data sets for their course.
Default data sets are provided with the Training Manual, but you may follow
these instructions if you wish to replace the default data sets.

The :ref:`sample data provided <data_downloadlink>` with the Training Manual
refers to the town of |majorUrbanName| and its surroundings. |majorUrbanName| is
located about 2 hours' east of Cape Town in the Western Cape of South Africa.
The dataset contains feature names in both English and Afrikaans.

Anyone can use this dataset without difficulty, but you may prefer to use data
from your own country or home town. If you choose to do so, your localised
data will be used in all lessons from Module 3 to Module 7.2. Later modules use
more complex data sources which may or may not be available for your region.

.. note:: These instructions assume you have a good knowledge of QGIS and are
not intended to be used as teaching material.

|hard| |TY| Create OSM based vector Files
--------------------------------------------------------------------------------

If you wish to replace the default data set with localised data for your course,
this can easily be done with tools built into QGIS. The region you choose to use
should have a good mix of urban and rural areas, containing roads of differing
significance, area boundaries (such as nature reserves or farms) and surface
water, such as streams and rivers.

#. Open a new QGIS project
#. Select :menuselection:`Layer --> Data Source Manager` to open the
:guilabel:`Data Source Manager` dialog
#. In the :guilabel:`Browser` tab, expand the :guilabel:`XYZ Tiles` drop-down menu
and double-click the :guilabel:`OpenStreetMap` item.

.. image:: img/browser_xyztiles.png
:align: center
:scale: 60

A map of the world is now visible on the map canvas.
#. Close the :guilabel:`Data Source Manager` dialog
#. Move to the area you'd like to use as study area

.. image:: img/swellendam_neighborhood.png
:align: center
:scale: 60

Now that we have the area we'll extract the data from, let's enable the extraction
tools.

#. Go to :menuselection:`Plugins --> Manage/Install Plugins...`
#. In the :guilabel:`All` tab, type ``QuickOSM`` in the search box
#. Select the QuickOSM plugin, press :guilabel:`Install Plugin` and then :guilabel:`Close`
the dialog.

.. image:: img/quickosm_plugin_download.png
:align: center

#. Execute the new plugin from :menuselection:`Vector --> QuickOSM -->
QuickOSM...` menu
#. In the :guilabel:`Quick query` tab, select ``building`` in the :guilabel:`Key`
drop-down menu
#. Leave the :guilabel:`Value` field empty, meaning that you are querying all
buildings.
#. Select :guilabel:`Canvas Extent` in the next drop-down menu
#. Expand the :guilabel:`Advanced` group below and uncheck all
geometry types on the right except :guilabel:`Multipolygons`.
#. Press :guilabel:`Run query`

.. image:: img/building_query_builder.png
:align: center

A new ``building`` layer is added to the :guilabel:`Layers` panel, showing
buildings in the selected extent.
#. Proceed as above to extract other data:

#. ``Key = landuse`` and ``Multipolygons`` geometry type.
#. ``Key = boundary``, ``Value = protected_area`` and ``Multipolygons``
geometry type.
#. ``Key = natural``, ``Value = water`` and ``Multipolygons`` geometry type.
#. ``Key = highway`` and check ``Lines`` and ``Multilines`` geometry types.
#. ``Key = waterway``, ``Value = river`` and check ``Lines`` and ``Multilines``
geometry types.
#. ``Key = place`` and ``Points`` geometry type.

This process adds the layers as temporary files (indicated by the
|indicatorMemory| icon next to their name).

.. image:: img/osm_data_loaded.png
:align: center

You can sample the data your region contains in order to see what kind of
results your region will yield.

We now need to save the resulting data to use during your
course. We'll be using ESRI Shapefile, GeoPackage and SpatiaLite formats
depending on the data.

To convert the :guilabel:`place` temporary layer to another format:

#. Click the |indicatorMemory| icon next to the :guilabel:`place` layer
to open the :guilabel:`Save Scratch Layer` dialog.

.. note:: If you need to change any of the temporary layer's properties (CRS, extent,
fields...), use the :menuselection:`Export
--> Save Features as...` contextual menu instead, and ensure the :guilabel:`Add
saved file to map` option is checked. This adds a new layer.

#. Select the :guilabel:`ESRI Shapefile` format
#. Use the :guilabel:`...` button to browse to the :file:`exercise_data/shapefile/`
folder and save the file as :file:`places.shp`.

.. image:: img/save_osm_place.png
:align: center

#. Press :guilabel:`OK`

In the :guilabel:`Layers` panel, the temporary :guilabel:`place` layer is
replaced with the saved :guilabel:`places` shapefile layer and
the temporary icon next to it removed.
#. Double-click the layer to open its :menuselection:`Layer Properties -->
Source` tab and update the :guilabel:`Layer name` property to match the file
name.

#. Repeat the process for other layers, renaming them as follows:

* ``natural_water`` into ``water``
* ``waterway_river`` into ``rivers``
* ``boundary_protected_area`` into ``protected_areas``

Each resulting data set should be saved in the :file:`exercise_data/shapefile/`
directory.

The next step is to create a GeoPackage file from the :guilabel:`building` layer
to use during the course:

#. Click the |indicatorMemory| icon next to the :guilabel:`building` layer
#. Select the :guilabel:`GeoPackage` format
#. Save the file as :file:`training_data.gpkg` under the :file:`exercise_data/`
folder
#. By default, the :guilabel:`Layer name` is filled as the file name.
Replace it with ``buildings``.

.. image:: img/save_osm_building.png
:align: center

#. Press :guilabel:`OK`
#. Rename the layer in its properties dialog
#. Repeat the process with the :guilabel:`highway` layer, saving it as ``roads`` in
the same GeoPackage database.

The last step is to save the remaining temporary file as a SpatiaLite file.

#. Click the |indicatorMemory| icon next to the :guilabel:`landuse` layer
#. Select the :guilabel:`SpatiaLite` format
#. Save the file as :file:`landuse.sqlite` under the :file:`exercise_data/`
folder. By default, the :guilabel:`Layer name` is filled as the file name.
Do not change it.

.. image:: img/save_osm_landuse.png
:align: center

#. Press :guilabel:`OK`

You should now have a map which looks something like this (the symbology will
certainly be very different, because QGIS randomly assigns colors when layers
are added to the map):

.. image:: img/post_osm_import.png
:align: center

The important thing is that you have 7 vector layers matching those
shown above and that all those layers have some data.

|hard| |TY| Create SRTM DEM tiff Files
--------------------------------------------------------------------------------

For modules :ref:`tm_create_vector_data` and :ref:`tm_rasters`, you'll also need
raster images (SRTM DEM) which cover the region you have selected for your
course.

The CGIAR-CGI (http://srtm.csi.cgiar.org/) provides some SRTM DEM you can download
from http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp.

You'll need images which cover the entire region you have chosen to use.
To find the extent coordinates, in QGIS , |zoomToLayer| zoom to the extent of
the largest layer and pick the values in the |extents| :guilabel:`Extents`
box of the status bar.
Keep the :file:`GeoTiff` format. Once the form is filled, click on the
:guilabel:`Click here to Begin Search >>` button and download the file(s).

Once you have downloaded the required file(s), they should be saved in the
:file:`exercise_data` directory, under :file:`raster/SRTM` subfolders.

|hard| |TY| Create imagery tiff Files
--------------------------------------------------------------------------------

In Module :ref:`tm_create_vector_data`, :ref:`tm_datasources` lesson shows
close-up images of three school sports fields
which students are asked to digitize. You'll therefore need to reproduce these
images using your new SRTM DEM tiff file(s). There is no obligation to use school
sports fields: any three school land-use types can be used (e.g. different
school buildings, playgrounds or car parks).

For reference, the image in the example data is:

.. image:: img/field_outlines.png
:align: center

|basic| |TY| Replace Tokens
--------------------------------------------------------------------------------

Having created your localised dataset, the final step is to replace the tokens
in the :file:`substitutions.txt` file so that the appropriate names will appear
in your localised version of the Training Manual.

The tokens you need to replace are as follows:

* ``majorUrbanName``: this defaults to "Swellendam". Replace with the name of
the major town in your region.
* ``schoolAreaType1``: this defaults to "athletics field". Replace with the
name of the largest school area type in your region.
* ``largeLandUseArea``: this defaults to "Bontebok National Park". Replace
with the name of a large landuse polygon in your region.
* ``srtmFileName``: this defaults to :file:`srtm_41_19.tif`. Replace this
with the filename of your SRTM DEM file.
* ``localCRS``: this defaults to ``WGS 84 / UTM 34S``. You should replace
this with the correct CRS for your region.

.. Substitutions definitions - AVOID EDITING PAST THIS LINE
This will be automatically updated by the find_set_subst.py script.
If you need to create a new substitution manually,
please add it also to the substitutions.txt file in the
source folder.