Accuracy assessment

Tutorial: Accuracy Assessment[edit]

Note: For this tutorial, you will need to install the Semi-Automatic Classification Plugin and have a finished classification loaded into QGIS. Learn how to create a classification here [link to supervised classification]. Sampling strategy

Before starting the accuracy assessment, the sampling strategy has to be chosen:

For random sampling, points are randomly distributed throughout the area without specific criteria.

For equalized stratified random sampling, the number of random samples per class is equal. For stratified random sampling, random sample points are determined for each class relative to its area. Further, you need to determine the overall number of samples. This can be done mathematically (see e.g. Olofsson et al. 2014). As a rule of thumb, you can aim for approximately 50 samples per class.

For this tutorial we will use the classification created in this tutorial [link to supervised classification]. Because the classes are quite unequally distributed here, we will try a stratified approach. For the purpose of this tutorial, we will set the overall number of samples to 50.

Classification report

For stratified random sampling, the number of sampling points per class is determined based on its area proportion. Consequently, the first step is to run a classification report, which will tell us the area proportion of each class. Open the SCP, go to Postprocessing → Classification report. Select the classification you want to assess (use the refresh button), and click Run.

You will get a table with the results and they will also be displayed in the SCP.

Distribute samples among classes

Now that we know the area proportion of each class, we can set up a table to calculate the number of samples per class. Multiply the proportion with the overall number of samples (n) to get the weighted distribution. Because the sample size for the rarer classes will be very small with the low number of samples in this tutorial, we will take the mean between the equal distribution (equal number of samples per class) and the weighted distribution here.

Generate random sample points The next step is to generate the sample points. First, open the SCP and add your classification under Band Set.

[[

]]

Next, create a new training input in the SCP Dock.

In the working toolbar, we will set Min and Max to 1 to generate single pixels. Then, in the SCP, go to Basic tools → Multiple ROI creation. Make sure that the correct band set (your classification) is selected. Check the option stratified for the values and add the expression raster == 1 to generate the samples for the first class. Set the number of points to the value you calculated. Then click Create Points. Repeat for the other classes, by adjusting the expression (e.g. raster == 2 for the second class) and the number of points until all sample points have been added to the table.

Once you have generated all sample points, uncheck Calculate signatures and Run. The points will be added to your training input. This might take a while.

Assign correct class IDs

Now comes the time-consuming part: Based on visual interpretation (ideally with higher resolution images), we will now manually assign the correct Macroclass IDs to all the samples in the new training input. Mind that these have to match the classes of your classification. Go to the training input in the SCP Dock and double click on a row to zoom into a pixel. By default, all classes are set to one. Based on a reference (e.g. a true colour image) change the macroclass number so that it represents the correct land cover class according to your classification system. Simply click on the MC ID field and type in the correct value. You need to do this for every sample point.

Accuracy Assessment

Once you have labelled all samples correctly, open the SCP and go to Postprocessing → Accuracy. Select the classification to assess and the layer with the labelled samples. You might have to use the refresh button. For Vector field, choose macroclass. Then you can run the accuracy assessment!

Results

Once the accuracy assessment is finished, you will receive a detailed output with the results. The overall accuracy in this example was 78 %, which is not too bad, but usually one should aim for > 80%. We can see that the producer’s accuracy was low for water (2) and infrastructure (3), and the user’s accuracy was low for agriculture (5), so these should be improved. However, the number of samples in this example was of course very low.