Introduction

This tutorial is designed to supplement Module 6 of the OBIS/OTGA self-paced course: Contributing and publishing datasets to OBIS. It demonstrates how to run some basic quality control (QC) checks in R. The dataset used in the tutorial is the example fishing dataset, “ExampleDataset-1” as labeled in the course. Always remember when we want to make any changes we make to a file it is good practice to ensure a copy of the original data remains unchanged. For this QC tutorial, we will check the following:

• Taxa information,
• Required fields,
• Coordinates,
• Dates,
• Identifiers,
• Presence of outliers.

We will use two R packages to check the data: obistools and Hmisc. For this tutorial, all required packages are installed and loaded for you, but would have to install them if you chose to run these checks on your own computer. Keep in mind that installing obistoolsrequires the devtools package. See here for details on installing obistools. Each code chunk in the tutorial will have a “Run Code” button which will allow you to run the code as it would in your own R instance. You may also be prompted to write your own code at times.

While this tutorial can be used as an aide for the OBIS/OTGA Course Exercise 6, particularly if you are new to R, please ensure assignment answers are written in your own words.

Note also that this tutorial focuses on demonstrating QC checks in R, but is not comprehensive and you will have to complete certain aspects of QC (e.g. adding missing required columns) on your own.

Load data files

The first step is of course to load our Event and Occurrence table into R. We have already separated the Excel file “OBIS_Exercise6-1_QC.xlsx” into two separate .csv files to make the QC process easier.

# Read the data files
event<-read.csv("data/QC_event.csv")
occur<-read.csv("data/QC_occur.csv")

Right away we get an error that there’s an invalid string in the data. This is likely due to an encoding error, usually if there’s a symbol or other special character in the data. To get around this, let’s specify the encoding for our data tables.

# Read the data files
event<-read.csv("data/QC_event.csv", fileEncoding = "latin1")
occur<-read.csv("data/QC_occur.csv", fileEncoding = "latin1")

Now the files are being read correctly, hurray!

str(event)

str(occur)

event$decimalLongitude
event$decimalLatitude

library(measurements)

new_long<- conv_unit(c('54 17 00','54 10 0.450') , from = "deg_min_sec", to = "dec_deg") #we added 00 to the long/lat for row 27 so that the coordinates convert properly 
new_lat <- conv_unit(c('-132 57 00', '-132 59 0.536'), from = "deg_min_sec", to = "dec_deg")


# Replace data in rows 27 and 46 for our latitude and longitude columns
event[27, c("decimalLatitude", "decimalLongitude")] <- c(new_lat[1], new_long[1])
event[46, c("decimalLatitude", "decimalLongitude")] <- c(new_lat[2], new_long[2])

# Change the longitude and latitude columns to be read as numeric
event$decimalLatitude<-as.numeric(event$decimalLatitude)
event$decimalLongitude<-as.numeric(event$decimalLongitude)

# Double check that the columns are now numeric
str(event$decimalLatitude) 
str(event$decimalLatitude)

Check required fields

One of the first checks we will do is to make sure all the OBIS-required fields are present in our data. To do we will use the obistools::check_fields function. We should make sure to do this for all relevant tables, so we will run the function for both the Event and Occurrence table. Use this function in the code block below to check for required fields in our two data tables.

obistools::check_fields(event)
obistools::check_fields(occur)

# Rename the date column to eventeDate
colnames(event)[3]<-"eventDate"

Check taxon names

The next step is to check that the taxon names match with WoRMS. We will use obistools::match_taxa and then inspect the output to make sure all the names in the Occurrence table match with WoRMS and are marine. Note that the code below will not be executable, but we provide the relevant output below the code chunk.

worms<-match_taxa(unique(occur$scientificName), ask=TRUE) # conduct taxon matching by telling the function to only match the unique species names in the scientificName column


## If you did not already have the scientificNameID field in the dataset, you could use the code below to attach the worms data with the Occurrence table
## Note that we set all=TRUE --> this will keep every record in the worms table which can be handy if there are any errors with taxon names. The matched name will appear in the Occurrence table with NAs for any non-WoRMS field

occur_match<-merge(occur, worms, by="scientificName", all= TRUE) # merges the worms dataframe with the occurrence table (occur) by matching according to scientificName

colnames(occur_match)[colnames(occur_match) == "scientificNameID.y"] = "scientificNameID" # rename the new column as scientificNameID

We see that one taxon has a near match, “Sebastes melanops” (Table 1), and when we look for this species in the Occurrence table (Table 2), we see that and that the name presumably had a typo, as it is written as “Sebastes melanps”. Changes to the scientificName can be made when you are certain, and verbatimIdentification can be populated with the original name.

Check coordinates

Next, we’ll visualize the data using obistools::plot_map_leaflet(). This tool allows us to quickly spot any coordinates that might be out of place, potentially indicating that longitude and latitude values have been swapped.

We’ll use obistools::plot_map_leaflet instead of obistools::plot_map because it opens an interactive leaflet map, allowing us to identify the row number of any coordinates that fall outside the expected geographic area. Be sure to zoom out to view the entire map, as some coordinates might be significantly distant from the expected range. Use this function below to investigate coordinates.

obistools::plot_map_leaflet(event)

obistools::check_depth()
obistools::check_onland()

obistools::check_depth(event, report=TRUE, depthmargin = 30)
obistools::check_onland(event,report=TRUE, buffer=20)

Check table identifiers

We need to check that eventID and parentEventID are present and correspond with one another, for which we’ll use obistools::check_eventids(). Then, to ensure all eventIDs in the extension table(s) have matching eventIDs in the Event core table, we’ll use obistools::check_extension_eventids().

Modify the code below to run the checks.

obistools::check_eventids()
obistools::check_extension_eventids(event, extension)

obistools::check_eventids(event)
obistools::check_extension_eventids(event,occur)

Check date formatting

Next we’ll confirm that all the dates are in ISO 8601 formatting using obistools::check_eventdate. Use this function in the code block below.

obistools::check_eventdate(event)

Look at the data with Hmisc

The final check we will perform is using the Hmisc::describe function to identify any inconsistencies in the data. We will be running this on both the Event and Occurrence table, highlighting important fields below. Remember, although we will only highlight a few data fields, it is good practice to verify that each column contains data as expected.

The describe function gives a short summary for each data column, including the number of non-blank rows (n), the number of rows missing a value (missing), and the number of distinct values for that column (distinct).

Use this function in the code block below to describe both the Event and Occurrence tables. We’ve included some additional code to demonstrate how you can also plot the results of the describe function.

Hmisc::describe()

# Plot some results of describe
options(grType='plotly')
e<-Hmisc::describe(event)
p<-plot(e)
p[[1]]; p[[2]]

Hmisc::describe(event)
Hmisc::describe(occur)

# Plot some results of describe
options(grType='plotly')
e<-Hmisc::describe(event)
p<-plot(e)
p[[1]]; p[[2]]

Conclusion

This tutorial has covered some basic QC steps you can conduct in R to check the data quality of a dataset. You can use the output from this tutorial to help you with exercises in the course, but again please ensure the answers are your own.

Note we could also use obistools::report() to generate a quick overview of QC issues, but it’s good to make sure you understand each check individually, which is why we have presented it step by step here. Running this code will open a report.html file in your browser.

obistools::report(event)
obistools::report(occur)