Home

Maritimes/Mar.datawrangling - Introduction

April 27, 2018 - Mike McMahon (mike.mcmahon@dfo-mpo.gc.ca)

This is an R-package containing tools for extracting data from the Maritimes fisheries science databases.
These functions are specific to these databases, and will not be useful to anyone without the appropriate permissions.

The purpose of this package is to:

1. enable authorized users to extract the data they need for local use;
2. faciliate the generation of meaningful datasets from the observer database without forcing a knowledge of SQL;
3. faciliate the combination of extracted data into a single entity that is easy to work with (e.g. R, excel, ArcGIS).

Approach

• Generic enough to allow for multiple data sources (e.g. groundfish, marfissci, observer databases, etc)
• GUI available if desired, but not necessary (can still embed components in scripts)
• R package (like you'd install from CRAN)
  • includes documentation
  • brings in required packages
  • no configuration necessary

How it Works

• Connects to Oracle, and extracts data locally (only if necessary)

  • local copies of data load faster;
  • allows people to work from home

• Configuration file contains all of the information about the various data sources, e.g.

  • what tables exist,
  • how they are linked (primary/foreign relationships),
  • fields that might be good to allow filtering on...

• If using the GUI, a series of popups allow you to filter the data

• Following the application of filters, a series of merges are done within the data to drop all records that no longer apply.

Available Data

At present, this package works with the following Maritimes databases:

Name	parameter	Abstract
Groundfish/RV/Ecosystem Surveys	db='rv'	Bottom Trawl Surveys conducted in shore waters off south west Nova Scotia
Pre 70s Groundfish/RV/Ecosystem Surveys	db='rvp70'	Bottom Trawl Surveys conducted in shore waters off south west Nova Scotia conducted prior to 1970
Industry Surveys Database	db='isdb'	DFO at-sea fish catch observations from commercial fishing vessels operating in the North West Atlantic.
Cape Chidley	db='chid'	Exploratory fishing surveys of the benthic fish fauna at 900-1800m.
Redfish	db='redfish'	A survey using stratified random design with day/night replication targeting deep sea redfish
MARFIS	db='marfis'	A Policy and Economics Branch database that houses information on the fisheries \n\tof the Scotia-Fundy region, including data related to catch and effort.
COMLAND (1967-1985)	db='comland67'	Commercial Landings (1967-1985) - like MARFIS, but earlier.
COMLAND (1986-2001)	db='comland86'	Commercial Landings (1986-2001) - like MARFIS, but earlier. This is separated from the 1967 - 1985 COMLAND data due to differences in the code tables.
Stomach Database	db='stomach'	Stomach Contents Database
Atlantic Salmon Enumeration Facility Database	db='asef'	Tagged salmon records at fishways
Mesopelagic Database	db='meso'	Mesopelagic Database
Mesopelagic Gully Database	db='meso_gully'	Mesopelagic Gully Database
Juvenile Silver Hake Database	db='juvesh'	Juvenile Silver Hake Database

Installation

In R, you first need to install the devtools package, then you can install this package directly from github. In practice, this looks like:

library(devtools)
install_github('Maritimes/Mar.datawrangling')

Downloading GitHub repo Maritimes/Mar.datawrangling@master
from URL https://api.github.com/repos/Maritimes/Mar.datawrangling/zipball/master
Installing Mar.datawrangling
....
Reloading installed Mar.datawrangling
> 

(If you don't already have it, this package will also install the more general-purpose Maritimes/Mar.utils package.)