Skip to contents
library(motus)
library(tidyverse)

sql.motus <- tagme(176, update = TRUE, new = TRUE, dir = "./data/")

As work progresses and data are fine-tuned, batches of data may be re-run on the Motus servers, and as a result, be assigned new batch numbers (see the section on Reprocessing Data in the Understanding Motus Data article).

This results is a disconnect between the user’s local data and the data on the server. When the user updates their data, they’ll have the new batches, but will also have the older, deprecated batches.

Users can see which batches have been deprecated in the deprecated table:

tbl(sql.motus, "deprecated")
## # Source:   table<deprecated> [6 x 3]
## # Database: sqlite 3.39.3 [/home/runner/work/motus/motus/vignettes/articles/data/project-176.motus]
##   batchID batchFilter removed
##     <int>       <int>   <int>
## 1    6000           4       0
## 2    6001           4       0
## 3    6002           4       0
## 4    6003           4       0
## 5    6004           4       0
## 6    6005           4       0

Each batchID listed represented a deprecated batch. The column removed indicates whether this batch has been removed from your data (1) or not (0).

This table is updated every time you update your data with a call to tagme(). If you want to update it again (without removing anything), you can use

sql.motus <- deprecateBatches(sql.motus, fetchOnly = TRUE)
## Fetching deprecated batches
## Total deprecated batches: 6
## New deprecated batches: 0

To see where these batches are in your data, you can filter for the IDs in a specific table

tbl(sql.motus, "alltags") %>%
  filter(batchID %in% c(6000, 6001, 6002))
## # Source:   SQL [?? x 62]
## # Database: sqlite 3.39.3 [/home/runner/work/motus/motus/vignettes/articles/data/project-176.motus]
##       hitID   runID batchID       ts tsCor…¹   sig sigsd noise  freq freqsd   slop burst…²
##       <int>   <int>   <int>    <dbl>   <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl>  <dbl>   <dbl>
##  1 23271881 1822216    6000   1.49e9  1.49e9 -68.2 10.9  -76.3  2.07 0.127  0.0034  0     
##  2 23271882 1822216    6000   1.49e9  1.49e9 -67.9 18.9  -75.7  1.98 0.185  0.0019  0.026 
##  3 23284651 1828340    6000   1.49e9  1.49e9 -68.7 34.9  -76.4  2.00 0.11   0.0045  0     
##  4 23284652 1828340    6000   1.49e9  1.49e9 -68.9 15.3  -76.7  1.90 0.0315 0.0047 -0.0197
##  5 23306136 1838711    6000   1.49e9  1.49e9 -68.1  8.84 -76.9  1.92 0.0351 0.0047  0     
##  6 23306137 1838711    6000   1.49e9  1.49e9 -69.3 14.8  -76.4  1.91 0.0675 0.0013  0.0442
##  7 23306412 1838841    6000   1.49e9  1.49e9 -69.7 22.9  -76.8  1.88 0.0356 0.0047  0     
##  8 23306413 1838841    6000   1.49e9  1.49e9 -69.3 17.8  -76.4  1.88 0.0716 0.0022 -0.0229
##  9 23322817 1846692    6000   1.49e9  1.49e9 -67.3 15.0  -75.3  1.91 0.116  0.0046  0     
## 10 23322818 1846692    6000   1.49e9  1.49e9 -67.9 21.5  -76.0  2.00 0.166  0.0048 -0.0194
## # … with more rows, 50 more variables: done <int>, motusTagID <int>, ambigID <int>,
## #   port <chr>, nodeNum <chr>, runLen <int>, motusFilter <dbl>, bootnum <int>,
## #   tagProjID <int>, mfgID <chr>, tagType <chr>, codeSet <chr>, mfg <chr>,
## #   tagModel <chr>, tagLifespan <dbl>, nomFreq <dbl>, tagBI <dbl>, pulseLen <dbl>,
## #   tagDeployID <int>, speciesID <int>, markerNumber <chr>, markerType <chr>,
## #   tagDeployStart <dbl>, tagDeployEnd <dbl>, tagDepLat <dbl>, tagDepLon <dbl>,
## #   tagDepAlt <dbl>, tagDepComments <chr>, tagDeployTest <int>, fullID <chr>, …

Deprecated batches can also be removed with the deprecateBatches() function, which, by default, fetches the update-to-date record of deprecated batches and then removes them.

However, once removed, deprecated batches are gone for good. It is advisable to backup your dataset before proceeding (see below).

Just in case, deprecateBatches() will ask you if you are sure you want to remove the batches before proceeding.

sql.motus <- deprecateBatches(sql.motus)
You are about to permanently delete up to 6 deprecated batches from project-176.motus
Continue? 

1: Yes
2: No

Selection: 1
## Fetching deprecated batches
## Total deprecated batches: 6
## New deprecated batches: 0
##   232 deprecated rows deleted from runs
##   473 deprecated rows deleted from hits
##   780 deprecated rows deleted from activity
##   232 deprecated rows deleted from batchRuns
##   6 deprecated rows deleted from projBatch
##   6 deprecated rows deleted from batches
## Total deprecated batches removed: 6

After removal, you can see that the batches have been removed from the data

tbl(sql.motus, "alltags") %>%
  filter(batchID %in% c(6000, 6001, 6002))
## # Source:   SQL [0 x 62]
## # Database: sqlite 3.39.3 [/home/runner/work/motus/motus/vignettes/articles/data/project-176.motus]
## # … with 62 variables: hitID <int>, runID <int>, batchID <int>, ts <dbl>,
## #   tsCorrected <lgl>, sig <dbl>, sigsd <dbl>, noise <dbl>, freq <dbl>, freqsd <dbl>,
## #   slop <dbl>, burstSlop <dbl>, done <int>, motusTagID <lgl>, ambigID <int>, port <chr>,
## #   nodeNum <chr>, runLen <int>, motusFilter <dbl>, bootnum <int>, tagProjID <int>,
## #   mfgID <chr>, tagType <chr>, codeSet <chr>, mfg <chr>, tagModel <chr>,
## #   tagLifespan <dbl>, nomFreq <dbl>, tagBI <dbl>, pulseLen <dbl>, tagDeployID <int>,
## #   speciesID <int>, markerNumber <chr>, markerType <chr>, tagDeployStart <dbl>, …

Also that the deprecated table now lists removed as 1.

tbl(sql.motus, "deprecated")
## # Source:   table<deprecated> [6 x 3]
## # Database: sqlite 3.39.3 [/home/runner/work/motus/motus/vignettes/articles/data/project-176.motus]
##   batchID batchFilter removed
##     <int>       <int>   <int>
## 1    6000           4       1
## 2    6001           4       1
## 3    6002           4       1
## 4    6003           4       1
## 5    6004           4       1
## 6    6005           4       1

Why not remove deprecated batches?

In active projects it is a good idea to remove deprecated batches routinely, to ensure your data is update to date and as accurate as possible.

However, once removed, deprecated batches are gone for good. If you have an analysis or publication based on older data versions, this analysis is no longer repeatable as you no longer have the original batches.

It is therefore advisable to keep a copy of the database (i.e. the XXXX.motus file) you need for a specific analysis backed up and static (i.e. no updates and no removal of deprecated batches).

What Next? Explore all articles