% Voyages of Sinbad the Sailor
% Yihui Xie
% 2012/04/23

``` {r setup, include=FALSE}
opts_chunk$set(cache=TRUE)
opts_knit$set(upload.fun = imgur_upload)
options(width=40)
read_chunk('stat585x-shipping-yihui-xie.R')
```

# Data extraction

![Voyages of Sinbad the Sailor](https://i.imgur.com/wAhOy.jpg)

# Data cleaning

![Voyages of Sinbad the Sailor](https://i.imgur.com/TA7rp.jpg)

# Explore the Treasure Island

![Treasure Island](https://i.imgur.com/aisEo.jpg)

---

> The existence of abundant meteorological data from logbooks of different European countries constitutes a common and invaluable heritage of the most outstanding scientific interest. The analysis of the logbooks content will contribute to characterise climate during XVIII th and XIX th centuries and to asses climate change.
>
> http://www.ucm.es/info/cliwoc/

# The data

- Climatological Database for the World's Oceans 1750-1850
- available at <http://www.ucm.es/info/cliwoc/cliwoc15.htm>
- fixed width + semicolon separated values

# Sample data

```
1790 424                0106    10     1446ES                                                               99 0 MNM;Archivo Museo Naval Madrid;Madrid;Spain;;;;;MS. 204;Spanish;;;EL FERROL;RIO DE LA PLATA;FRAGATA;;;GREGORIO DE JOSE Y LLAÑOS;ALFEREZ DE FRAGATA;;;;;Unknown;MEDICION Y DATOS DEL MEDIODIA MENOS DESCRIPTORES QUE SUELEN SER DEL ANOCHECER;Diario de navegación desde El Ferrol al Río de la Plata del alférez de fragata don Gregorio de José y Llaños hecho en la fragata Santa Perpetua del mando del cap. José Postillo desde el 1-2-1774. SIGUEN LA CARTA HOLANDESA;Unknown;Unknown;DESCONOCIDO;;;360 degrees;17740201;;;;;;;;48.00;;;;0;;;;;;;;;;;;;SSE;SSE;CALMOSO;;CALMOSO;;HORIZONTES CLAROS;;;;;;0;0;0;0;0;0;0;ALGO PICADA DEL VIENTO;;;;;;;;;;;0;;0;;0;;0;;0;;0;0;;;0
1790 7 31700-1518 28258 0106    10      926ES 1225123                                                       99 0 MNM;Archivo Museo Naval Madrid;Madrid;Spain;;;;;MS. 204;Spanish;;;EL FERROL;RIO DE LA PLATA;FRAGATA;;;GREGORIO DE JOSE Y LLAÑOS;ALFEREZ DE FRAGATA;;;;;Unknown;MEDICION Y DATOS DEL MEDIODIA MENOS DESCRIPTORES QUE SUELEN SER DEL ANOCHECER;Diario de navegación desde El Ferrol al Río de la Plata del alférez de fragata don Gregorio de José y Llaños hecho en la fragata Santa Perpetua del mando del cap. José Postillo desde el 1-2-1774. SIGUEN LA CARTA HOLANDESA;Unknown;Unknown;DESCONOCIDO;;;360 degrees;17740201;;;;;;;;;;;;0;;;;;;;;;;;;;NO1/4N;NO 1/4 N;BONANCIBLE;;BONANCIBLE;;HORIZONTES CARGADOS. POR LA NOCHE EMPEZO A RELAMPAGUEAR MUY A MENUDO Y CON MUCHA VIVEZA. A LAS 3:00 ENTRO UNA NUBE MUY HORRIBLE Y NEGRA POR EL SO Y LO;;;;;;0;0;0;0;0;0;0;LLANA;;;;;;;;;;;0;;0;;0;;0;;0;;0;0;;;0
```

# Processing text data in R

- brutal force: `substring()`
- smarter: `read.fwf()`
- too slow!!

# But R is sweet in web scraping

- variable descriptions at <http://www.ucm.es/info/cliwoc/content/CLIWOC15all.htm>
- we can easily get the widths of variables to read the first part of the data

# Scrape the data description

``` {r desc-page}
```

# Field widths

``` {r variable-widths}
```

# Variable names

``` {r variable-names}
```

# (G)Awk is awesome

```awk
# save to fix2pip.awk
BEGIN  {
  FIELDWIDTHS = "4 2 2 4 5 6 2 1 1 1 1 1" \
    " 2 2 9 2 1 3 1 3 1 2 2 1 5 1 3 1 4" \
    " 1 4 1 4 2 4 1 1 1 1 1 1 1 2 2 2 2" \
    " 2 2 2 2 1 10000" 
}
{
  for (i=1; i <= NF; i++) {
    if (i < NF) {
      sub(/^[ ]+/, "", $i)
      sub(/[ ]+$/, "", $i)
      printf "%s|", $i
    } else {
      gsub(";", "|", $i)
      printf "%s\n", $i
    }
  }
}
```

# Neat Awk One-liners

- <http://www.pement.org/awk/awk1line.txt>
- what does `awk NF` mean?

# Run gawk and R

```bash
gawk -f fix2pipe.awk CLIWOC15 > CLIWOC15pipe
```

But text data is really awful! (lack of rigorous field separators)

``` {r read-data, eval=FALSE}
```

# I have to turn to MS products

- MS Access 2000 database
- exported to CSV using `mdbtools`
    
    ```bash
    wget http://www.knmi.nl/cliwoc/download/CLIWOC15_2000.zip
    unzip CLIWOC15_2000.zip
    mdb-export CLIWOC15_2000.mdb CLIWOC15 > CLIWOC15.csv
    bzip2 CLIWOC15.csv
    ```
    
    Now we get a compressed CSV file `CLIWOC15.csv.bz2` (one copy at <http://xie.public.iastate.edu/CLIWOC15full.csv.bz2>)

# Yeah, CSV!

``` {r read-csv}
```

# Clean the data a little bit

``` {r subset-data}
```

# Traces of Captain Cook (and others)

``` {r map-all, fig.keep='none'}
```

# All possible locations

``` {r map-all-plot, ref.label='map-all', echo=FALSE, fig.width=10, fig.height=6}
```

# Animations

``` {r year-changes, eval=FALSE}
```

# A short video

<iframe src="https://player.vimeo.com/video/40619253?title=0&amp;byline=0&amp;portrait=0" width="800" height="460" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe>

# Acknowledgement

- animation ideas are not mine; they are from [Ben Schmidt's blog](http://sappingattention.blogspot.co.uk/2012/04/visualizing-ocean-shipping.html)
- big thanks to people behind the CLIWOC project putting together the data

# Tools

The slides were made by the R package [**knitr**](https://yihui.org/knitr/) and a great document converter [pandoc](http://johnmacfarlane.net/pandoc) which converted the markdown output from **knitr** to [DZslides](http://paulrouget.com/dzslides/). Markdown source at <https://yihui.org/slides/stat585x-shipping-yihui-xie.Rmd>, and you can compile it by the function `knit()` in **knitr**.

# Long live the RR!

``` {r sessionInfo}
sessionInfo()
```
