--- title: "Getting started with nadaverse" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Getting started with nadaverse} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") library(httptest2) start_vignette("start") set_redactor(function(x) { gsub_response(x, "index.php/api/catalog/", "") }) ``` Many statistical agencies and international organizations publish survey and census microdata through catalogs built with the NADA software. All of these catalogs share the same web API, so one package can search them all. nadaverse wraps that API in a few functions that return data frames and lists. This guide follows one example from start to finish. You will find a study in the World Bank catalog, read its description, and look at its files and variables. Then you will run the same search on a second catalog. The output shown here comes from saved responses, so the guide builds without an internet connection. When you run the code yourself, the functions call the live catalogs. ```{r library} library(nadaverse) ``` ## Choose a catalog Every function takes a catalog code as its first argument. `catalogs()` lists the codes and prints a short summary. Use `show = FALSE` to get the table without the printout. ```{r catalogs} cats <- catalogs(show = FALSE) cats[, c("catalog", "name")] ``` Codes are not case sensitive, so `"wb"` and `"WB"` mean the same catalog. This guide uses `"wb"` for the World Bank and `"fao"` for the Food and Agriculture Organization. ## Search for studies `search_catalog()` returns one row per study. The `ps` argument sets how many studies come back on a page. ```{r search} res <- search_catalog( "wb", keyword = "Ethiopia Socioeconomic Survey", ps = 5 ) head(res[, c("idno", "nation", "year_start")], 5) ``` Two columns identify a study. `id` is a number and `idno` is a text ID. Every function that takes a study accepts either one. The table also has columns such as `title`, which this guide leaves out to keep the output narrow. If nothing matches, `search_catalog()` gives a warning and returns `NULL` instead of an empty table. ## Read a study description `metadata()` returns the full description of one study as a nested list. ```{r metadata} study <- "ETH_2015_ESS_v03_M" md <- metadata("wb", study) names(md) ``` The `doc_desc` part describes the documentation itself, such as who prepared it and when. The `study_desc` part describes the study: ```{r study-desc} names(md$study_desc) ``` You can follow the list down to a single item with `$`, for example `md$study_desc$title_statement`. ## List the data files A study is usually split into several data files. `data_files()` shows them. ```{r files} files <- data_files("wb", study) head(files[, c("file_id", "file_name", "case_count")], 5) ``` `case_count` is the number of rows in a file. The table has more columns, such as `var_count` (the number of variables) and `description`; the guide shows only a few so the rows stay short. The `file_id` column is what you pass to `data_dictionary()` to look at one file. ## Look up the variables `data_dictionary()` lists the variables, with a name and a label for each. ```{r dictionary} dict <- data_dictionary("wb", study) head(dict[, c("vid", "name", "labl")], 5) ``` To see only one file, pass its ID: ```{r dictionary-file, eval = FALSE} data_dictionary("wb", study, file_id = files$file_id[[1]]) ``` Large studies have thousands of variables, so it often pays to start with one file. ## Try another catalog The functions work the same way on every catalog. Only the catalog code changes. ```{r fao} fao <- search_catalog("fao", keyword = "food insecurity", ps = 5) head(fao[, c("idno", "nation", "year_start")], 5) ``` Column names come from each catalog's API, so a column that exists in one catalog can be missing in another. Check `names()` before you rely on a column in code that runs across catalogs. ## Where to go next - `vignette("searching", package = "nadaverse")` covers every `search_catalog()` argument, with examples. - `vignette("studies", package = "nadaverse")` goes deeper into metadata, files and dictionaries. - `vignette("catalogs", package = "nadaverse")` explains how the catalogs differ, what to do when one fails, and the user agent setting. ```{r teardown, include = FALSE} end_vignette() ```