---
title: "Using projectData"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Using projectData}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Why projectData?

R packages are an effective way to distribute research data. A package carries a
version, documentation for each dataset, and a defined installation location,
and R already provides mature ways to reach the contents: `data()`, lazy
loading of package data, and namespace-qualified access such as `pkg::object`.
For a single data package these tools are usually all that is needed, and
projectData is designed to work alongside them, not to replace them.

Difficulties tend to appear when an analysis depends on several independently
distributed collections. Data may be divided between a primary project package
and supplementary or reference packages, and different collections may define
objects with the same name. Attaching packages or calling `data()` for each one
leaves the choice of source to attachment order or to repeated manual
qualification, which is hard to keep consistent across a project. Some
resources are also large or stored in formats that ordinary deserialization
cannot restore, such as disk-backed matrices, so they need a format-specific
restoration step and should stay unopened until they are used. Finally, it is
often useful to ask what a project has available, and which installed package
version and library each object came from, without reading any data. This
matters most when several versions or libraries are present. These are
workflow and coordination problems. They do not reflect a limitation of base
R, and many projects never meet them.

projectData is a small coordination layer over installed data packages. Calling
`data_attach()` on a package registers it as a named
*collection* and records an inventory of its resources, along with the package
name, version, and the installation root that was resolved. Registering reads
package metadata only; no data values are read. Objects are then read on first
access, and `data_get()` can return an object from a named collection or
package without attaching anything. When collections share a name, projectData does
not pick a winner by attachment order. You can designate a primary collection,
state a preference for individual objects, or add suffix aliases. An unresolved
conflict produces an error that lists the candidates. Attachment is optional
and happens only when you request it, so objects can also be reached without
adding names to the search path. `data_catalog()` and `data_info()` report
provenance from the recorded metadata alone. Ordinary data packages need no
changes to work this way. An optional, validated manifest can describe special
files or object mappings, and a versioned loader interface lets an *adapter*
supply the discovery and opening steps for a specialized storage format.

This division keeps the core package light. The core handles registration,
resolution, caching, and provenance, and it reads ordinary datasets and
explicitly described RDS files. Format-specific code lives in separate adapter
packages that you choose to install. For example, the separate `projectData.monocle3` package restores Monocle 3
`cell_data_set` objects backed by BPCells matrices, so that projectData does not
depend on monocle3 or BPCells. See the [adapter vignette](adapters.html) for how
adapters work. Missing or unavailable adapters affect only
the resources that need them.

projectData is not a package manager. It never installs or updates data
packages, and it does not replace distribution through CRAN, Bioconductor,
Zenodo, or similar channels. It cannot make compatible software versions or
external backing files available, and it does not by itself make an analysis
reproducible. It is also not a data-processing framework. If `data()` and
namespace-qualified access already meet your needs, you can keep using them.
projectData is intended for projects that need consistent access to several
collections, deferred opening of large or specialized resources, and a record of
where each object came from.

The normal workflow has three steps:

1. **Install** the data package.
2. **Attach** it.
3. **Use** its objects, interactively or in a script.

Need another data package later? Attach it too. If the packages define the same
name, projectData asks you to decide which one you mean (see
[Conflicts](#conflicts)).

## 1. Install

Installing is ordinary R package installation and is not done by projectData. A
data package distributed as a tarball (for example from Zenodo) is installed
like any source package:

```r
install.packages("path/to/rnaseqData_1.0.0.tar.gz", repos = NULL, type = "source")
```

To keep this vignette offline, the next chunk builds two tiny throwaway data
packages in a temporary library. In real use, skip this chunk and omit the
`lib.loc` argument below.

```{r fixtures}
library(projectData)
source(system.file("examples", "fixtures.R", package = "projectData"))

lib <- fixture_library()
fixture_package(lib, "rnaseqData", "1.0.0", lazy = TRUE, datasets = list(
  counts_rna = list(counts_rna = matrix(1:6, 2)),
  meta_rna   = list(meta_rna = data.frame(sample = c("a", "b")))
))
fixture_package(lib, "proteinData", "2.1.0", lazy = FALSE, datasets = list(
  counts_prot = list(counts_prot = matrix(7:12, 2)),
  meta_prot   = list(meta_prot = data.frame(sample = c("a", "b")))
))
```

## 2. Attach

`data_attach()` takes the name of an installed data package. It records what
the package contains and puts its objects on the search path under their own
names. This reads package metadata only: no data values are read, and nothing is
installed. Nothing is attached unless you call this function.

```{r attach}
data_attach("rnaseqData", lib.loc = lib)
```

## 3. Use

Use the objects by their ordinary names. The first use of an object reads it.

```{r use}
data_config(messages = TRUE)
counts_rna
```

The message names the resource and says it is opened from the installed package;
projectData never installs packages. The same code works in a script after the
`data_attach()` call.

Attach another package whenever you need it. Objects already in use are not
affected.

```{r attach-second}
data_attach("proteinData", lib.loc = lib)
counts_prot
```

## Looking at what is attached

These read only recorded metadata. They never open any data.

```{r catalog}
data_catalog()[, c("collection", "exposed_name", "resource", "state")]
```

`state` is `unopened` until the object has been used, then `ready`. The
`resource` column is the loadable unit, which can differ from the object name.
A resource that produces several objects is opened once for all of them.

`data_info("counts_rna")` returns the full provenance row for a name: package,
version, installed location, source file, where it was discovered (`origin`), and
whether its object names have been checked (`mapping`).
`data_help("counts_rna")` opens the help page of the original topic in the
source package. It is separate from inspection and never runs implicitly.

## Getting objects explicitly

Bare names are convenient, but `data_get()` returns an object without relying on
what is on the search path, so it is the more robust choice inside functions
and scripts. (Attached bindings can also be evaluated by IDE completion or
previews; see `system.file("ide-repro", package = "projectData")`.)

```{r explicit}
data_get("meta_prot")
data_get("meta_rna", collection = "rnaseqData")
```

Give several names to open several objects in one call. This is a good way to
open large objects at the start of a batch job: the first-access messages
appear together, before the rest of the analysis runs.

```{r several}
str(data_get(c("meta_rna", "meta_prot")))
```

## Getting one object without attaching

Sometimes you want a single object and no new names on the search path, much as
you would write `package::object` for an exported object. Give `data_get()` the
`package`:

```{r get-unattached}
fixture_package(lib, "imagingData", "0.3.0", lazy = TRUE, datasets = list(
  counts_img = list(counts_img = matrix(13:18, 2))
))

counts_img <- data_get("counts_img", package = "imagingData", lib.loc = lib)
counts_img
"imagingData" %in% data_collections()$package[data_collections()$attached]
```

There is nothing to register first. `data_get()` inventories the package if
necessary (metadata only), then returns the object. It does not attach anything
and does not change collections that are already attached. The name is looked up
in that package itself, so preferences, primary collections and suffixes you set
for attached access do not affect it.

`data_get()` still **opens** the object you ask for; it is not a metadata query.
Use `data_catalog()` or `data_query()` to look without opening. If you attach the
package later, the object is not read again.

## First-access messages

Before each resource is read for the first time, projectData prints a message
naming the collection, the objects, the size of the source file when it is a
single file, and the loader. For large data this is the moment to notice what is
about to load. Turn the messages off with `data_config(messages = FALSE)`; the
setting lasts for the session.

## Looking at a package without attaching it

`data_query()` shows what a package contains, and which names would collide with
the packages you list or with what is already attached, **without attaching
anything or remembering anything**. Use it to look before you attach.

```{r query}
data_detach()

fixture_package(lib, "rnaseqBatch2", "1.0.0", lazy = TRUE, datasets = list(
  counts_rna = list(counts_rna = matrix(101:106, 2)),
  meta_rna   = list(meta_rna = data.frame(sample = c("c", "d")))
))

data_query(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib,
           id = c("batch1", "batch2"))[, c("collection", "exposed_name", "binding")]
nrow(data_collections())
```

The query already shows the problem: `counts_rna` and `meta_rna` exist in both
packages, so their `binding` is `ambiguous`. The query changed nothing: no
collections are attached.

## Conflicts {#conflicts}

When two attached collections define the same name, projectData never resolves
it by attachment order. Attaching still succeeds, and the catalog flags the conflict. Using the
bare name fails with an error that lists the candidates. Use `id` to give each
collection a name of your own:

```{r conflict, error = TRUE}
data_attach(c("rnaseqData", "rnaseqBatch2"), lib.loc = lib, id = c("batch1", "batch2"))
counts_rna
```

There are three ways to resolve conflicts, and they can be combined.

**Per-object preference.** Choose the collection for a particular name. This is
the right tool when an object from one package is always preferred over an
object with the same name in another package.

```{r prefer}
data_prefer("counts_rna", "batch1")
counts_rna
```

Other contested names stay unresolved until you decide them:

```{r prefer-others, error = TRUE}
meta_rna
```

**A primary collection.** If one collection should win *all* its contested
names, designate it. A per-object preference for a non-primary collection
overrides the primary designation for that name. A primary collection can also
be set when attaching: `data_attach("rnaseqData", id = "batch1", primary = "batch1")`.

```{r primary}
data_primary("batch1")
meta_rna
```

**Suffix aliases and qualified access.** Qualified access,
`data_get(name, collection = )`, is the most robust way to access objects within
scripts or functions. It returns the named object and also reaches the copy that
lost a conflict.

```{r get}
data_get("counts_rna", collection = "batch2")
```

A suffix exposes every object in a collection as an alias of the form `<name><suffix>`. Set it when you attach.

```{r suffix}
data_attach("rnaseqBatch2", lib.loc = lib, id = "batch2", suffix = ".batch2",
            replace = TRUE)
counts_rna.batch2
```

Suffix aliases must be unique. A clash is reported when you attach, before
anything is changed.

To clear decisions, use `data_prefer("counts_rna", NULL)`
and `data_primary(NULL)`.

## Script datasets

A package that ships non-lazy data scripts (`data/*.R`) is opened by running
the script, only on first access. The script sees the functions and data of the
standard startup packages (stats, graphics, grDevices, utils, datasets,
methods and base, in the order of an ordinary R session), so `head()` or
`model.matrix()` work unqualified. That set is fixed; it does not follow your
`defaultPackages` option or your search path. Your workspace and any other
attached package are not visible, so qualify calls to other packages
(`otherpkg::fn()`). This isolates names; it is not a security sandbox. See the
"Script datasets" section of `?data_attach`.

## Cleaning up

```{r cleanup}
data_detach()
```

`data_detach()` removes the attached collections and the environment projectData
created, and forgets them. It does not delete files or modify objects you
already hold, so an object kept in a variable can keep reading its backing
files after detachment. That is not a promise about replacing or removing the
installed package: a disk-backed object that references installed files may stop
working then. See `?data_detach`. `data_detach("batch2")` removes just one collection. Packages you
only reached through `data_get(package = )` are forgotten the same way (they
were never on the search path); a later `data_get()` simply opens them again.

## What this does not do

* Class compatibility, software versions, backing files, and platform-specific
  readers still matter; projectData does not guarantee reproducibility.
* Returned objects are shared references to one cached resource, not
  independent copies.
* Related tools: `conflicted` resolves conflicts between package exports and
  `library(include.only = )` selects exports. They work with exported data but
  do not provide collection inventory, deferred access beyond lazy data,
  format-specific restoration, or provenance.