--- title: "Author keywords and references" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Author keywords and references} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = FALSE, comment = "") # Console colour carries no meaning on a rendered page. pkgdown turns it on for # its own build, and the escape sequences then reach the reader as literal text, # so colour is switched off here for a plain vignette render and a site build # alike. The fixed width keeps tibbles inside the documentation column. options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE, width = 80) # Print data frames and tibbles as formatted tables. local({ kp <- function(x, ...) { if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x)) knitr::knit_print(knitr::kable(x)) } for (cls in c("data.frame", "tbl_df", "tbl")) { registerS3method("knit_print", cls, kp, envir = asNamespace("knitr")) } }) ``` Keyword co-occurrence and citation-network analysis both need something the Search API alone does not return: a document's author-supplied keywords, and its own reference list. This walks through retrieving both, what each costs, and what your 'Scopus' entitlement needs to cover. The API calls below are shown exactly as you would run them with a configured key. No key is available when the site and the package are built, and Elsevier's terms do not permit redistributing retrieved records in any case, so each call is paired with its output assembled offline from one representative example, the 2015 *Nature* review 'Deep learning'. The corpus of real articles that the other articles run on carries neither author keywords nor reference lists, the two fields this one is about, so it cannot stand in here. The vignette therefore renders the same with or without a key, and `scopus_has_key()` is the switch you would use to run the calls for real. ```{r setup} library(scopusflow) ``` ## Author keywords from a search The Search API's `COMPLETE` view carries an `authkeywords` field alongside the usual title, DOI and date. Requesting it costs no extra request beyond `COMPLETE`'s own smaller page size (25 records per page, against 200 for `STANDARD`), which already means more requests, and so more quota, for the same number of records. ```{r, eval = FALSE} recs <- scopus_fetch("DOI(10.1038/nature14539)", view = "COMPLETE") recs$authkeywords ``` ```{r, include = FALSE} # Representative COMPLETE-view result, assembled offline. authkeywords is the # single string scopus_fetch() returns under view = "COMPLETE", one element # per record, in Scopus' own " | "-delimited form. The bibliographic fields are # the document's own. The 'Scopus' identifier and citation count are left NA, # since nothing on the page prints them and a guess would be a fabrication. recs <- tibble::tibble( entry_number = 1L, scopus_id = NA_character_, doi = "10.1038/nature14539", title = "Deep learning", authors = "LeCun Y.; Bengio Y.; Hinton G.", year = 2015L, date = "2015-05-28", publication = "Nature", citations = NA_integer_, authkeywords = "deep learning | neural networks | representation learning | backpropagation" ) ``` ```{r} recs$authkeywords ``` In development, this field came back `NA` even on a live, otherwise fully-entitled key, for documents that do carry author keywords in 'Scopus' itself. Those documents do have keywords, so what this points to is an entitlement gap specific to this one field. If your own keywords come back all `NA`, it is worth raising with your Scopus/Elsevier account contact. ## References via Abstract Retrieval The reference list is not available from Search under any view. It needs Abstract Retrieval's `FULL` or `REF` view, an entitlement separate from ordinary abstract access and from Search access. This is a per-document endpoint, so retrieving references for *n* documents costs *n* requests against Abstract Retrieval's own, smaller weekly quota, separate from Search's. Each row of the returned `references` data frame is one cited work. ```{r, eval = FALSE} ab <- scopus_abstract( "10.1038/nature14539", view = "FULL", include = c("references", "keywords") ) ab$references[[1]][, c("title", "authors", "source", "year")] ``` ```{r, include = FALSE} # Representative Abstract Retrieval result: one row carrying a `references` # list-column (a data frame of cited works, in scopus_abstract()'s schema) and # the same n_requests / quota attributes the function attaches. refs <- tibble::tibble( position = as.character(1:4), # The 'Scopus' identifier of each cited work is left NA, as it is whenever # the API does not resolve one. Inventing one for the illustration would # misrepresent what a real call returns. id = NA_character_, doi = c("10.1109/5.726791", "10.1038/nature14236", NA, NA), title = c( "Gradient-based learning applied to document recognition", "Human-level control through deep reinforcement learning", "ImageNet classification with deep convolutional neural networks", "Learning representations by back-propagating errors" ), authors = c( "LeCun Y.; Bottou L.; Bengio Y.; Haffner P.", "Mnih V.; Kavukcuoglu K.; Silver D.", "Krizhevsky A.; Sutskever I.; Hinton G.", "Rumelhart D.; Hinton G.; Williams R." ), source = c( "Proceedings of the IEEE", "Nature", "Advances in Neural Information Processing Systems", "Nature" ), year = c(1998L, 2015L, 2012L, 1986L), # NA is what view = "FULL" actually returns for this column. Only view = # "REF" populates a cited work's own citation count. citedbycount = NA_integer_ ) ab <- tibble::tibble( id = "10.1038/nature14539", doi = "10.1038/nature14539", title = "Deep learning", year = 2015L, # scopus_abstract() joins keywords "; ", like its authors column. Only # Search's COMPLETE view uses the " | " form shown above. authkeywords = "deep learning; neural networks; representation learning; backpropagation", references = list(refs) ) attr(ab, "n_requests") <- 1L attr(ab, "quota") <- list(remaining = 24999L, reset = NA_character_) ``` ```{r} ab$references[[1]][, c("title", "authors", "source", "year")] ``` `view = "FULL"` is the recommended default. In development, it returned a complete, correctly counted reference list for every document tried, while `view = "REF"` returned an inconsistent, sometimes-truncated subset, on an otherwise identical request made moments apart. `scopus_abstract()` warns when the number of references returned does not match the document's own reported count, so a partial list never arrives unannounced. The number of requests spent and the remaining Abstract Retrieval quota are attached as attributes, since this endpoint draws on a separate, smaller quota that is easy to exhaust unnoticed. ```{r} attr(ab, "n_requests") # requests spent so far attr(ab, "quota")$remaining # Abstract Retrieval quota left ``` A key or subscription tier that does not cover the requested view raises a `scopus_error_forbidden` condition that names the view, where a generic HTTP failure would leave you guessing. It stops the whole batch too, since entitlement is a property of the account, so the same refusal would only recur on every remaining identifier. For more than a handful of identifiers, pass `cache_dir` so an interrupted or quota-limited batch resumes without re-spending quota already spent. ```r ab <- scopus_abstract( dois, view = "FULL", include = c("references", "keywords"), cache_dir = "abstract-cache" ) ``` ## A minimal, cross-tool corpus `scopus_corpus()` combines a search result with this Abstract Retrieval step, returning a minimal shape close to what OpenAlex's `works` API already returns: `id`, `title`, `year`, `keywords` (a list-column of character vectors) and `references` (a list-column of data frames), where bibliometrix would give you semicolon-joined citation strings. It does not replace [`as_bibliometrix()`], which keeps its own field-mapping convention for users who want that. ```{r, eval = FALSE} recs <- scopus_fetch("DOI(10.1038/nature14539)", max_results = 1) corpus <- scopus_corpus(recs, view = "FULL") corpus$keywords[[1]] nrow(corpus$references[[1]]) ``` ```{r, include = FALSE} # scopus_corpus() pairs each record with its Abstract Retrieval references and # splits scopus_abstract()'s "; "-joined authkeywords string into a character # vector per record. corpus <- tibble::tibble( id = "10.1038/nature14539", title = "Deep learning", year = 2015L, keywords = list(trimws(strsplit(ab$authkeywords, ";", fixed = TRUE)[[1]])), references = list(refs) ) ``` ```{r} corpus$keywords[[1]] nrow(corpus$references[[1]]) ``` The keywords list-column is ready for the co-occurrence analysis this article is named for, unnested and tallied into a per-keyword document count across the corpus. ```{r} sort(table(unlist(corpus$keywords)), decreasing = TRUE) ``` This costs one Abstract Retrieval request per record in `recs`, on top of whatever retrieved `recs` in the first place.