Package {rdomains}

rdomains: Classify Domains by their Content

Description

Want to know what kind of content is carried on a domain? Get the results quickly using rdomains. The package provides access to virustotal API, UT-Capitole, archived lists, OpenAI models, Anthropic Claude models, and a historical model trained on Shallalist data. Use pie_cat() to classify fetched content through the Python piedomains text model.

Details

To learn how to use rdomains, see this vignette: ../doc/rdomains.html.

Author(s)

Gaurav Sood

See Also

Useful links:

• https://themains.github.io/rdomains/

• https://github.com/themains/rdomains

• Report bugs at https://github.com/themains/rdomains/issues

Probability that Domain Hosts Adult Content Based on features of Domain Name and Suffix alone.

Description

Uses a historical ML model that uses keywords in the domain name and suffix to predict probability that the domain hosts adult content. For more information see https://github.com/themains/keyword_porn

Usage

adult_name_probability(domains)

Arguments

Details

The bundled model artifact has not changed since November 2017. Its probabilities have not been validated on current domains. It uses only the name and suffix, not page content; use pie_cat() for content-based classification. Feature encoding remains the training-era encoding rather than silently changing a fitted model.

Value

A tibble with input identity, p_adult, model identity and artifact date.

Examples

## Not run: 
adult_name_probability("http://www.google.com")

## End(Not run)

Empty the page cache

Description

Empty the page cache

Usage

cache_clear(dir = NULL)

Arguments

Value

Invisibly, the number of cached pages removed.

Examples

## Not run: 
cache_clear()
cache_clear(rdomains_cache_dir())

## End(Not run)

Classification result contract

Description

Classifiers preserve input order and return one row per input. input_id is the one-based position in that call; retain it when saving and resuming a batch. status is ok, no_match or failed. An absent lookup membership is not a failed request. category is populated only when there is one selected label; labels preserves every label in a tibble with category, score and vendor. Unknown scores are NA, not probabilities inferred from list membership.

Details

provider identifies the classifier or list; evidence_type distinguishes page content, archived content, name inference and list membership. details retains source-specific records, including the collected page for piedomains results. Model and source identifiers are reported where known; missing dates stay missing.

Value

A tibble with input identity, provider, evidence, category, confidence, labels, status, stage, error_code, retryable, http_status, model_id, model_revision, source_last_published, source_fetched_at and details.

Get Category from Anthropic Claude

Description

Infers a category from the domain name using Anthropic's Claude API. No page content is fetched. The answer reflects the provider's knowledge and the domain name, not a verified observation of the current website.

Usage

claude_cat(
  domains,
  api_key = NULL,
  categories = NULL,
  model = "claude-haiku-5-5",
  delay = 0.5
)

Arguments

Value

A tibble following classification_results, including provider failures and model identity.

Examples

## Not run: 
claude_cat("google.com")
claude_cat(c("google.com", "facebook.com"))
claude_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

Fetch homepage HTML and text for domains

Description

Every requested domain comes back, in order, whether or not it was reachable. Failures carry a code from fetch_error_codes() rather than a bare NA.

Usage

collect_content(
  domains = NULL,
  delay = 1,
  timeout = 10,
  max_bytes = 10 * 1024^2,
  obey_robots = TRUE,
  max_crawl_delay = 30,
  max_redirects = 5,
  cache_dir = NULL,
  cache_ttl = 7 * 86400,
  cache_max_size = 1024^3,
  allow_hosts = character(),
  archive_fallback = FALSE,
  archive_date = NULL,
  user_agent = rdomains_user_agent()
)

Arguments

Details

The crawler identifies itself as ⁠rdomains/<version>⁠, obeys robots.txt including Crawl-delay, spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.

Value

A tibble with one row per input: input_id, input, domain_name, archive_error_code, status, stage, error_code, retryable, http_status, final_url, fetched_at, content_bytes, title, description, lang, text, html, n_tokens, page_state, block_vendor, robots_allowed, from_cache, source, snapshot_timestamp, source_last_published. Live pages have no inferred publication date; fetched_at records retrieval.

See Also

fetch_report() to summarise the run, page_signals() for what the page states are, source_vintage() for how a live fetch compares with the static lists.

Examples

## Not run: 
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)

# retry only what is worth retrying
again <- collect_content(res$input[res$retryable])

## End(Not run)

Get Category from DMOZ

Description

Fetches category (or categories) of content hosted by a domain according to DMOZ. The function checks if path to the DMOZ file is provided by the user. If not, it looks for dmoz_domain_category.csv in the working directory. It also returns results for prominent subdomains.

Usage

dmoz_cat(domains, use_file = NULL)

Arguments

Details

DMOZ closed in March 2017 and the snapshot shipped here is from 2015, so these labels describe the domain as it was a decade ago. The returned source_last_published column carries that vintage when the file has verified download metadata. A warning is issued once per session. See source_vintage.

Publication dates come from the .metadata.json sidecar written by the downloader, checked against the file contents. source_fetched_at is the download time. Files without matching metadata return NA dates; filesystem timestamps are not used.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
dmoz_cat(domains = "http://www.google.com")
dmoz_cat(domains = c("http://www.google.com", "http://plus.google.com"))

## End(Not run)

Reasons a domain can fail to get a category

Description

The closed set of error_code values that collect_content() and pie_cat() can return, with the pipeline stage each belongs to and whether it is worth retrying.

Usage

fetch_error_codes()

Details

Shared strings retain their meaning in piedomains. R-specific acquisition failures and retry-policy exceptions are recorded in the versioned contract.

Value

A tibble with columns code, stage, retryable and description.

See Also

fetch_report() to summarise an actual run, source_vintage() for the provenance of the static label sources.

Examples

fetch_error_codes()

# which failures are worth trying again?
subset(fetch_error_codes(), retryable)

Summarise the outcome of a fetch or classification run

Description

A pure function over the returned tibble, deliberately not a stored summary. A snapshot goes stale the moment you filter the rows; recomputing means fetch_report(subset(x, retryable)) is always correct.

Usage

fetch_report(x)

Arguments

Value

A one-row tibble: n, ok, failed, no_match, retryable, and list-columns by_stage, by_reason and missing (the domains that produced nothing).

Examples

rows <- data.frame(
  domain_name = c("a.com", "b.com", "c.com"),
  status = c("ok", "failed", "failed"),
  stage = c("infer", "fetch", "process"),
  error_code = c(NA, "timeout", "thin_content"),
  retryable = c(FALSE, TRUE, FALSE)
)
fetch_report(rows)

Get DMOZ Data

Description

Downloads archived DMOZ (Open Directory Project) data. DMOZ was discontinued in March 2017. This function downloads our preserved 2015 DMOZ snapshot. For more details, check: https://github.com/themains/rdomains/tree/master/data-raw/dmoz/

Usage

get_dmoz_data(outdir = ".", overwrite = FALSE)

Arguments

Value

Invisibly, the absolute path to the downloaded file.

References

https://archive.org/details/dmoz-rdf-20150327

Examples

## Not run: 
get_dmoz_data()

## End(Not run)

Get Shalla Data

Description

Shallalist service was discontinued in January 2022. This function downloads the last archived copy (from 1/14/22) that we have preserved on GitHub. The original service at shallalist.de is no longer available. Downloads, unzips and saves the final version of shallalist data. By default, saves shalla data as shalla_domain_category.csv.

Usage

get_shalla_data(outdir = ".", overwrite = FALSE)

Arguments

References

https://web.archive.org/web/20210502020725/http://www.shallalist.de/

Examples

## Not run: 
get_shalla_data()

## End(Not run)

Get Steven Black's Host List Data

Description

Downloads the latest version of Steven Black's unified hosts file. The hosts file contains domains known for serving ads, malware, and tracking.

Usage

get_stevenblack_data(outdir = ".", variant = "base", overwrite = FALSE)

Arguments

Value

Invisibly, the absolute path to the downloaded file.

References

https://github.com/StevenBlack/hosts

Examples

## Not run: 
get_stevenblack_data()
get_stevenblack_data(variant = "all")

## End(Not run)

Download the UT-Capitole blacklists

Description

Downloads and flattens the University of Toulouse (UT1) blacklists into a single ⁠domain,category,usage⁠ CSV. These are the maintained successor to Shallalist: where Shallalist last published in January 2022, UT1 is updated continuously.

Usage

get_ut1_data(outdir = "./", overwrite = FALSE)

Arguments

Details

The archive carries ~69 categories under its own French-influenced vocabulary (drogue, publicite, tricheur), which does not line up with Shallalist's names. ut1_cat() therefore reports UT1's category verbatim rather than mapping it onto another taxonomy and losing the distinction.

A domain may appear in more than one category. Every membership is retained; duplicate identical records are removed and rows are sorted by domain and category.

Value

Invisibly, the path to the written CSV.

References

https://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz

See Also

ut1_cat(), and source_vintage() for how this compares with the discontinued sources.

Examples

## Not run: 
get_ut1_data()
ut1_cat("example.com")

## End(Not run)

ML Model

Description

ML Model

Usage

glm_shalla

Format

A list

Author(s)

Gaurav Sood

Source

ML model based on shallalist using keywords and domain suffixes,

Extract text, title, description and language from HTML

Description

Extract text, title, description and language from HTML

Usage

html_text_content(html)

Arguments

Value

A list with text (cleaned, lowercased), title, description and lang. Missing elements are NA_character_; text is "" when nothing could be extracted.

See Also

page_signals(), which uses this to decide whether a page is classifiable.

Examples

html <- "<html lang='en'><head><title>Example</title>
  <meta name='description' content='A demo page'>
  <script>ignored()</script></head>
  <body><p>Hello   World</p><style>p{}</style></body></html>"
html_text_content(html)

Detect soft- and hard-news keywords in URLs

Description

Based on a slightly amended version of the regular expression used to classify news, and non-news in: “Exposure to ideologically diverse news and opinion on Facebook” by Bakshy, Messing, and Adamic. Science. 2015.

Usage

news_url_signals(urls)

Arguments

Details

Amendment: sport rather than sports

URL containing any of the following words is classified as soft news: "sport|entertainment|arts|fashion|style|lifestyle|leisure|celeb|movie|music|gossip|food|travel|horoscope|weather|gadget" # nolint: line_length_linter.

URL containing any of following words is classified as hard news: "politi|usnews|world|national|state|elect|vote|govern|campaign|war|polic|econ|unemploy|racis|energy|abortion|educa|healthcare|immigration" # nolint: line_length_linter.

Note that it is based on patterns existing in a small set of domains. See paper for details.

Value

A tibble with input_id, input, soft_news and hard_news. The two flags are independent; neither is a general news/non-news decision.

References

https://www.science.org/doi/10.1126/science.aaa1160

Examples

## Not run: 
news_url_signals("http://www.bbc.com/sport")
news_url_signals(c("http://www.bbc.com/sport", "http://www.washingtontimes.com/news/politics/"))

## End(Not run)

Get Category from OpenAI

Description

Infers a category from the domain name using OpenAI's chat completion API. No page content is fetched. The answer reflects the provider's knowledge and the domain name, not a verified observation of the current website.

Usage

openai_cat(
  domains,
  api_key = NULL,
  categories = NULL,
  model = "gpt-6-luna",
  delay = 0.5
)

Arguments

Value

A tibble following classification_results, including provider failures and model identity.

Examples

## Not run: 
openai_cat("google.com")
openai_cat(c("google.com", "facebook.com"))
openai_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other"))

## End(Not run)

What kind of page is this?

Description

Inspects already-fetched HTML for the three things that are not a classifiable site: an anti-bot interstitial, a domain-parking placeholder, and a server's "nothing here" page. No network access.

Usage

page_signals(html, text = NULL, domain = "", status = NULL)

Arguments

Details

parked and unavailable are answers, not failures – they are facts about the domain, plainly stated in the page, that a caller can act on. They are also free: no classification service needs to be consulted.

Value

A one-row tibble: page_state (one of "content", "blocked", "parked", "unavailable", "thin"), blocked, block_vendor, block_reason, parked, unavailable, thin, n_tokens.

See Also

fetch_error_codes() for how these map onto run outcomes.

Examples

# A Cloudflare challenge, not a website
page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>")

# A parking page
page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")

# A real page that merely embeds reCAPTCHA is not blocked
page_signals(paste0(
  "<html><title>Reddit</title><body>",
  paste(rep("real discussion content", 200), collapse = " "),
  "<script src='recaptcha/api.js'></script></body></html>"
))

Classify fetched HTML with piedomains

Description

Fetches pages with collect_content() and passes their original HTML to piedomains for extraction, tokenization, calibration and classification. The Python package must be version 0.15.0 or newer. This uses its text classifier, not its image or fusion models.

Usage

pie_cat(domains = NULL, pages = NULL, threshold = 0.1, model = NULL, ...)

Arguments

Details

A current fetch does not guarantee a correct current label. The shipped checkpoint has known retail/drugs confusion; see its model card. Archive and cached content retain their collection dates. details preserves the page and its ⁠fetch_*⁠ collection outcomes separately from the terminal classification status, stage, error_code and retryable columns.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

Setup

Install reticulate and declare Python requirements before initializing Python: reticulate::py_require("piedomains==0.15.0", python_version = ">=3.12"). Set RETICULATE_PYTHON to select an existing Python environment. Model weights download from Hugging Face on first use. The Python package pins its default checkpoint; PIEDOMAINS_TEXT_MODEL can select another checkpoint or local path. Results report the resolved model, revision when available, Python package version, extractor, cleaning mode and calibration temperature.

Examples

## Not run: 
pages <- collect_content(c("cnn.com", "wikipedia.org"))
pie_cat(pages = pages)

## End(Not run)

A persistent cache directory for fetched pages

Description

Returns the conventional per-user cache location for the package. It deliberately does not create the directory: pass it to collect_content() to opt in, and until you do nothing is written outside the session's temporary directory.

Usage

rdomains_cache_dir()

Value

Path to the per-user cache directory, as a string.

Examples

rdomains_cache_dir()
## Not run: 
# opt in to caching that survives the session
collect_content("example.com", cache_dir = rdomains_cache_dir())

## End(Not run)

Get Category from Shallalist

Description

Fetches category of content hosted by a domain according to Shalla. The function checks if path to the shalla file is provided by the user. If not, it looks for shalla_domain_category.csv in the working directory.

Usage

shalla_cat(domains, use_file = NULL)

Arguments

Details

Shallalist stopped publishing in 2022, so these labels describe the domain as it was then or earlier, not as it is now. The returned source_last_published column carries that vintage when the file has verified download metadata. A warning is issued once per session. See source_vintage.

Publication dates come from the .metadata.json sidecar written by the downloader, checked against the file contents. source_fetched_at is the download time. Files without matching metadata return NA dates; filesystem timestamps are not used.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
shalla_cat(domains = "http://www.google.com")

## End(Not run)

Report what each category source is and when it was last published

Description

The sources listed here answer from static lists, and those lists have very different vintages. Two of them are no longer published at all. Use this to see publication dates and maintenance status; freshness alone does not establish accuracy.

Usage

source_vintage(source = NULL)

Arguments

Value

A tibble with one row per source: key, source, last_published, status, successor and note.

Examples

source_vintage()
source_vintage("dmoz")

Get Category from Steven Black's Host List

Description

Classifies domains based on Steven Black's unified host list which blocks ads, malware, and tracking domains. The function checks if a domain appears in the blocklist and categorizes it accordingly.

Usage

stevenblack_cat(domains, use_file = NULL)

Arguments

Details

Steven Black's host list is a consolidated list from multiple sources including adaway.org, mvps.org, malwaredomainlist.com, and someonewhocares.org.

Listed hosts receive blocked; absent hosts have status = "no_match". Neither outcome determines safety or a content subclass.

Publication dates come from the .metadata.json sidecar written by the downloader, checked against the file contents. source_fetched_at is the download time. Files without matching metadata return NA dates; filesystem timestamps are not used.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

References

https://github.com/StevenBlack/hosts

See Also

source_vintage for the provenance of every category source

Examples

## Not run: 
stevenblack_cat("doubleclick.net")
stevenblack_cat(c("google.com", "googleadservices.com", "malware-example.com"))

## End(Not run)

Get Category from University Domain List

Description

Fetches university domain json from: https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json # nolint: line_length_linter.

Usage

uni_cat(domains, use_file = NULL)

Arguments

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

Examples

## Not run: 
uni_cat(domains = "http://www.google.com")

## End(Not run)

Get category from the UT-Capitole blacklists

Description

Looks a domain up in the University of Toulouse (UT1) blacklists, the maintained successor to Shallalist. Unlike shalla_cat() and dmoz_cat(), which answer from lists that stopped publishing in 2022 and 2017, this list is updated continuously — so publication and download dates are recorded separately.

Usage

ut1_cat(domains, use_file = NULL)

Arguments

Details

The categories are UT1's own (drogue, publicite, tricheur, agressif, ...) and are reported verbatim. They do not share names with Shallalist's, and mapping one onto the other would discard exactly the distinctions the two taxonomies disagree about.

Publication dates come from the .metadata.json sidecar written by the downloader, checked against the file contents. source_fetched_at is the download time. Files without matching metadata return NA dates; filesystem timestamps are not used.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

See Also

get_ut1_data() to download the data, source_vintage() for how this source compares with the discontinued ones.

Examples

## Not run: 
get_ut1_data()
ut1_cat(domains = c("example.com", "wikipedia.org"))

# what a maintained list says versus one that stopped in 2022
merge(ut1_cat("example.com"), shalla_cat("example.com"), by = "domain_name")

## End(Not run)

Get Category from VirusTotal

Description

Returns category of content from multiple security vendors using the VirusTotal API v3. The function retrieves domain analysis results including categories from various security services. Not all services will have categories for all domains.

Usage

virustotal_cat(domains, api_key = NULL)

Arguments

Details

Get the API Access Key from https://www.virustotal.com/. Either pass the API Key to the function # nolint: line_length_linter. or set the environmental variable: VirustotalToken. Environment variables persist within a R session.

Value

A tibble following classification_results. Source-specific data are retained in details; all source labels are retained in labels.

References

https://docs.virustotal.com/reference/domains

Examples

## Not run: 
virustotal_cat("http://www.google.com")
virustotal_cat(c("google.com", "facebook.com"))

## End(Not run)

Default categories for domain-name inference

Description

Default categories for domain-name inference

Usage

web_categories()

Value

A character vector of category names shared by the LLM adapters.

Examples

web_categories()