---
title: "Python interop"
format:
  html:
    toc: true
vignette: >
  %\VignetteIndexEntry{Python interop}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

The process at the other end of a channel does not have to be R.
[pymizu](https://github.com/shikokuchuo/pymizu) is the Python counterpart of mizu, built on the same shared-memory core.
A channel connects the two languages as directly as two R sessions.

Pass the peer program as a source string instead of a quoted expression, and set the launcher to `mizu_py_launcher()`.
The launcher checks for the interpreter and pymizu before the channel is created, then spawns `python3 -m pymizu.child`.

In this example, a Python process doubles the Old Faithful eruption durations.
(The Python examples in this vignette run when a `python3` with pymizu and NumPy installed is on the `PATH`.)

``` r
library(mizu)

ch <- mizu_channel(
  r"(
import pymizu
while True:
    x = ch.recv(30)
    if pymizu.is_sentinel(x):
        break
    ch.send(x * 2)
)",
  launcher = mizu_py_launcher()
)

mizu_send(ch, faithful$eruptions)
head(mizu_recv(ch, timeout = 5))
#> [1] 7.200 3.600 6.666 4.566 9.066 5.766
mizu_close(ch)
```

## Large data crosses without copying

Above a size floor, mizu writes the data to a shared-memory region once, and both sides read it in place.
Nothing is serialized, parsed, or copied again.
A large value you receive arrives as an ALTREP view: R reads it straight off the shared pages, and no copy lands in your session.

This is the same zero-copy mechanism you get from another R process.
A large value you send arrives in Python backed by those same pages.

The peer below echoes back whatever it receives.
This channel stays open for the next few examples:

``` r
ch <- mizu_channel(
  r"(
import pymizu
while True:
    x = ch.recv(30)
    if pymizu.is_sentinel(x):
        break
    ch.send(x)
)",
  launcher = mizu_py_launcher()
)

# 1.36 million waiting times, ten megabytes
big <- rep(faithful$waiting, 5000)
mizu_send(ch, big)
back <- mizu_recv(ch, timeout = 5)
identical(back, big)
#> [1] TRUE
```

Apart from the one write into the shared region, no data moved.
Python’s echo came back as a reference, with no payload bytes.
If you send a received view onward, it crosses the same way, as the region’s name alone.
So does an unmodified data frame echoed back.

The zero-copy tier covers every large value: a plain vector (no attributes), a matrix or array, row- or column-major, a character vector, or a nested Python tree of lists, dicts, and arrays.
A large data frame crosses as one shared region in both directions.
The floor is 256 KB for a plain vector on a channel, and 32 KB everywhere else.[^1]

## What crosses

R’s built-in datasets illustrate the range of what crosses.
Each form below comes back from Python `identical()` to what you sent, down to values, shape, class, and labels.

``` r
echo <- function(x) {
  mizu_send(ch, x)
  mizu_recv(ch, timeout = 5)
}

identical(echo(mtcars), mtcars)             # a data frame
#> [1] TRUE
identical(echo(iris), iris)                 # a factor column stays a factor
#> [1] TRUE
identical(echo(state.x77), state.x77)       # a matrix keeps its dimnames
#> [1] TRUE
identical(echo(HairEyeColor), HairEyeColor) # a table keeps class and labels
#> [1] TRUE
identical(echo(islands), islands)           # a named vector keeps its names
#> [1] TRUE
identical(echo(esoph), esoph)               # ordered factors keep their order
#> [1] TRUE

days <- as.Date(paste(1973, airquality$Month, airquality$Day, sep = "-"))
identical(echo(days), days)                  # a Date keeps its meaning
#> [1] TRUE
```

Python sees the natural equivalent throughout: a NumPy array for a vector, a dict for a named list, a `datetime64` array for a `Date`, a `pymizu.Frame` for a data frame.

## What doesn’t cross

Some values have no equivalent on the other side.
`mizu_send()` declines them with a `mizu_error_not_portable` error that points at the offending value, and the channel stays usable.

Among the built-in datasets, `CO2` and `ChickWeight` cross once their groupedData metadata is dropped.
Their `formula` attributes are language objects, and the `units` attribute name is reserved by the interchange format.
The frames themselves, their ordered-factor columns, and the `labels` attribute cross intact.

``` r
tryCatch(mizu_send(ch, CO2), error = conditionMessage)
#> [1] "mizu: value is not portable to the peer (an attribute set (formula, outer, labels) outside the portable whitelist at x)"
mizu_close(ch)
```

`mizu_send()` also declines a `POSIXlt` time, a character matrix, an S4 object, a data.table, and a named character vector or named factor (names ride on the plain atomic vectors only).
From the Python side, the declined types are nested, decimal, `date64`, `time`, and `interval` Arrow columns.

## Data frames and Arrow

A data frame crosses as a `pymizu.Frame`, which `pl.from_arrow(f)` or `pa.table(f)` converts to polars or pyarrow.
A polars, pyarrow, or pandas frame arrives back as a `data.frame`.
A tibble’s class and extra attributes cross unchanged, so an unmodified round trip is `identical()` and a tibble comes back as a tibble.

Any Arrow producer is staged directly from its buffers: an `arrow` Table or RecordBatchReader, a struct-typed `nanoarrow_array_stream`, `DBI::dbGetQueryArrow()`, duckdb, or adbc.
Staging is a single write into the shared region, not a conversion to R values first.

`mizu_as_arrow()` converts a received frame back to Arrow.
Columns without attributes (double, integer, `integer64`, character) convert without copying:

``` r
ch <- mizu_channel(quote(mizu_send(ch, mizu_recv(ch))))
mizu_send(ch, arrow::arrow_table(data.frame(
  eruptions = rep(faithful$eruptions, 500),
  waiting = rep(faithful$waiting, 500)
)))

df <- mizu_recv(ch, timeout = 30)
arrow::as_arrow_table(mizu_as_arrow(df))
#> Table
#> 136000 rows x 2 columns
#> $eruptions <double>
#> $waiting <double>

# a subset keeps the same backing pages
arrow::as_arrow_table(mizu_as_arrow(df["waiting"]))
#> Table
#> 136000 rows x 1 columns
#> $waiting <double>
mizu_close(ch)
```

Sending a stream consumes it, since a RecordBatchReader is fully materialized at send.

The values survive exactly, but the Arrow schema does not.
A dictionary returns as a factor, and unsigned integers widen.
The [dtype matrix](https://shikokuchuo.net/pymizu/user-guide/interop.html#the-dtype-matrix) on the pymizu site has the exact per-dtype correspondence.

## Mixed-language pools

Channels move values.
Pools move computation, and a pool’s workers can be Python processes too.
`mizu_py_launcher()` spawns the workers.

`mizu_call()` builds a language-neutral task: a qualified `"mod.fn"` name or a `.source =` string, plus constant arguments.
`mizu_submit_call()` submits it.
Unnamed arguments become Python’s positional arguments, and named ones become its keyword arguments:

``` r
p <- mizu_pool(2L, launcher = mizu_py_launcher())

t <- mizu_submit_call(
  p,
  mizu_call("numpy.quantile", faithful$eruptions, q = c(0.25, 0.5, 0.75))
)
mizu_collect(t)
#> [1] 2.16275 4.00000 4.45425
quantile(faithful$eruptions, c(0.25, 0.5, 0.75)) # R's own answer
#>     25%     50%     75% 
#> 2.16275 4.00000 4.45425

s <- mizu_submit_call(p, mizu_call(.source = "y = x * 2\ny + 1", x = 20))
mizu_collect(s)
#> [1] 41
```

A `.source =` task runs in a fresh namespace with the named arguments bound as names.
The trailing expression’s value is the result.

A plain `mizu_submit()` on a Python pool fails locally, and the error points to `mizu_submit_call()`.
Task errors cross as `mizu_error_remote` conditions that keep the remote type.

Arguments follow the same rules as channel payloads, zero-copy included.
A large array crosses as a shared view, and a view already in shared memory crosses as its name alone.

The reverse direction works the same way: `pymizu.Pool.create()` with `pymizu.r_launcher()`, and `Pool.submit()` with `pymizu.call()` specs.

## The cross-language map

To map a Python function over an R vector, pass a `mizu_call()` spec as `mizu_map()`’s `.f`.
Each element of `.x` becomes the spec’s first positional argument, or binds as `x` for a `.source =` spec.

`.template`, `.collect = "view"`, and `.seed` work as they do on a same-language map:

``` r
m <- mizu_map(
  p,
  faithful$waiting[1:6],
  mizu_call("numpy.log"),
  .template = numeric(1)
)
mizu_collect(m)
#> [1] 4.369448 3.988984 4.304065 4.127134 4.442651 4.007333
mizu_pool_stop(p)
```

[^1]: Below the floor a value crosses as one quick copy.
    At any size, these types cross: atomic vectors (logical, integer, double, complex, character, raw, and `integer64`, bit64’s exact layout constructed without bit64), strings (`NA_character_` reads as Python `None`, and `None` arrives as `NULL`), lists (a named list becomes a dict), and `Date`, `POSIXct`, and `difftime`.
    A few round trips change the container but not the values.
    A standalone factor returns as character strings, a length-1 `NA` returns as `NULL`, and a logical vector with missing values returns as integer.
    A `POSIXct` timezone becomes `"UTC"`, and `difftime` units become seconds.
    The bit patterns `INT32_MIN` and `INT64_MIN` collide with R’s own `NA` sentinel and read as `NA`.
    A `uint64` past 2^63 - 1 crosses as `NA` with a warning at send (in range, it converts exactly to `integer64`).