---
title: "Custom Activation Function"
output:
    rmarkdown::html_vignette:
        toc: true
vignette: >
  %\VignetteIndexEntry{Custom Activation Function}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE, warning=FALSE, message=FALSE}
knitr::opts_chunk$set(
    eval = identical(Sys.getenv("NOT_CRAN"), "true"),
    fig.width = 7,
    fig.height = 5,
    warning = FALSE,
    message = FALSE
)
library(kindling)
```

# Rationale

The biggest strength of `{kindling}` when modelling neural networks is its versatility — it inherits `{torch}`'s versatility while being human friendly, including the ability to apply custom optimizer functions, loss functions, and per-layer activation functions. Learn more: [https://kindling.joshuamarie.com/articles/special-cases](https://kindling.joshuamarie.com/articles/special-cases).

With `act_funs()`, you are not limited to the activation functions available in `{torch}`'s namespace. Use `new_act_fn()` to wrap any compatible function into a validated custom activation. This feature, however, is only available on version 0.3.0 and above. 

# Function to use

To do this, use `new_act_fn()`. It takes a user-supplied function, validates it against a small dummy tensor at *definition time* (a dry-run probe), and wraps it in a call-time type guard. This means errors surface early — before your model ever starts training.

The function you supply must:

- Accept at least one argument (the input tensor).
- Return a `torch_tensor`.

## Basic Usage

Currently, `nnf_tanh` doesn't exist in the `{torch}` namespace, so `tanh` is not a valid argument to `act_funs()`. With `new_act_fn()`, you can wrap `torch::torch_tanh()` to make it usable. 

Here's a basic example that wraps `torch::torch_tanh()` as a custom activation:

``` r
hyper_tan = new_act_fn(\(x) torch::torch_tanh(x))
```

You can also pass it directly into `act_funs()`, just like any built-in activation:

``` r
act_funs(relu, elu, new_act_fn(\(x) torch::torch_tanh(x)))
```

## Using Custom Activations in a Model

Naturally, functions for modelling, like `ffnn()`, accept `act_funs()` into the `activations` argument. Again, you can pass a custom activation function within `new_act_fn()`, then pass it through `act_funs()`. 

Here's a basic example: 

```{r}
model = ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32, 16),
    activations = act_funs(
        relu,
        silu,
        new_act_fn(\(x) torch::torch_tanh(x))
    ),
    epochs = 50
)
model
```

Each element of `act_funs()` corresponds to one hidden layer, in order. Here, the first hidden layer uses ReLU, the second uses SiLU (Swish), and the third uses Tanh.

You can also use a single custom activation recycled across all layers:

```{r}
ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(new_act_fn(\(x) torch::torch_tanh(x))),
    epochs = 50
)
```

## Skipping the Dry-Run Probe

By default, `new_act_fn()` runs a quick dry-run with a small dummy tensor to validate your function before training. You can disable this with `probe = FALSE`, though this is generally not recommended:

```{r, eval = FALSE}
my_act = new_act_fn(\(x) torch::torch_tanh(x), probe = FALSE)
```

## Naming Your Custom Activation

You can provide a human-readable name via `.name`, which is used in print output and diagnostics:

```{r}
my_act = new_act_fn(\(x) torch::torch_tanh(x), .name = "my_tanh")
```

Here's a simple application:

```{r}
ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(
        relu, 
        new_act_fn(\(x) torch::torch_tanh(x), .name = "hyper_tanh")
    ),
    epochs = 50
)
```

# Error Handling

`new_act_fn()` is designed to fail loudly and early. Common errors include:

1.  Function returns a non-tensor. This will error at definition time:

    ```{r, error = TRUE}
    new_act_fn(\(x) as.numeric(x))
    ```

2.  Function accepts no arguments. This will error immediately:

    ```{r, error = TRUE}
    new_act_fn(function() torch::torch_zeros(2))
    ```

These checks ensure your model's architecture is valid before any data ever flows through it.

# Summary

| Feature | Details |
|---|---|
| Wraps any R function | Must accept a tensor, return a tensor |
| Dry-run probe | Validates at definition time (`probe = TRUE` by default) |
| Call-time guard | Type-checks output on every forward pass |
| Compatible with `act_funs()` | Use alongside built-in activations freely |
| Closures supported | Parametric activations work naturally |