Metadata-Version: 2.4
Name: afar
Version: 0.7.0
Summary: Run code on a Dask cluster via a context manager or IPython magic
Author: afar contributors
Author-email: Erik Welch <erik.n.welch@gmail.com>
Maintainer-email: Erik Welch <erik.n.welch@gmail.com>
License-Expression: BSD-3-Clause
Project-URL: homepage, https://github.com/eriknw/afar
Project-URL: repository, https://github.com/eriknw/afar
Project-URL: changelog, https://github.com/eriknw/afar/releases
Keywords: dask,distributed,remote,context manager,ipython,jupyter,magic
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: IPython
Classifier: Framework :: Jupyter
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: distributed>=2026.1.1
Requires-Dist: innerscope>=0.8.0
Provides-Extra: test
Requires-Dist: pytest>=9.0.3; extra == "test"
Requires-Dist: ipython>=8.10; extra == "test"
Dynamic: license-file

# **Afar**

[![conda-forge](https://img.shields.io/conda/vn/conda-forge/afar.svg)](https://anaconda.org/conda-forge/afar)
[![pypi](https://img.shields.io/pypi/v/afar.svg)](https://pypi.org/project/afar/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/afar)](https://pypi.org/project/afar/)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/eriknw/afar/blob/main/LICENSE)
[![Tests](https://github.com/eriknw/afar/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/eriknw/afar/actions)
[![Coverage](https://codecov.io/gh/eriknw/afar/graph/badge.svg)](https://codecov.io/gh/eriknw/afar)

> **_One man's magic is another man's engineering_**<br>
> _Robert A. Heinlein_

<hr>

## Installation

`afar` may be installed with pip:

```bash
pip install afar
```

or with [conda](https://docs.conda.io/en/latest/):

```bash
conda install -c conda-forge afar
```

## What is it?

`afar` allows you to run code on a remote [Dask](https://dask.org/) [cluster](https://distributed.dask.org/en/latest/) using context managers and [IPython magics](#Magic). For example:

```python
import afar
from dask.distributed import Client
client = Client()

with afar.run, remotely:
    import dask_cudf
    df = dask_cudf.read_parquet("s3://...")
    result = df.sum().compute()
```

Outside the context, `result` is a [Dask Future](https://docs.dask.org/en/latest/futures.html) whose data resides on a worker. `result.result()` is necessary to copy the data locally.

Variables are set outside the context at module scope and in IPython or Jupyter. Inside a function, they are set on Python 3.13 and later; on earlier versions, use the dictionary `afar.run` returns (`with afar.run as data, remotely:`), described below.

By default, only the last assignment is saved. One can specify which variables to save:

```python
with afar.run("one", "two"), remotely:
    one = 1
    two = one + 1
```

`one` and `two` are now both Futures. They can be used directly in other `afar.run` contexts:

```python
with afar.run as data, remotely:
    three = one + two

assert three.result() == 3
assert data["three"].result() == 3
```

`data` above is a dictionary of variable names to Futures. It may be necessary at times to get the data from here. Alternatively, you may pass a mapping to `afar.run` to use as the data.

```python
run = afar.run(data={"four": 4})
with run, remotely:
    seven = three + four
assert run.data["seven"].result() == 7
```

If you want to automatically gather the data locally (to avoid calling `.result()`), use `afar.get` instead of `afar.run`:

```python
with afar.get, remotely:
    five = two + three
assert five == 5
```

## Interactivity in Jupyter

There are several enhancements when using `afar` in Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend.

The rich repr of the final expression will be displayed if it's not an assignment:

```python
with afar.run, remotely:
    three + seven
# displays 10!
```

As in IPython, a final expression that ends with `;`, or whose value is `None`, is not displayed.

Printing is captured and displayed locally and asynchronously:

```python
with afar.run, remotely:
    print(three)
    print(seven, file=sys.stderr)
# 3
# 7
```

These are done asynchronously using `ipywidgets`. Errors raised remotely are shown the same way, as tracebacks.

### Magic!

First load `afar` magic extension:

```python
%load_ext afar
```

Now you can use `afar` as line or cell magic. `%%afar` is like `with afar.run, remotely:`. It can optionally accept a list of variable names to save:

```python
%%afar x, y
x = 1
y = x + 1
```

and

```python
z = %afar x + y
```

`z` is a Future of the result (or the result itself with `%afar -g`, which uses `afar.get`).

### Custom Magic

You can create your own IPython magic like `%afar` that has arguments baked in:

```python
afar.new_magic("on_gpus", where=remotely(resources={"GPU": 1}))
```

then:

```python
%%on_gpus
import dask_cudf
df = dask_cudf.read_parquet("s3://...")
result = df.sum().compute()
```

## Is this a good idea?

I don't know, but it sure is a joy to use 😃 !

For motivation, see https://github.com/dask/distributed/issues/4003

It's natural to be skeptical of unconventional syntax. And magic. `afar` is both unconventional and magical, yet it also works well and is surprisingly _fun_! Why not give it a try to see what you think?

We're still exploring the usability of `afar` [and want to hear what you think](https://github.com/eriknw/afar/discussions). As you're learning `afar`, please ask yourself questions such as:

- can we spell anything better?
- does this offer opportunities?
- what is surprising?
- what is lacking?

Here's an example of an opportunity:

```python
on_gpus = afar.remotely(resources={"GPU": 1})

with afar.run, on_gpus:
    ...
```

This now works! Keyword arguments to `remotely` will be passed to [`client.submit`](https://distributed.dask.org/en/latest/api.html#distributed.Client.submit).

I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)

## Caveats and Gotchas

### Repeatedly copying data

`afar` automatically gets the data it needs--and only the data it needs--from the outer scope
and sends it to the Dask cluster to compute on. Since we don't know whether local data has been modified
between calls to `afar`, we serialize and send local variables every time we use `run` or `get`.
This is generally fine: it works, it's safe, and is usually fast enough. However, if you do this
frequently with large-ish data, the performance could suffer, and you may be using
more memory on your local machine than necessary.

With Dask, a common pattern is to send data to the cluster with `scatter` and get a `Future` back. This works:

```python
A = np.arange(10**7)
A = client.scatter(A)
with afar.run, remotely:
    B = A + 1
# A and B are now both Futures; their data is on the cluster
```

Another option is to pass `data` to `run`:

```python
run = afar.run(data={"A": np.arange(10**7)})
with run, remotely:
    B = A + 1
# run.data["A"] and B are now both Futures; their data is on the cluster
```

Here's a nifty trick to use if you're in an IPython notebook: use `data=globals()`!

```python
run = afar.run(data=globals())
A = np.arange(10**7)
with run, remotely:
    B = A + 1
# A and B are now both Futures; their data is on the cluster
```

### Mutating remote data

As with any Dask workload, one should be careful to not modify remote data that may be reused.

### Mutating local data

Similarly, code run remotely isn't able to mutate local variables. For example:

```python
d = {}
with afar.run, remotely:
    d['key'] = 'value'
# d == {}
```

## _✨ This code is highly experimental and magical! ✨_
