Metadata-Version: 2.4
Name: pelutils
Version: 4.3.1
Summary: The Swiss army knife of Python projects
Home-page: https://github.com/peleiden/pelutils
Download-URL: https://pypi.org/project/pelutils/
Author: Asger Laurits Schultz, Søren Winkel Holm
Author-email: asger.s@protonmail.com, swholm@protonmail.com
License: MIT
Keywords: utility,logger,parser,profiling,plotting
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: Free Threading :: 3 - Stable
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: pelutils/_c/hashmap/LICENSE
Requires-Dist: numpy>=2.0
Requires-Dist: gitpython>=3.1.0
Requires-Dist: rich>=10.0.0
Requires-Dist: py-cpuinfo>=8.0.0
Requires-Dist: psutil>=5.8.0
Requires-Dist: matplotlib>=3.3
Requires-Dist: scipy>=1.6
Requires-Dist: tqdm>=4.55
Requires-Dist: pydantic>=2
Requires-Dist: typing_extensions>=4.6
Provides-Extra: docs
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: sphinx-autobuild; extra == "docs"
Requires-Dist: furo; extra == "docs"
Requires-Dist: myst-parser; extra == "docs"
Provides-Extra: dev
Requires-Dist: torch>=2; extra == "dev"
Requires-Dist: pytest==8.4.2; extra == "dev"
Requires-Dist: pytest-cov==7.0.0; extra == "dev"
Requires-Dist: coveralls>=4.0.0; extra == "dev"
Requires-Dist: coverage==7.10.7; extra == "dev"
Requires-Dist: wheel; extra == "dev"
Requires-Dist: setuptools>=60.0.0; extra == "dev"
Requires-Dist: ruff==0.15.8; extra == "dev"
Requires-Dist: basedpyright==1.39.0; extra == "dev"
Requires-Dist: freezegun>=1.5; extra == "dev"
Requires-Dist: ipdb; extra == "dev"
Requires-Dist: sphinx; extra == "dev"
Requires-Dist: sphinx-autobuild; extra == "dev"
Requires-Dist: furo; extra == "dev"
Requires-Dist: myst-parser; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: download-url
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# pelutils

[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![basedpyright](https://img.shields.io/endpoint?url=https://docs.basedpyright.com/latest/badge.json)](https://docs.basedpyright.com)
[![checks](https://github.com/peleiden/pelutils/actions/workflows/checks.yml/badge.svg?branch=master)](https://github.com/peleiden/pelutils/actions/workflows/checks.yml)
[![Coverage Status](https://coveralls.io/repos/github/peleiden/pelutils/badge.svg?branch=master)](https://coveralls.io/github/peleiden/pelutils?branch=master)
[![PyPi](https://img.shields.io/pypi/v/pelutils.svg)](https://pypi.org/project/pelutils/)
[![Python versions](https://img.shields.io/pypi/pyversions/pelutils)](https://img.shields.io/pypi/pyversions/pelutils)
[![image](https://img.shields.io/pypi/l/pelutils.svg)](https://github.com/peleiden/pelutils/blob/master/LICENSE.txt)
[![readthedocs](https://app.readthedocs.org/projects/pelutils/badge/?version=latest)](https://pelutils.readthedocs.io/en/latest/)

**The Swiss army knife of Python projects**

Every project, experiment, or one-off script inevitably ends up reinventing much of
the same plumbing: some way to time a loop, saving and loading data to and from disk
in a convenient and human-readable manner, a decent logger, parsing a config file, a readable table.
`pelutils` bundles the good versions of these so you can get straight to the actual
work. It has no required dependencies beyond the scientific-Python staples, ships
type hints (including `py.typed`), and is easy to start using.

📖 **Full documentation: [pelutils.readthedocs.io](https://pelutils.readthedocs.io)**

## Highlights

- **Logger** — easy-to-use, colourful console output, log files with rotation, automatic
  stacktrace capture, and safe logging from multiple processes.
- **Timer & profiler** — a Matlab-style `tick`/`tock` timer and a near-zero-overhead
  profiler that prints a readable breakdown of where your time goes.
- **`UniversalJsonModel`** — a `pydantic.BaseModel` that can save *any* attribute to a
  human-readable JSON file (numpy arrays, tensors, and other unserialisable types are
  pickled transparently) and load it straight back.
- **`JobParser`** — one parser that unifies command-line arguments and config files,
  with support for running many jobs from a single config and auto-documenting them.
- **`unique`** — a linear-time drop-in for `numpy.unique`, dramatically faster on
  large arrays (backed by a small C extension).
- **Data-science helpers** — a `matplotlib` `Figure` context manager with improved default
  settings over `matplotlib`, histogram binning, reparametrised scipy distributions,
  `z_score`, LaTeX-ready tables, and numpy type aliases.

## Installation

```sh
pip install pelutils
```

`pelutils` supports Python 3.11+. A small subset of functionality can additionally
make use of [`PyTorch`](https://pytorch.org), which must be installed separately.

> **Importing:** every feature lives in its own submodule and must be imported from
> there — e.g. `from pelutils.logging import log`. Only `__version__` is exported at
> the top level. See the [docs](https://pelutils.readthedocs.io) for the full API.

## Logging

Python's built-in `logging` is powerful but fiddly to set up, and a bare `print` gives
you no importance levels, no timestamps, and nothing on disk to look at afterwards. This logger hits
the sweet spot: one `configure` call and you get colour-coded, timestamped output to both
the console and a log file, with severity levels, log rotation, one-line exception logging,
and multiprocessing-safe collection.

```py
from pelutils.logging import log, LogLevels

# Set up the logger by giving it the file to write to
# Omit the path to only print, never write a file
log.configure("train.log")

# If an exeption occurs anywhere in the code inside `log.log_errors`, it is logged with its full, chained stacktrace, then re-raised
with log.log_errors:
    log.section("Training run")  # Highlighted section header
    log(f"Loaded {len(dataset):,} samples")  # Logs at INFO level
    for epoch in range(epochs):
        loss = train_one_epoch()
        log.debug(f"Epoch {epoch}: loss {loss:.4f}")  # Logs at DEBUG level - by default, only saved to the log file, but not printed to the console
        if loss > 1e3:
            log.warning("Loss is diverging")  # Logs at WARNING level
    log.debug("Final weights", model.state_dict().keys())

    save_checkpoint(model)

# Temporarily change or silence the log level
with log.level(LogLevels.ERROR):
    log.warning("Suppressed")

# Rotate the log file by time or size
log.configure("train.log", rotation="day")    # or "1 GB", "hour", ...
```

When using multiprocessing, wrap a worker in `with log.collect():` so its lines are
written together instead of interleaving with other processes. See the
[logging docs](https://pelutils.readthedocs.io/en/latest/api/pelutils.logging.html) for input helpers, multiple loggers,
and more.

## Timing and profiling

When you want to know where a script spends its time, `cProfile` gives you a wall of
function-level numbers, and manual `time.perf_counter()` calls quickly turn into
bookkeeping. `TickTock` sits in between: wrap the sections *you* care about in named,
nestable context managers and print a readable table of totals, hit counts, averages,
and each section's share of its parent's time. The per-profile overhead is tiny, so it
happily lives inside hot loops and long-running jobs.

```py
from pelutils.ticktock import TT

# Time a single block, Matlab style
TT.tick()
model = train_model(data)
print(f"Training took {TT.tock():.1f} s")

# Profile named sections across a loop, nesting them however you like
for image in images:
    with TT.profile("Process image"):
        with TT.profile("Load"):
            img = load(image)
        with TT.profile("Resize"):
            img = resize(img, (224, 224))
        with TT.profile("Inference"):
            predict(model, img)

# Profile loop-body work element by element without manual context managers
for image in TT.profile_loop("Process image", images):
    process(image)

# Or profile the time spent fetching elements from a lazy iterator
for batch in TT.profile_next("Load batch", iter(dataloader)):
    train_on(batch)

# Print a table of hits, total time, and average time for each section
print(TT)
```

`with TT.profile("name", hits=n):` records `n` hits at once, which is handy for very
tight loops or for a block that processes `n` items in parallel. `TT.do_at_interval(...)`
turns the same instance into a throttle for periodic tasks. The default `TT` is a
shared instance; construct your own with `TickTock()` when you need isolation — most
importantly one per thread, as profiling is not thread-safe.

## Serialisation

`UniversalJsonModel` is for storing Pydantic models that contain a mixture of ordinary JSON
values and Python-specific values. Ordinary values remain JSON, so the file structure can be
inspected and diffed. Values that JSON cannot represent, such as NumPy arrays, tensors, and
other Python objects, are stored as base64-encoded pickle payloads.

The `save()` and `load()` methods support convient file-based workflows.
`save()` serialises the model to a JSON file, while `load()` reconstructs the model from one.
Use `to_json_dict()` and `from_json_dict(...)` when the serialized representation belongs inside
another structure or is handled by another storage layer.

```py
import numpy as np
from pydantic import BaseModel
from pelutils.serialization import UniversalJsonModel
from pelutils.types import FloatArray

class Nested(BaseModel):
    label: str

class Result(UniversalJsonModel):
    accuracy: float
    predictions: FloatArray   # numpy arrays are handled automatically
    meta: Nested

result = Result(
    accuracy=0.97,
    predictions=np.arange(5, dtype=np.float16),
    meta=Nested(label="run-1"),
)

result.save("results/run-1.json")
result = Result.load("results/run-1.json")
```

`pretty_json` is a string-returning formatter for standalone dictionaries and lists. It fills the
same role as `json.dumps`, with compact nested containers and wrapped primitive lists; write the
returned string to a file when needed. The module also includes JSONL read/write helpers
(`jsonl_dump`, `jsonl_load`, ...) for newline-delimited records.

This format is intended for trusted configuration objects, application state, experiment results, etc., not as a
language-neutral interchange format. Pickle payloads are Python-specific and loading them can
execute arbitrary code. Do not load files from untrusted sources.

## Config and command-line argument parsing

`JobParser` combines typed command-line options with INI config files. CLI values
override config values, which override defaults. Declare `RequiredArg` for values
every job must provide, `OptionalArg` for values with defaults, and `Flag` for
booleans. Names are `--kebab-case` on the command line and `snake_case` attributes on
the resulting job.

```py
from pathlib import Path
from pelutils.job_parser import Flag, JobParser, OptionalArg, RequiredArg

parser = JobParser(
    RequiredArg("data-path", help="Training data directory"),
    OptionalArg("learning-rate", default=1e-4, type=float, help="Optimizer learning rate"),
    Flag("fp16", help="Use mixed precision"),
    multiple_jobs=True,
)

for job in parser.parse_jobs():
    print(job.name, job.data_path, job.learning_rate, job.fp16)
    job.write_documentation(Path("runs") / job.name / "arguments.ini")
    # ... run your application with the resolved job values
```

A single config file can define several named jobs (with a shared `[DEFAULT]`
section), and one CLI override applies to all of them:

```console
python main.py --config-file config.ini --learning-rate 5e-5
```

For a single job, drop `multiple_jobs=True` and call `parse_job()` instead. A config
path can target one section directly, e.g. `--config-file config.ini:low-lr`. See the
[job parser docs](https://pelutils.readthedocs.io/en/latest/api/pelutils.job_parser.html) for auto-documentation details.

## Fast `unique`

A linear-time alternative to `numpy.unique`, significantly faster on large arrays. Unlike with `np.unique`, the returned elements are unsorted.

```py
import numpy as np
from pelutils.array import unique

x = np.random.randint(0, 100, size=10_000_000)
values = unique(x)
values, index, inverse, counts = unique(
    x, return_index=True, return_inverse=True, return_counts=True,
)
```

## Data science

### Statistics

Common statistical helpers, plus wrappers around SciPy distributions reparametrised
as in Jim Pitman's *Probability* (rather than SciPy's `loc`/`scale`, which are
unintuitive for many distributions).

```py
from pelutils.stats import expon
from pelutils.stats import z_score

# 95 % confidence interval half-width for a standard normal
half_width = z_score()  # standard deviation is 1

# Upper 1 % critical value for an Exponential(lambda=2)
zval = z_score(alpha=0.01, two_sided=False, distribution=expon(lambda_=2))
```

### Plotting

The `Figure` context manager fixes common `matplotlib` annoyances — sensible default
figure and font sizes, easy styling — and saves and closes the figure for you while
restoring `rcParams` afterwards.

```py
import matplotlib.pyplot as plt
from pelutils.plots import Figure, histogram, normal_binning

with Figure("plot.png", figsize=(20, 10), fontsize=20):
    plt.scatter(x, y, label="Data")
    plt.grid()
    plt.title("Very nice plot")
# Saved to plot.png and closed here

# histogram returns x and y coordinates ready for unpacking
plt.plot(*histogram(data, binning_fn=normal_binning))
```

Three binning functions are provided — `linear_binning`, `log_binning`, and
`normal_binning` (more resolution near the centre of roughly-normal data) — and custom
binning functions are supported. See the
[plotting docs](https://pelutils.readthedocs.io/en/latest/api/pelutils.plots.html).

### Numpy type aliases

Type aliases so you (and your type checker) do not have to track array dtypes by hand.

```py
from pelutils.types import FloatArray, IntArray, BoolArray

def process(features: FloatArray, labels: IntArray, mask: BoolArray): ...
```

## Also included

- `pelutils.misc.Table` — build aligned text tables which can also be easily export to LaTeX with `Table.to_latex()`.
- `pelutils.misc.hardware_info` / `OS` — describe the machine the code runs on.
- `pelutils.misc.git_repo_info` — the repo and commit the code is executing in.
- Assorted file and dict helpers (`reverse_line_iterator`, `except_keys`, ...).
- `pelutils.tests` — pytest helpers: a `UnitTestCollection` base class with a managed temp directory, and a `restore_argv` decorator.

## Supported platforms

Precompiled wheels are provided for most common platforms.
If no wheel matches, `pip` builds from source which requires `<Python.h>` — install it with
`sudo apt install python3-dev` (Ubuntu) or `sudo dnf install python3-devel` (Fedora).

32-bit systems are not fully supported.
Most of the library is Python-only and should work but using any C-dependent code (namely `unique` and `SparseGridBlobDetection`) is likely to end in a segfault.

## Updating and releasing

`pelutils` uses the `master` branch as a stable development branch.
The `release` branch contains the latest version on PyPI.
New features should be made in feature branches from `master` that can be merged into `master` once ready.
When a new release is ready, update `pelutils/__version__.py` and rebase `master` onto `release`.
Then push a new tag from `release` named `vX.Y.Z`.

When new code is merged into `master`, a number of checks are run.
These can be tested locally with the following commands.
```sh
# Linting and formatting
ruff format pelutils tests
ruff check pelutils tests
# Type checking
basedpyright pelutils
# Unit tests
python -m pytest tests --cov pelutils
# Build docs
# Once build, open docs/build/html/index.html in your browser to see them
make -C docs html
```
