Metadata-Version: 2.5
Name: micm-nlp
Version: 0.3.1
Summary: NLP research toolkit built on HuggingFace Transformers
Project-URL: Homepage, https://github.com/bmikaberidze/micm-nlp
Project-URL: Repository, https://github.com/bmikaberidze/micm-nlp
Project-URL: Issues, https://github.com/bmikaberidze/micm-nlp/issues
Author-email: Beso Mikaberidze <beso.mikaberidze@gmail.com>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: accelerate
Requires-Dist: aenum
Requires-Dist: datasets<6
Requires-Dist: datasketch
Requires-Dist: evaluate
Requires-Dist: lightning
Requires-Dist: matplotlib
Requires-Dist: nltk
Requires-Dist: numpy<3
Requires-Dist: nvidia-ml-py3
Requires-Dist: opentsne
Requires-Dist: pandas
Requires-Dist: peft==0.14.0
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: rich
Requires-Dist: scikit-learn
Requires-Dist: sentencepiece
Requires-Dist: seqeval
Requires-Dist: spacy
Requires-Dist: torch<3,>=2.2
Requires-Dist: tqdm
Requires-Dist: transformers<4.50,>=4.48
Requires-Dist: wandb
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# micm-nlp

[![PyPI](https://img.shields.io/pypi/v/micm-nlp.svg)](https://pypi.org/project/micm-nlp/)
[![Python](https://img.shields.io/pypi/pyversions/micm-nlp.svg)](https://pypi.org/project/micm-nlp/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docs](https://readthedocs.org/projects/micm-nlp/badge/?version=latest)](https://micm-nlp.readthedocs.io/en/latest/)

<!-- start:tagline -->
NLP research toolkit for tokenization, pretraining, fine-tuning, and PEFT across encoder-only, decoder-only, and encoder-decoder architectures. Built on top of HuggingFace `transformers`, `peft`, and `datasets`.
<!-- end:tagline -->

<!--
The blocks between the start/end markers below are pulled into the documentation
site with MyST {include} directives. README is the canonical copy; docs/source/
never holds a second one. Moving or renaming a marker breaks a docs page --
grep for the marker name in docs/source/ before editing.
-->

## About

<!-- start:about -->
`micm-nlp` is a config-driven research toolkit for multilingual NLP work. It wraps the HuggingFace stack with a small set of high-level building blocks — `CONFIG`, `TOKENIZER`, `DATASET`, `MODEL`, and a unified `TRAINER` — that compose into reproducible training, fine-tuning, and evaluation pipelines.

It has backed two peer-reviewed publications: *Cross-Prompt Encoder for Low-Performing Languages* (Findings of IJCNLP–AACL 2025; [ACL Anthology](https://aclanthology.org/2025.findings-ijcnlp.144/)) and *A Comparison of Different Tokenization Methods for the Georgian Language* (ICNLSP 2024; [ACL Anthology](https://aclanthology.org/2024.icnlsp-1.22/)).

The package currently ships **two examples** that exercise a single use case end-to-end: preprocessing and decoder-only PEFT fine-tuning (XPE) on an FTP-reframed multilingual dataset hosted on the HuggingFace Hub. The toolkit's underlying surface is broader than these two examples demonstrate.

Additional examples covering encoder-only text classification, encoder-decoder seq2seq, and MLM pretraining will land in subsequent releases. Contributions and issue reports are welcome.
<!-- end:about -->

📖 **Full documentation: [micm-nlp.readthedocs.io](https://micm-nlp.readthedocs.io/)**

## Install

<!-- start:install-requires -->
Requires **Python 3.10 or newer**. On an older interpreter `pip` reports
`No matching distribution found for micm-nlp`, which does not say why — check with
`python3 --version` first.

A fresh Debian or Ubuntu machine has neither `pip` nor `venv`; install them with
`apt install python3-venv` (or use `uv`, `conda`, `pyenv` — anything that gives you a
Python 3.10+ environment with pip):

```bash
python3 -m venv .venv && source .venv/bin/activate
```
<!-- end:install-requires -->

From PyPI:

<!-- start:install-pypi -->
```bash
pip install micm-nlp
```
<!-- end:install-pypi -->

From source (development):

<!-- start:install-source -->
```bash
git clone https://github.com/bmikaberidze/micm-nlp.git
cd micm-nlp
pip install -e ".[dev]"
```

The `dev` extra adds `pytest` and `ruff`.
<!-- end:install-source -->

Docker (recommended for reproducibility on GPU machines):

<!-- start:install-docker -->
```bash
docker build -t micm-nlp .
docker run --gpus all -it --rm -v $(pwd):/app -w /app micm-nlp bash
```
<!-- end:install-docker -->

Credentials and the workspace root come from a `.env` file. `.env.example` is in the
repository, so `cp` only works from a clone — installing from PyPI, set the variables
in your environment instead, or pass `root_path` straight to `micm_nlp.init()`:

<!-- start:install-env -->
```bash
cp .env.example .env          # from a clone
```

| Variable | Purpose |
|---|---|
| `PROJECT_ROOT_PATH` | Workspace directory; `artefacts/` (datasets, models, evals, wandb) is created under it. Used as the fallback when `init()` is called without `root_path`. |
| `WANDB_API_KEY` | Needed to log a run to the W&B service. The shipped example configs set `WANDB_MODE: offline` in their `env:` block, so they write to `artefacts/wandb/` on disk and need no account; set it to `online` once you have run `wandb login`. |
| `HF_TOKEN` | Required only for gated HuggingFace models or datasets. |
<!-- end:install-env -->

## Quickstart

<!-- start:quickstart -->
```python
import micm_nlp
from micm_nlp.config import CONFIG
from micm_nlp.pipeline import run

# Sets the workspace root (where artefacts/ goes) and, optionally,
# enables Rich pretty-printing and traceback formatting.
micm_nlp.init({'root_path': '/path/to/your/workspace', 'pretty_output': True})
# Or, if PROJECT_ROOT_PATH is set in .env or the environment:
# micm_nlp.init()

config = CONFIG.from_yaml('micm-nlp-examples/xsc_finetune.yml')
model, test_output = run(config)
```

The example configs ship with the package. Write them out once with:

```bash
micm-nlp init-examples
```

They land in `micm-nlp-examples/`, and the copy you get always matches the version
you installed.

`init()` resolves the workspace root from its `root_path` argument, falling back to `PROJECT_ROOT_PATH` in the environment. Call it once before any pipeline call so `artefacts/` lands in the right place. It is **not** triggered on import.

`run(config)` chains: load tokenizer → load and preprocess dataset → load model (with PEFT if configured) → train → evaluate. Every stage is configured by YAML; no plumbing code required.
<!-- end:quickstart -->

## Package tour

```
micm_nlp/
├── pipeline.py     # Top-level wiring: load_dataset, preprocess_dataset, load_model, run
├── config.py       # CONFIG.from_yaml; resolves nested namespaces
├── tokenizers/     # Tokenizer factory, custom tokenizer classes, Georgian sentence splitter
├── datasets/       # DATASET class — local + HF Hub + HF saved + CSV/TXT/JSON
├── models/         # MODEL wrapper, PEFT dispatch, Cross-Prompt Encoder
├── training/       # TRAINER — wraps HF Trainer with custom callbacks + WandB
└── evals/          # Metrics, confusion matrices, plotting helpers
```

<!-- start:stages -->
The same flow, unwrapped — useful when a consumer repository needs to intervene between stages (swap a dataset, concatenate languages, reuse one tokenizer):

```python
from micm_nlp.config import CONFIG
from micm_nlp.tokenizers.tokenizer import load as load_tokenizer
from micm_nlp.datasets.dataset import DATASET
from micm_nlp.models.model import MODEL
from micm_nlp.training.runner import TRAINER

config = CONFIG.from_yaml('path/to/config.yml')
tokenizer = load_tokenizer(config)
dataset = DATASET(config)
dataset.preprocess(tokenizer)
model = MODEL(config)
trainer = TRAINER(model, dataset, tokenizer)
test_output = trainer.run()
```
<!-- end:stages -->

## Examples

<!-- start:examples -->
Two runnable examples ship with the repository. Together they cover one use case end to end — preprocessing and decoder-only PEFT fine-tuning on an FTP-reframed multilingual dataset from the HuggingFace Hub.

| Script | Config | What it does |
|---|---|---|
| `examples/preprocess_dataset.py` | `xsc_preprocess.yml` | Loads FTP-reframed XStoryCloze (English) from the Hub, tokenizes it for BLOOM-560M, saves the result locally. |
| `examples/run_model.py` | `xsc_finetune.yml` | Fine-tunes BLOOM-560M with Cross-Prompt Encoder PEFT on the Arabic split, then evaluates. |

Both scripts take the config as a flag:

```bash
micm-nlp init-examples
python examples/preprocess_dataset.py --config micm-nlp-examples/xsc_preprocess.yml
python examples/run_model.py          --config micm-nlp-examples/xsc_finetune.yml
```

The configs ship inside the package — `micm-nlp init-examples` writes them to
`micm-nlp-examples/`. The scripts are four lines each and live in the repository;
both are reproduced in the Quickstart above, so a `pip install` is enough to run
either one.

The toolkit's surface is broader than these two demonstrate. Examples for encoder-only text classification, encoder-decoder seq2seq and MLM pretraining are planned.
<!-- end:examples -->

## Supported architectures

<!-- start:architectures -->
| Architecture | Toolkit support | Covered by a shipped example |
|---|---|---|
| Decoder-only (BLOOM, Aya) | yes | yes |
| Encoder-only (BERT, XLM-R, mDeBERTa) | yes | planned |
| Encoder-decoder (T5) | yes | planned |

PEFT methods: LoRA, Prefix Tuning, P-Tuning / soft prompt tuning, and the Cross-Prompt Encoder. The shipped examples demonstrate the Cross-Prompt Encoder only.
<!-- end:architectures -->

## Development

```bash
pip install -e ".[dev]"
ruff check src/
ruff format src/
pytest
```

## Contributing

Pull requests are welcome. For non-trivial changes, please open an issue first to discuss the proposed change. A `CONTRIBUTORS.md` will be added with the first external contribution.

## Acknowledgements

<!-- start:acknowledgements -->
`micm-nlp` was developed at the Muskhelishvili Institute of Computational Mathematics (MICM, Georgian Technical University), in close research collaboration with Teimuraz Saghinadze (MICM), Simon Ostermann (DFKI / CERTAIN), and Philipp Müller (Max Planck Institute for Intelligent Systems), whose joint work on the Cross-Prompt Encoder (XPE) drove much of the toolkit's design and validation.

This work was partially supported by the European Union under Horizon Europe project "GAIN" (GA #101078950) and by the German Federal Ministry of Research, Technology and Space (BMFTR) as part of the project TRAILS (01IW24005).
<!-- end:acknowledgements -->

## Citation

<!-- start:citation -->
If you use `micm-nlp` in your research, please cite the package and (if relevant to your work) the XPE paper that drove its design:

```bibtex
@software{micm_nlp,
  author = {Mikaberidze, Beso},
  title = {micm-nlp: NLP research toolkit for multilingual fine-tuning and PEFT},
  url = {https://github.com/bmikaberidze/micm-nlp},
  version = {0.3.1},
  year = {2026},
}

@misc{mikaberidze2025crosspromptencoderlowperforminglanguages,
  title         = {Cross-Prompt Encoder for Low-Performing Languages},
  author        = {Beso Mikaberidze and Teimuraz Saghinadze and Simon Ostermann and Philipp Muller},
  year          = {2026},
  eprint        = {2508.10352},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2508.10352},
}
```
<!-- end:citation -->

## Contact

`beso.mikaberidze@gmail.com`
