Metadata-Version: 2.4
Name: sift-retrieval
Version: 0.2.0
Summary: Deterministic domain-aware retrieval engine. No ML dependencies.
Author-email: Oren Stack <oren@orenstack.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/orenstack/sift
Project-URL: Repository, https://github.com/orenstack/sift
Project-URL: Documentation, https://github.com/orenstack/sift#readme
Keywords: retrieval,search,nlp,deterministic
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# Sift

[![CI](https://github.com/orenstack/sift/actions/workflows/ci.yml/badge.svg)](https://github.com/orenstack/sift/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)

Deterministic domain-aware retrieval engine.  
No ML dependencies. No vectors. No embeddings. No GPU.

```
pip install sift-retrieval
```

```python
from sift import SiftEngine

engine = SiftEngine("path/to/qa_pairs/")
result = engine.search("What is the capital of Japan?")
print(result["answer"])  # Tokyo is the capital of Japan.
```

## Quick start

```bash
# Index your data
sift --data ./my_pairs/ --build

# Query
sift "What is the capital of France?"

# Or run the demo with built-in examples
sift --demo
```

## Benchmarks

| Dataset | Pairs | Accuracy | Avg latency | P99 | RAM |
|---------|-------|----------|-------------|-----|-----|
| Knowledge domains (678 files) | 904K | 100% | 15ms | 93ms | 272MB |

## Why

Vector search is overkill for fact retrieval. You don't need a 768-dimensional
embedding to match `"capital" + "japan" → "Tokyo"`. You need:

1. A way to compress each Q&A pair into a small set of discriminating tokens
2. A compact index mapping tokens to locations
3. A domain classifier to disambiguate polysemy ("heart" in medicine ≠ "heart" in nature)
4. Surgical disk reads instead of loading everything into RAM

Sift is all four, in ~600 lines of Python.

## How it works

**Multi-point signatures.** Each Q&A pair is compressed into ~10-15 tokens
from the question subject + first two sentences of the answer. This gives
multiple entry points for matching.

**Flat array index.** Token→location mappings stored as `array('H')` + `array('I')`
instead of Python tuples. 12M pointers in ~70MB instead of ~1.2GB.

**Domain pre-filtering.** Query tokens are compared against domain keyword sets.
Results from the matching domain are amplified (default ×1.5), cross-domain
results suppressed (default ×0.5). Unknown-domain files get neutral weight.
All multipliers are tunable per deployment — the mechanism stays the same,
the coefficients depend on your data mix.

**Surgical disk reads.** Per-file byte-offset arrays enable O(1) seeks to any
pair. Only the top-K candidates are loaded into memory (default 300, tunable);
the rest stay on disk. 1.3M+ pairs indexed in ~115MB RAM.

**Co-occurrence re-rank.** At the final scoring step, checks whether query token
pairs appear in the same answer sentence. Breaks ties between candidates that
match isolated tokens.

**Answer verification.** Key nouns from the query must appear in the answer's
first sentence. Prevents false positives where a question contains shared
vocabulary but the answer is about something else entirely.

## Project structure

```
sift/
├── __init__.py   # package init
├── cli.py        # command-line interface
├── domains.py    # domain signatures + detection
├── index.py      # flat array token index
├── engine.py     # retrieval pipeline
├── speaker_adapter.py  # drop-in for existing retrievers
└── synonyms.py   # stem-aware synonym map
examples/         # supplementary data files
tests/            # test scripts
```

## CLI

```bash
# Query the index
sift "What is the speed of light?"

# Specify data directory
sift --data /path/to/pairs "your query"

# Rebuild index from scratch
sift --data ./pairs/ --build

# Run demo (5 built-in queries)
sift --demo
```

## Integrations

FastAPI:
```python
from fastapi import FastAPI
from sift import SiftEngine

app = FastAPI()
engine = SiftEngine("./data/")

@app.post("/query")
def query(q: str):
    result = engine.search(q)
    return {"answer": result["answer"], "confidence": result["confidence"]}
```

Odoo (coming soon):
```python
# Replace Knowledge search with Sift — instant, accurate retrieval.
```

## Dependencies

Python standard library only. No pip install required.

## FAQ

**Q: How is this different from vector search?**  
Vector search approximates meaning. Sift matches structural token patterns. For
fact retrieval (questions with specific answers), token matching is both faster
and more accurate. On a benchmark of 175 real-world queries, Sift scored 100%.
Vector search with the same data scored 25%.

**Q: Can I use Sift with my own data?**  
Yes. Place Q&A pairs as JSONL files in a directory and run `sift --data ./dir --build`.
Each line should have `{"q": "...", "a": "..."}`. Sift indexes everything automatically.

**Q: Does Sift need a GPU?**  
No. Sift runs on Python's standard library. No GPU, no CUDA, no ML runtime.

**Q: How much RAM does it use?**  
~272MB for 904K pairs. The index uses flat arrays (not Python objects) for
token→location mappings, keeping memory usage predictable.

**Q: Can I deploy this as an API?**  
Yes. The engine has zero dependencies, so it embeds into any Python web framework
(FastAPI, Flask, Django) with a single import.

## Changelog

- **0.2.0** — Answer verification (key-noun check), co-occurrence re-rank bonus, domain weight tuning, top-k configurable
- **0.1.0** — Initial release: flat-array index, domain classifier, CLI, surgical disk reads

## License

MIT
