Metadata-Version: 2.4
Name: dataspoc-pipe
Version: 0.1.0
Summary: Motor de ingestão de dados — Singer + Parquet + Bucket
Author-email: DataSpoc <dev@dataspoc.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data,etl,ingestion,parquet,singer
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Requires-Python: >=3.10
Requires-Dist: fsspec>=2024.1
Requires-Dist: pyarrow>=15.0
Requires-Dist: pydantic-settings<3.0,>=2.0
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: python-crontab>=3.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer<1.0,>=0.9
Provides-Extra: azure
Requires-Dist: adlfs>=2024.1; extra == 'azure'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: gcs
Requires-Dist: gcsfs>=2024.1; extra == 'gcs'
Provides-Extra: s3
Requires-Dist: s3fs>=2024.1; extra == 's3'
Description-Content-Type: text/markdown

<h1 align="center">DataSpoc Pipe</h1>

<p align="center">
  <a href="#"><img src="https://img.shields.io/github/actions/workflow/status/dataspoc/dataspoc-pipe/ci.yml?branch=main&label=CI&style=flat-square" alt="CI"></a>
  <a href="https://pypi.org/project/dataspoc-pipe/"><img src="https://img.shields.io/pypi/v/dataspoc-pipe?style=flat-square" alt="PyPI"></a>
  <a href="https://github.com/dataspoclab/dataspoc-pipe/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License"></a>
  <a href="#"><img src="https://img.shields.io/badge/python-3.10%2B-blue?style=flat-square" alt="Python 3.10+"></a>
  <a href="#"><img src="https://img.shields.io/badge/discord-join-7289da?style=flat-square&logo=discord&logoColor=white" alt="Discord"></a>
</p>

<p align="center"><i>Singer taps to Parquet in cloud buckets. That simple.</i></p>

<p align="center">
  <a href="#">Docs</a> &middot;
  <a href="#">Tutorial</a> &middot;
  <a href="#">Discord</a>
</p>

<!-- TODO: Add hero GIF/screenshot of the CLI running a pipeline end-to-end -->

## Why DataSpoc Pipe?

Most data ingestion tools drown you in orchestration complexity. DataSpoc Pipe does one thing well: connect to any of the **400+ Singer taps** (databases, APIs, SaaS), convert to Parquet, and land it in your cloud bucket — cataloged and ready to query. Handles tables from kilobytes to **hundreds of GBs** via streaming. No DAGs, no servers, no infrastructure.

> **400+** data sources · **Streaming** (no memory limits) · **Zero** infrastructure · **< 15 min** setup

## Highlights

- **Singer-compatible** — use any of the 400+ existing Singer taps
- **Parquet output** — columnar, compressed (zstd), ready for analytics
- **Multi-cloud** — S3, GCS, Azure Blob, or local filesystem
- **Auto-catalog** — generates `manifest.json` so downstream tools discover your tables automatically
- **Incremental ingestion** — bookmark-based state tracking, only fetch new data
- **Convention-based transforms** — drop a Python file in `transforms/` to clean data during ingestion, per batch, no config needed
- **Built-in taps** — Google Sheets (public) works out of the box, no extra install
- **CLI-first** — one command to run a pipeline, cron to schedule it
- **Stateless** — all state lives in the bucket, not on your machine

## Installation

```bash
pip install dataspoc-pipe[s3]
```

<details>
<summary>Other cloud providers</summary>

```bash
# Google Cloud Storage
pip install dataspoc-pipe[gcs]

# Azure Blob Storage
pip install dataspoc-pipe[azure]

# Local filesystem only (no extras needed)
pip install dataspoc-pipe
```

</details>

## Quick start

```bash
# 1. Initialize config structure
dataspoc-pipe init

# 2. Create a pipeline (interactive wizard)
dataspoc-pipe add my-pipeline

# 3. Edit the generated source config if needed
#    ~/.dataspoc-pipe/sources/my-pipeline.json

# 4. Run it
dataspoc-pipe run my-pipeline

# 5. Check results
dataspoc-pipe status
```

Your data is now at `<bucket>/raw/<source>/<table>/` as Parquet.

Config structure created by `init`:
```
~/.dataspoc-pipe/
  config.yaml           # Global defaults
  sources/              # Source configs (1 JSON per source, generated by `add`)
  pipelines/            # Pipeline definitions (1 YAML per pipeline)
  transforms/           # Optional Python transforms (same name as pipeline)
```

## How it works

```
                          stdout
┌─────────────┐    ┌──────────┐    ┌───────────────┐    ┌──────────────┐
│ Data Source │───>│ Singer   │───>│ DataSpoc Pipe │───>│ Cloud Bucket │
│ (DB, API, …)│    │ Tap      │    │ transform(df) │    │ (S3/GCS/Az)  │
└─────────────┘    └──────────┘    └───────┬───────┘    └──────────────┘
                                           │
                                    manifest.json
                                     state.json
                                       logs/
```

1. Singer tap extracts data from the source, emits JSON on stdout
2. Pipe reads the stream, buffers in batches (~10K records)
3. If `transforms/<pipeline>.py` exists → applies `transform(df)` per batch
4. Converts to Parquet and uploads to bucket
5. Updates the manifest catalog and saves execution logs

## Built-in taps

| Tap | Source | Config template | Extra install |
|-----|--------|----------------|---------------|
| `parquet` | Parquet files (local or S3/GCS/Azure) | Built-in | None |
| `google-sheets-public` | Public Google Sheets | Built-in | None |
| `tap-postgres` | PostgreSQL | Yes | `pip install tap-postgres` |
| `tap-mysql` | MySQL | Yes | `pip install tap-mysql` |
| `tap-csv` | CSV files | Yes | `pip install tap-csv` |
| `tap-s3-csv` | CSV on S3 | Yes | `pip install tap-s3-csv` |
| `tap-github` | GitHub API | Yes | `pip install tap-github` |
| `tap-rest-api` | Any REST API | Yes | `pip install tap-rest-api` |
| `tap-mongodb` | MongoDB | Yes | `pip install tap-mongodb` |
| `tap-salesforce` | Salesforce | Yes | `pip install tap-salesforce` |
| `tap-google-sheets` | Google Sheets (OAuth) | Yes | `pip install tap-google-sheets` |

Any Singer-compatible tap works. Run `dataspoc-pipe add` to see available templates.

## Access control

DataSpoc delegates all access control to your cloud provider's IAM. Best practices:

- **One bucket per permission boundary** — e.g., `s3://company-public`, `s3://company-finance`, `s3://company-hr`
- **Pipe needs write access** to the destination bucket; users need only read access
- **Use IAM roles and policies** — never store credentials in pipeline configs
- If credentials lack permission, the pipeline fails with "Access Denied"

## Part of the DataSpoc Platform

| Project | Role |
|---------|------|
| **DataSpoc Pipe** (this) | Ingestion: Singer taps to Parquet in cloud buckets |
| **DataSpoc Lens** | Virtual warehouse: SQL + Jupyter + AI over your data lake |
| **DataSpoc ML** | AutoML: train and deploy models from your lake |

The bucket is the contract. Pipe writes. Lens reads. ML consumes and produces.

## Community

- **Discord** — [Join the conversation](#) for questions and support
- **GitHub Issues** — [Report bugs or request features](https://github.com/dataspoclab/dataspoc-pipe/issues)
- **Contributing** — PRs welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines

## License

[Apache 2.0](LICENSE) — free to use, modify, and distribute.
