anystore

Store anything anywhere. anystore provides a high-level storage and retrieval interface for various supported store backends, such as redis, sql, file, http, cloud-storages and anything else supported by fsspec.

Think of it as a key -> value store, and anystore acts as a cache backend. And when keys are filenames and values are byte blobs, anystore becomes actually a file-like storage backend – but always with the same and interchangeable interface.

Quickstart

pip install anystore

anystore --help

Get started

Why?

In our several data engineering projects we always wrote boilerplate code that handles the featureset of anystore but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.

Used by

• ftmq, a query interface layer for followthemoney data
• investigraph, a framework to manage collections of structured followthemoney data
• ftmq-api, a simple api on top off ftmq built with FastApi
• leakrfc, a library to crawl, sync and move around document collections (in progress)