anystore

Store anything anywhere. anystore provides a high-level storage and retrieval interface for various supported store backends, such as redis, sql, file, http, cloud-storages and anything else supported by fsspec.

Think of it as a key -> value store, and anystore acts as a cache backend. And when keys are filenames and values are byte blobs, anystore becomes actually a file-like storage backend – but always with the same and interchangeable interface.

Quickstart

pip install anystore

anystore --help

Get started

Why?

In our several data engineering projects we always wrote boilerplate code that handles the featureset of anystore but not in a reusable way. This library shall be a stable foundation for data wrangling related python projects.

Used by

ftmq, a query interface layer for followthemoney data
investigraph, a framework to manage collections of structured followthemoney data
ftmq-api, a simple api on top off ftmq built with FastApi
ftm-geocode, batch parse and geocode addresses from followthemoney entities
ftm-datalake, a library to crawl, sync and move around document collections

License and Copyright

anystore is licensed under the AGPLv3 or later license.

Prior to version 0.3.0, anystore was released under the GPL-3.0 license.

see NOTICE and LICENSE