Skip to content

Overview

anystore provides a simple high-level interface for retrieving and writing data in a store. Data is identified by a key. A diverse range of storage backends is supported.

This library is a collection of shared basic features used for projects and applications related to data storage, data streaming and data engineering.

Use cases

  • A simple storage interface for multiple tenants
  • A shared cache for collaborative data analysis work
  • A caching implementation, e.g. for an api
  • Read and write data from different local or remote backends
  • Crawl documents from various remote endpoints

Basic features

  • List available keys, optionally by glob or prefix filters
  • Get data at a given key
  • Write data to a given key
  • Delete data at a given key
  • Stream data line by line
  • Read and write contents with a BinaryIO handler
  • Serialize data in different ways, including pydantic models

Limitations

The goal of anystore is to provide a simple high-level interface that works the same way for multiple backends. This allows to develop applications with configurable or swappable backends, depending on the actual use case and scalability requirements. This has, on purpose, some limitations or issues to consider:

  • Writing data just overwrites existing keys without checking
  • No logic for renaming keys or moving data
  • No logic for changing (e.g. appending) data to an existing key
  • More backend specific features, like SET datatype in redis or queries in sql are not supported via this interface

Supported backends

  • All file-like backends supported by fsspec (here and here)
  • Sql via sqlalchemy such as sqlite, postgres, mysql
  • Redis compatible (redis or e.g. kvrocks)
  • ZipFile: Use a compressed archive as a store backend
  • A simple in-memory implementation