ftmq.io
smart_read_proxies(uri, query=None, entity_type=ValueEntity, **store_kwargs)
Stream proxies from an arbitrary source
Example
from ftmq import Query
from ftmq.io import smart_read_proxies
# remote file-like source
for proxy in smart_read_proxies("s3://data/entities.ftm.json"):
print(proxy.schema)
# multiple files
for proxy in smart_read_proxies("./1.json", "./2.json"):
print(proxy.schema)
# nomenklatura store
for proxy in smart_read_proxies("redis://localhost", dataset="default"):
print(proxy.schema)
# apply a query to sql storage
q = Query(dataset="my_dataset", schema="Person")
for proxy in smart_read_proxies("sqlite:///data/ftm.db", query=q):
print(proxy.schema)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
uri
|
Uri | Iterable[Uri]
|
File-like uri or store uri or multiple uris |
required |
query
|
Query | None
|
Filter |
None
|
**store_kwargs
|
Any
|
Pass through configuration to statement store |
{}
|
Yields:
Type | Description |
---|---|
Entities
|
A generator of |
Source code in ftmq/io.py
smart_write_proxies(uri, proxies, mode='wb', **store_kwargs)
Write a stream of proxies (or data dicts) to an arbitrary target.
Example
Parameters:
Name | Type | Description | Default |
---|---|---|---|
uri
|
Uri
|
File-like uri or store uri |
required |
proxies
|
Iterable[Entity]
|
Iterable of proxy data |
required |
mode
|
str | None
|
Open mode for file-like targets (default: |
'wb'
|
**store_kwargs
|
Any
|
Pass through configuration to statement store |
{}
|
Returns:
Type | Description |
---|---|
int
|
Number of written proxies |