CLI Reference
memorious
Crawler framework for documents and structured scrapers.
Usage
memorious [OPTIONS] COMMAND [ARGS]...
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
-v, --version |
Show version | No | - |
--settings / --no-settings |
Show current settings [default: no-settings] | No | - |
--install-completion |
Install completion for the current shell. | No | - |
--show-completion |
Show completion for the current shell, to copy it or customize the installation. | No | - |
--help |
Show this message and exit. | No | - |
Commands
| Name | Description |
|---|---|
run |
Run a crawler from a YAML config file. |
worker |
Start the procrastinate worker to process... |
cancel |
Cancel execution of a crawler. |
flush |
Delete all data and tags generated by a... |
Sub Commands
memorious run
Run a crawler from a YAML config file.
Usage
memorious run [OPTIONS] URI
Arguments
| Name | Description | Required |
|---|---|---|
URI |
URI or path to crawler YAML config file | Yes |
Options
| Name | Description | Required | Default |
|---|---|---|---|
--continue-on-error |
Don't stop crawler execution on error | No | - |
--flush |
Delete all existing data before execution | No | - |
-c, --concurrency INTEGER |
Number of concurrent jobs (use >1 for I/O-bound crawlers) [default: 1] | No | - |
-w, --wait |
Keep worker running after jobs complete (until interrupted) | No | - |
--help |
Show this message and exit. | No | - |
memorious worker
Start the procrastinate worker to process crawler jobs.
Usage
memorious worker [OPTIONS]
Arguments
No arguments available
Options
| Name | Description | Required | Default |
|---|---|---|---|
-c, --concurrency INTEGER |
Number of concurrent jobs (use >1 for I/O-bound crawlers) [default: 1] | No | - |
--help |
Show this message and exit. | No | - |
memorious cancel
Cancel execution of a crawler.
Usage
memorious cancel [OPTIONS] URI
Arguments
| Name | Description | Required |
|---|---|---|
URI |
URI or path to crawler YAML config file | Yes |
Options
| Name | Description | Required | Default |
|---|---|---|---|
--help |
Show this message and exit. | No | - |
memorious flush
Delete all data and tags generated by a crawler.
Usage
memorious flush [OPTIONS] URI
Arguments
| Name | Description | Required |
|---|---|---|
URI |
URI or path to crawler YAML config file | Yes |
Options
| Name | Description | Required | Default |
|---|---|---|---|
--help |
Show this message and exit. | No | - |