CLI Reference

Crawler framework for documents and structured scrapers.

The memorious command-line interface manages crawler execution and crawler state. Run any command with --help to see options inline.

Standalone worker processes are started with procrastinate's own CLI, see Running a standalone worker.

`memorious`

memorious [OPTIONS] COMMAND [ARGS]...

Top-level command. Without a subcommand, displays the help screen.

Option	Short	Description
`--version`	`-v`	Show the installed memorious version and exit.
`--settings`		Print the resolved runtime settings and exit.
`--install-completion`		Install shell completion for the current shell.
`--show-completion`		Print shell completion script for copying or customization.
`--help`		Show the help message and exit.

Subcommands

Command	Description
`run`	Run a crawler from a YAML config file.
`cancel`	Cancel pending jobs for a crawler.
`flush`	Delete all data and tags generated by a crawler.
`status`	Show crawler status: recent runs and stored document count.

`memorious run`

Run a crawler from a YAML config file.

memorious run [OPTIONS]

The crawler is loaded from the YAML config given with --config (local path or remote URI), queued as a job in procrastinate, and executed by an embedded worker until the queue drains (or --wait is given).

Options

Option	Short	Default	Description
`--config`	`-c`	required	URI or path to a crawler YAML config file.
`--continue-on-error`		`False`	Don't stop crawler execution on error.
`--flush`		`False`	Delete all existing data before execution.
`--concurrency`		`1`	Number of concurrent jobs (use `>1` for I/O-bound crawlers).
`--wait`	`-w`	`False`	Keep worker running after jobs complete (until interrupted).
`--idle-timeout`	`-t`	`30`	Auto-stop after N seconds of inactivity. Defaults to `30` when `concurrency>1`; pass `0` to disable.
`--clear-runs` / `--no-clear-runs`		`--clear-runs`	Cancel remaining tasks from previous runs before starting. Use `--no-clear-runs` to resume an interrupted crawl without losing queued jobs.
`--help`			Show the help message and exit.

Examples

# Run a crawler from a local file
memorious run -c ./crawlers/example.yml

# Run a crawler from a remote URI
memorious run -c https://example.org/crawlers/example.yml

# Run with higher concurrency for I/O-bound work
memorious run -c ./crawlers/example.yml --concurrency 8

# Flush prior data and re-run from scratch
memorious run -c ./crawlers/example.yml --flush

# Resume a previously interrupted crawl
memorious run -c ./crawlers/example.yml --no-clear-runs

`memorious cancel`

Cancel pending jobs for a crawler.

memorious cancel [OPTIONS]

Marks all queued and in-progress tasks for the crawler as cancelled. Already-completed work is preserved.

Options

Option	Short	Default	Description
`--config`	`-c`	required	URI or path to a crawler YAML config file.
`--help`			Show the help message and exit.

Example

memorious cancel -c ./crawlers/example.yml

`memorious flush`

Delete all data and tags generated by a crawler.

memorious flush [OPTIONS]

Removes archive entries, tags, and incremental state for the crawler. Run with care — the operation is destructive and cannot be undone.

Options

Option	Short	Default	Description
`--config`	`-c`	required	URI or path to a crawler YAML config file.
`--help`			Show the help message and exit.

Example

memorious flush -c ./crawlers/example.yml

`memorious status`

Show crawler status: recent runs and stored document count.

memorious status [OPTIONS]

Prints a table of recent runs (run id, start time, age) and the total number of stored documents.

Options

Option	Short	Default	Description
`--config`	`-c`	required	URI or path to a crawler YAML config file.
`--runs`	`-r`	`5`	Number of recent runs to show.
`--help`			Show the help message and exit.

Example

memorious status -c ./crawlers/example.yml --runs 10

Running a standalone worker

Memorious does not ship a worker command. In deployments where workers run as long-lived processes separate from memorious run, use procrastinate's CLI directly and point it at the memorious queue:

procrastinate worker -q memorious

The worker needs to know which procrastinate app to load. Either export it once:

export PROCRASTINATE_APP=memorious.tasks.app
export PROCRASTINATE_DB_URI=postgresql://user:pass@localhost/memorious
procrastinate worker -q memorious

...or pass it per invocation:

procrastinate --app=memorious.tasks.app worker -q memorious

Common options

Option	Short	Description
`--queues`	`-q`	Comma-separated queues to listen to. Use `memorious`.
`--concurrency`	`-c`	Number of parallel jobs to process at once.
`--name`	`-n`	Name of the worker (shows up in the job table).
`--wait` / `--one-shot`	`-w`	Wait for new jobs when the queue drains, or terminate.
`--shutdown-graceful-timeout`		How long to wait for running jobs on shutdown.

Run procrastinate worker --help for the full list.

Examples

# Four concurrent jobs, keeps running when the queue is empty
procrastinate worker -q memorious -c 4

# Drain the queue once and exit
procrastinate worker -q memorious --one-shot