CLI

Storage management

Read and write datasets and key-value stores, and inspect request queues.

The scrapely datasets and scrapely key-value-stores commands read and write storages on the platform through the API. They work from any folder once you are logged in. For the storages of the run your code is in, use the Actor runtime commands instead.

Every storage argument takes a storage ID or the name of one of your named storages. The CLI looks names up in your storage list. Delete commands ask for confirmation unless you pass -y, --yes.

Datasets

A dataset is an append-only list of JSON items. Each run has a default dataset for its results.

Command	Purpose
`datasets ls`	List datasets
`datasets create [name]`	Create a dataset, or get the one with that name
`datasets info <datasetId>`	Show a dataset's ID, name, item count, and creation time
`datasets get-items <datasetId>`	Print items
`datasets push-items <datasetId> [item]`	Append items
`datasets rename <datasetIdOrName> [newName]`	Rename a dataset or remove its name
`datasets rm <datasetIdOrName>`	Delete a dataset

List datasets

.bash

scrapely datasets ls
scrapely datasets ls --unnamed --desc --limit 20

Each line shows the ID, the name, and the number of items.

Option	Description
`--limit <number>`	Maximum number to return
`--offset <number>`	Number to skip
`--unnamed`	Only unnamed datasets, such as the default datasets of runs
`--desc`	Newest first
`--json`	Print the API response

Create a dataset

.bash

scrapely datasets create my-results

Without a name, create makes an unnamed dataset. With a name, it returns the existing dataset of that name if there is one. --json prints the dataset object.

Get items

.bash

scrapely datasets get-items <datasetId>
scrapely datasets get-items <datasetId> --format csv > results.csv
scrapely datasets get-items <datasetId> --format jsonl | jq -r '.url'

get-items writes the items to stdout. Redirect it to save a file.

Option	Description
`--format <format>`	`json` (default), `jsonl`, `csv`, `html`, `rss`, `xml`, or `xlsx`
`--limit <number>`	Maximum number of items
`--offset <number>`	Number of items to skip

With json, the output is a JSON array of the items. The other formats are passed through as the API returns them.

To find a run's dataset, run scrapely runs info <runId> and use the Dataset ID.

Push items

.bash

scrapely datasets push-items <datasetId> '{"url":"https://example.com","title":"Example"}'
scrapely datasets push-items <datasetId> '[{"url":"https://a.example"},{"url":"https://b.example"}]'
cat rows.json | scrapely datasets push-items <datasetId>

The item is a JSON object or an array of objects. Without the argument, the command reads it from stdin. Invalid JSON is rejected before anything is sent.

Rename or delete a dataset

.bash

scrapely datasets rename <datasetIdOrName> new-name
scrapely datasets rename <datasetIdOrName> --unname
scrapely datasets rm <datasetIdOrName> -y

--unname removes the name instead of setting a new one. rm deletes the dataset and its items.

Key-value stores

A key-value store holds records under string keys: JSON, text, HTML, images, or any other file. Each run has a default store, which holds its INPUT record and often an OUTPUT record.

kvs is an alias for key-value-stores:

.bash

scrapely key-value-stores ls
scrapely kvs ls

Command	Purpose
`kvs ls`	List stores
`kvs create [name]`	Create a store, or get the one with that name
`kvs info <storeId>`	Show a store's ID, name, and creation time
`kvs keys <storeId>`	List the keys in a store
`kvs get-value <storeId> <key>`	Print a record
`kvs set-value <storeId> <key> [value]`	Write a record, or delete it when the value is empty
`kvs delete-value <storeId> <key>`	Delete a record
`kvs rename <storeIdOrName> [newName]`	Rename a store or remove its name
`kvs rm <storeIdOrName>`	Delete a store

List stores

.bash

scrapely kvs ls

kvs ls takes the same --limit, --offset, --unnamed, --desc, and --json options as datasets ls.

Create a store

.bash

scrapely kvs create my-state

Without a name, create makes an unnamed store. With a name, it returns the existing store of that name if there is one. --json prints the store object.

List keys

.bash

scrapely kvs keys <storeId>
scrapely kvs keys <storeId> --limit 100 --exclusive-start-key last-key-from-previous-page

Each line shows the key and the record size.

Option	Description
`--limit <number>`	Maximum number of keys
`--exclusive-start-key <key>`	Start after this key
`--json`	Print the API response

Keys are paged by key, not by offset. To get the next page, pass the last key you received as --exclusive-start-key.

Read a record

.bash

scrapely kvs get-value <storeId> OUTPUT
scrapely kvs get-value <storeId> screenshot.png > screenshot.png
scrapely kvs get-value <storeId> screenshot.png --only-content-type

get-value writes the record to stdout as stored. Binary records are written byte for byte, so redirect them to a file. JSON is pretty-printed. --only-content-type prints only the record's content type. A missing key is an error.

Write a record

.bash

scrapely kvs set-value <storeId> STATE '{"cursor":"next"}'
scrapely kvs set-value <storeId> page.html '<h1>Hello</h1>' -c text/html

Option	Description
`-c, --content-type <type>`	Content type of the record. Default `application/json`.

With the default content type, the value must be valid JSON. The value is always the command-line argument: set-value doesn't read stdin or files, so it is meant for text records. Upload binary files with the key-value store API.

An empty or missing value deletes the record without asking:

.bash

scrapely kvs set-value <storeId> TEMPORARY

Delete a record

.bash

scrapely kvs delete-value <storeId> <key>
scrapely kvs delete-value <storeId> <key> -y

Rename or delete a store

.bash

scrapely kvs rename <storeIdOrName> new-name
scrapely kvs rename <storeIdOrName> --unname
scrapely kvs rm <storeIdOrName> -y

Request queues

scrapely request-queues exists but has no subcommands yet. To work with request queues from the terminal, call the request queue API with scrapely api:

.bash

scrapely api GET /v2/request-queues
scrapely api GET /v2/request-queues/<queueId>/requests -p '{"limit":10}'

What's next

Storage: when to use each storage type.
Results: find, export, and share run results.
Actor runtime: read and write the current run's storages from inside a container.
Datasets API and key-value stores API: the endpoints behind these commands.