Projects & running

These commands work on the Actor project in a folder on your machine: create it from a template or add a config to existing code, check its schemas, push it to the platform, download it again, and run it locally or on the platform.

Create a project

From a template

scrapely create my-actor --template crawlee-cheerio-js

create makes a new folder named after the Actor, in the current directory. It fails if the folder already exists. The name must be at least three characters long and contain only lowercase letters, digits, and hyphens, and it can't start or end with a hyphen.

Without a name, create asks for one. Without --template, it asks what you want to build, which language you prefer, and then which template to use.

What create does:

1. Fetches the template catalog from the API (GET /v2/actor-templates). If the API can't be reached, it uses the copy bundled with the CLI.
2. Clones scrapely-app/actor-templates with git clone --depth 1 and copies the template folder, without node_modules, .git, .github, storage, dist, and .env. Git and access to GitHub are required.
3. Sets the Actor name in package.json and in the template's .actor/scrapely.json. Python projects created from a repository with --git-repo also rename a my_actor/ package to the Actor name with hyphens replaced by underscores.
4. Adds .gitignore and .actorignore if the template has none.
5. Runs git init, git add ., and git commit -m "Initial commit".
6. Runs npm install for JavaScript and TypeScript templates. Python projects install nothing locally.

List the templates with:

scrapely templates ls
scrapely templates ls --json

--json prints each template's name, aliases, displayName, description, language, and useCases. templates ls doesn't need a login.

crawlee-playwright-js and crawlee-puppeteer-js still work as aliases for crawlee-playwright and crawlee-puppeteer-chrome.

Keep the source on GitHub

scrapely create my-actor --template crawlee-cheerio-js --source github --git-repo my-org/my-actor --auto-build on

With --source github, create creates a private GitHub repository with the GitHub CLI (gh) and pushes the template to it. Without gh, it adds the remote and pushes with Git. If the repository already exists and isn't empty, create clones it instead of copying the template.

--auto-build on then creates the Actor on the platform from the repository, starts the first build, and registers a webhook so each push to the linked branch rebuilds the Actor. Pushes to other branches are ignored. The webhook needs the GitHub integration with admin rights on the repository. If it can't be registered, create prints the webhook URL to add by hand. Don't run scrapely push for an Actor set up this way, because it replaces the Git source with uploaded files. Commit and git push instead.

In an existing folder

scrapely init
scrapely init my-actor

init adds Actor files to existing code. Without an argument it works in the current folder. With one, it works in the subfolder of that name and creates it if needed. It then asks for the Actor name, with the argument or my-actor as the default.

If the folder already has a package.json, init asks whether to continue. In a non-interactive shell without -y, it stops there. If the folder has files but no package.json, pyproject.toml, requirements.txt, or scrapy.cfg, it asks for confirmation first.

What init writes:

• scrapely.json with actorSpecification, name, version 0.0, buildTag latest, and dockerfile if you passed one. It skips this if any Actor config already exists.
• For a Scrapy project (a folder with scrapy.cfg): a main.py wrapper that runs the project's spiders inside the Actor and pushes every scraped item to the default dataset, plus scrapely-python (the SDK's PyPI name) and scrapy in requirements.txt. A scrapely line from an older CLI version is replaced, because the PyPI package scrapely is an unrelated library.
• For anything else: a package.json with a start script if there is none, and a main.js Playwright starter if there is none. An existing package.json is never changed.
• .gitignore and .actorignore if missing.

init doesn't create a Dockerfile. Add one before you push, or start from a template, whose Dockerfile builds on the Scrape.ly base images.

Check schemas

scrapely validate-schema
scrapely validate-schema .actor/input_schema.json

Without a path, validate-schema finds the Actor config and checks every schema it references: the input schema (input or inputSchema), the dataset schema (storages.dataset), the output schema (output or outputSchema), and the key-value store schema (storages.keyValueStore). Each can be inline or a path, resolved relative to the config file or the current folder. With a path, it checks that one file.

The check is basic: the schema must be an object, properties must be an object, required must be a list of strings, and title and description must be strings. It exits with code 1 on errors. It doesn't check editor types, defaults against types, or the full Actor input schema specification.

For the input schema it also checks categories. A category that names no defined category is an error, with the closest key suggested. So are a parent loop, nesting deeper than 3 levels, and a wrong value type. Unused categories and unknown category settings are warnings, which don't change the exit code.

Push to the platform

scrapely push
scrapely push <actorId>

push uploads the project, creates or replaces a version of the Actor, and builds it. scrapely actors push is the same command.

Step by step:

1. Reads the Actor config. It fails with No package.json found. Run scrapely init first. if the folder has neither a config nor a package.json.
2. Checks that every @secret in environmentVariables exists in your local secrets. Secret values are not uploaded. See Secrets on the platform.
3. Collects the files. It always skips node_modules, .git, storage, dist, .env, .DS_Store, *.log, and .scrapely, plus every pattern in .gitignore and in .actorignore. Both files apply. Symbolic links are skipped. Binary files are uploaded base64-encoded. If the upload is larger than 3 MB, the files are sent as one ZIP archive.
4. Finds the Actor: the ID you passed, or actorId from the config. If there is none, it creates a new Actor with the config's name, title, and description, and writes the new actorId into the config file it read. If no config file exists, it creates scrapely.json.
5. For an existing Actor, compares modification times. If the Actor on the platform is newer than your files, it asks before overwriting. In a non-interactive shell or with --json it fails instead. --force skips the check.
6. Uploads the version. If a version with that number exists, it is replaced, including any environment variables set on it in the dashboard.
7. Starts a build without the Docker cache and waits for it as -w says. If the build fails, it prints the build log and exits with code 1. If -w <seconds> runs out while the build is still running, it exits with code 0.

Without --json, push ends by printing scrapely call <actorId> so you can run the new build.

Pull from the platform

scrapely pull
scrapely pull <actorId> --version 0.1 --dir ./my-actor

pull downloads an Actor's source. Only the Actor's owner can pull it. Without an ID it uses actorId from the config in the current folder, then the config's name.

For an Actor built from a Git repository, pull runs git clone. Only the owner can see the repository URL. If the target folder isn't empty, it clones into a subfolder named after the repository.

Run locally

scrapely run
scrapely run -i '{"startUrls":[{"url":"https://crawlee.dev"}]}'
scrapely run -f input.json
echo '{"startUrls":[{"url":"https://crawlee.dev"}]}' | scrapely run -f -

run starts the project in the current folder as a process on your machine. It doesn't use the platform or Docker. To run on the platform, use call. If you pass an Actor ID, run only prints a note pointing you to call.

Before it starts the process, run:

1. Clears storage/datasets/default, storage/key_value_stores/default, and storage/request_queues/default, unless you pass --no-purge. It keeps INPUT.json and named storages.
2. Reads the input from -i or -f, or from storage/key_value_stores/default/INPUT.json. Passing both -i and -f is an error.
3. Applies the defaults from the input schema and fails if a required field is missing. The schema is the config's input, or .actor/input_schema.json.
4. Sets the input as ACTOR_INPUT (base64-encoded JSON) and ACTOR_INPUT_KEY=INPUT.
5. Sets the config's environmentVariables, replacing each @secret with its value from scrapely secrets.

How it starts the project:

• Node.js: --entrypoint if given, else npm run start if package.json has a start script, else node with package.json's main.
• Python: the interpreter from .venv if there is one, else python3. It runs --entrypoint (a file, or a module with -m), else main.py or src/main.py. If your Python code lives in a package, pass the package name, for example --entrypoint my_actor.

run exits with the exit code of the process.

Run on the platform

scrapely call
scrapely call <actorId> -i '{"startUrls":[{"url":"https://crawlee.dev"}]}' -o

call starts a run on the platform and waits for it to finish, printing new log output every two seconds. Without an ID it uses actorId from the config in the current folder. scrapely actors call is the same command.

Without -i or -f, the input is storage/key_value_stores/default/INPUT.json if that file exists. Otherwise the run starts without input. call exits with code 1 if the run ends in any status other than SUCCEEDED.

To start a run without waiting, use actors start.

Project configuration

The CLI reads the Actor config from the first of these files that exists:

1. .actor/actor.json
2. .actor/scrapely.json
3. scrapely.json
4. actor.json

If both a file in .actor/ and a root scrapely.json exist, the fields in scrapely.json override the ones in .actor/. Templates use .actor/scrapely.json. init writes scrapely.json.

The fields the CLI uses:

The platform reads the same file when it builds and runs the Actor, including dockerfile, dockerContextDir, the schemas, the memory limits, environmentVariables, and usesStandbyMode. See Actors for the full format.

Two ignore files decide what push uploads: .gitignore and .actorignore. .dockerignore only affects the Docker build on the platform.

Example

{
    "actorSpecification": 1,
    "name": "listing-scraper",
    "title": "Listing scraper",
    "description": "Scrapes product listings into a dataset",
    "version": "0.1",
    "buildTag": "latest",
    "dockerfile": "./Dockerfile",
    "environmentVariables": {
        "LOG_LEVEL": "info",
        "API_KEY": "@my-api-key"
    },
    "input": "./input_schema.json"
}

With this file at .actor/scrapely.json, the paths are resolved relative to .actor/.

What's next

• Actor runtime: commands your code can call inside the container.
• Runs & builds: inspect and control what you deployed.
• Config & troubleshooting: run push from a pipeline.