OpenAI Vector Store

Adds the dataset of each successful run to one of your OpenAI vector stores, so assistants and the Responses API can find it with file search. Scrape.ly uploads the results as JSON files with your own OpenAI API key and adds them to the vector store; OpenAI splits, embeds and indexes them. The key is stored encrypted, only ever sent to api.openai.com, and never shown again.

Uploading, storing and indexing the files is billed to your OpenAI account, not to Scrape.ly.

Get a key and a vector store

In the OpenAI dashboard:

1. Open Storage → Vector stores, create a vector store (or open an existing one), and copy its ID. It looks like vs_abc123….
2. Open API keys and create a key in the same project as the vector store. A restricted key works if it has Write access to Files and Vector stores.

Add results after each run

Set it up with the Send results to an OpenAI vector store recipe in New automation, or add OpenAI Vector Store as a delivery in the automation editor to add the results of several Actors, after filters and an AI step.

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Send results to an OpenAI vector store, or click Or create manually and, under Send, click OpenAI Vector Store.
2. Paste the OpenAI API key and the Vector store ID.
3. Optionally set Max items per run (empty adds every item).
4. Use Save and send test. The test only reads the vector store to check the key and the ID; it adds nothing.

Files: each run adds JSON files named <actor-name>-<run-id>-part-001.json, each a JSON array of whole items. Large runs are split into parts of about 10 MB, so every file stays within what file search indexes. The files are tagged with the attributes scrapely_run_id and scrapely_actor, which you can use as file search filters.

When it uploads: only for succeeded runs, and only when the run returned items. OpenAI indexes the files in the background after they are added; the vector store shows them as in progress until then. If adding the files fails, the uploaded files are deleted again. An export that takes longer than 8 minutes is stopped; for very large runs, set Max items per run.

Older runs: files from earlier runs stay in the vector store. To keep it small, delete old files in the OpenAI dashboard, or set an expiration policy on the vector store.

Limits: each OpenAI Vector Store automation of this kind uploads at most 30 times an hour, and an account at most 500 exports a day (all result deliveries together). If the key is refused, the account has no credit left, or the vector store doesn't exist, the delivery history on the automation's page says so.

API

A webhook with an openai notification. Only ACTOR.RUN.SUCCEEDED is allowed.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to OpenAI",
    "notification": {
      "channel": "openai",
      "openaiApiKey": "sk-proj-…",
      "vectorStoreId": "vs_abc123"
    }
  }'

maxItems is optional; leave it out or set it to null to add every item. Responses carry notification.openaiSettings with the vectorStoreId, maxItems and a keyHint (sk-… and the last four characters); the key is never returned. On PUT, leaving out openaiApiKey keeps the saved one. In an automation, use a delivery with "channel": "openai" and the same fields.