Skip to content

Build a dataset

opencolibri/starter-dataset-builder builds datasets with AI, into .castor/import/datasets/<slug>/. A built dataset is final: loading it (sylius:dataset:load, or a demo shop) needs neither the AI nor the source website.

Set the AI in the .env.local of the project root (or .env):

VariableDefaultDescription
AI_PROVIDERopenrouterollama (local) or openrouter
AI_MODELprovider-specificText / structured-output model
AI_IMAGE_MODELprovider-specificImage generation model
AI_BASE_URLhttp://127.0.0.1:11434Ollama URL (ignored for OpenRouter)
AI_API_KEY—Required when AI_PROVIDER=openrouter
Terminal window
castor sylius:dataset:build --name="Tea House" --description="Loose leaf teas and tea ware"

The AI writes the catalog (20 products at most, --limit to ask for fewer): products with their collection, price and a prompt of their image, then generates the images.

Terminal window
castor sylius:dataset:build --source=existing --name="Tea House" --url=https://www.example-tea.com

The builder discovers the sitemap (from robots.txt, with dedicated handling for Shopify and Magento), picks the product and collection pages, and uses the AI to extract the collections. The AI then selects the products to keep (--limit, 100 by default), assigns each one its collection and a price, and the images are downloaded.

  1. The source writes products.yaml and collections.yaml. Building a dataset again replaces its catalog, and keeps the previous one when the source fails.
  2. A fetched catalog is curated by the AI (above).
  3. When more than eight collections hold products, the AI groups them into a two-level menu (e.g. Men > T-shirts).
  4. dataset.yaml records the name, description, URL and source.
  5. The image of every product and collection is downloaded or generated into media/.
Terminal window
castor sylius:dataset:images tea-house # store the images still missing (after a failed download, for example)
castor sylius:dataset:delete tea-house # built datasets only: catalogs loaded from it are kept

The dataset slug is the slug of the name, or --dataset. A built-in dataset cannot be built or deleted.

Terminal window
castor sylius:dataset:export cellar # .castor/import/exports/cellar.zip
castor sylius:dataset:export tea-house --output=tea.zip

The zip holds a <slug>/ directory: unzipped in src/Datasets/datasets/, it becomes a ready-made dataset.

A project can register its own source, e.g. a CSV export of its current shop. It writes products.yaml and collections.yaml of the slug it receives; products without a collection or a price are curated by the AI, like a fetched catalog:

use OpenColibri\Starter\DatasetBuilder\DatasetSources;
use function OpenColibri\Starter\DatasetBuilder\castor_host_dir;
DatasetSources::register('csv', static function (string $slug, string $name, string $description, ?string $url, int $limit): void {
// Write castor_host_dir($slug) . '/products.yaml' and '/collections.yaml'.
});
Terminal window
castor sylius:dataset:build --source=csv --name="My shop"