Generation Pipeline

Concurrent AI workers, structured prompts, schema validation, and retry logic.

Overview

The generation pipeline takes a set of niches and a content schema and produces validated JSON content for each niche-subtopic combination. The pipeline runs in stages:

1. Subtopic expansion — Pull subtopics from each selected niche
2. Deduplication — Remove overlapping subtopics across niches
3. Title generation — Build deterministic titles from the schema's TitlePattern
4. Prompt assembly — Combine niche context, schema JSON, and prompt template
5. Concurrent generation — Send prompts to the AI provider in parallel
6. Validation — Check output against schema constraints
7. Retry — Re-generate failed pages with an adjusted prompt

Concurrent Workers

The generation engine runs multiple AI requests in parallel from your local machine. Concurrency defaults to 5 workers and can be tuned with --workers <n> to stay within your AI provider's rate limits.

contento collections generate col_8f3a...

Prompt Building

The AI receives a single prompt composed of three parts:

1. System context (from niche)

The niche's audience, painPoints, monetization, and contentThatWorks fields are injected to steer tone and examples.

2. Prompt template (from schema)

The schema's PromptTemplate contains tokens like {niche}, {subtopic}, {count}, and {schema} that are replaced with actual values.

3. Schema definition

The full SchemaJson is serialized and included so the AI knows the exact JSON structure to produce, including field types, array sizes, and constraints.

// Assembled prompt (simplified)
{
  "system": "You are writing for: Software developers, DevOps engineers.
    Key pain points: tool evaluation, integration complexity.
    Monetization context: freemium, per-seat SaaS.",

  "user": "Generate a JSON object with exactly 10 creative,
    actionable ideas about CI/CD pipelines for the Developer Tools
    audience. Each idea must have a title (5-12 words),
    description (50-200 chars), and difficulty (easy/medium/hard).
    Return ONLY valid JSON matching this schema:
    {\"type\":\"object\",\"required\":[\"intro\",\"items\",\"conclusion\"],...}"
}

Schema Validation

After the AI returns JSON, it is validated against the schema's constraints. Validation checks:

Rule	Example	On Failure
JSON parse	Output must be valid JSON	Retry
Required fields	"required": ["intro", "items"]	Reject
Exact item count	"exactCount": 10	Reject
Min/max string length	"minLength": 50, "maxLength": 200	Reject
Enum values	"enum": ["easy", "medium", "hard"]	Reject
Array min/max	"min": 15, "max": 30	Reject

Rejected pages are queued for retry. The retry prompt includes the validation error message, which helps the AI correct its output on the second attempt.

Retry Logic

When validation fails, the engine retries once with an adjusted prompt:

1. The original prompt is resent with an additional instruction: "Your previous response failed validation: [error details]. Please fix and return valid JSON."
2. If the retry also fails, the page is marked as failed and logged with the validation error to ~/.contento/logs/.
3. Failed pages do not block the rest of the batch from completing.

# Check generation job status
contento collections generate col_8f3a... --json

{
  "jobId": "gen_4b7e...",
  "status": "completed",
  "total": 48,
  "passed": 46,
  "retried": 4,
  "retriedAndPassed": 2,
  "failed": 0,
  "avgGenerationTimeMs": 3200
}

SSE Streaming

The CLI displays a real-time progress bar during generation. For programmatic monitoring (agents, CI/CD), use the --json flag to get NDJSON streaming output:

contento collections generate col_8f3a... \
    --schema "idea-list" \
    --niches "niche_01..." \
    --json

{"type":"progress","current":0,"total":48,"done":false}
{"type":"progress","current":5,"total":48,"done":false}
{"type":"progress","current":22,"total":48,"done":false}
{"type":"progress","current":48,"total":48,"done":true}

Event types:

Event Type	Description
progress	Overall job progress with completed/total counts
page	Individual page result (passed, retried, or failed)
done	Job complete with final tallies
error	Fatal error that stopped the job

CLI Commands

Command	Description
contento collections generate	Create a collection and start generation
contento collections list	List collections and their status
contento publish	Publish generated pages

See the CLI reference for all available flags and options.