← All work

01 · GENAI · VIDEO · OPEN-WEIGHTS

CreatorOS

Repository codename · Trend-First Creator Intelligence

A creator dashboard that takes a goal and returns a finished vertical video — collecting live signals, researching the topic against real sources, writing the script, rendering the frames, speaking the voiceover and synthesising the music. Every generative stage runs on open-weights models, and every stage tells you whether a model actually ran.

Role
Architecture + full build
Backend
FastAPI · SQLAlchemy · PostgreSQL
Frontend
React · Vite · TypeScript
Models
FLUX.1-schnell · Kokoro-82M · Whisper
○ Source private Jump to walkthrough ↓
6,310lines of Python API
200screened concepts per keyword
3voice providers, best-first
1080×1920rendered MP4 output

01 — THE PROBLEM

Every AI video tool has the same two credibility problems, and both are about honesty rather than capability.

It hides when the model didn't run. A generative pipeline with eight stages will have stages fail — no token, a rate limit, a malformed response. Most tools silently substitute a template and present the result as generated. The creator then publishes something they believe a model wrote.

It cannot tell you what you are allowed to publish. Generate a video about the Mahabharata and you are retelling a public-domain epic — fine. Generate one that reproduces B. R. Chopra's adaptation, or draws a character “in the style of” a living artist, and you have a rights problem you will not discover until a strike arrives.

CreatorOS treats both as first-class engineering problems: provenance is surfaced per stage, and rights are screened before anything is generated.

02 — ARCHITECTURE

goal → trends → research → ideas → script → packaging → thumbnail → repurpose
                                        │
                                        ▼
                     ┌────────── clip pipeline ──────────┐
                     │  IP guard → character sheet       │
                     │  FLUX.1-schnell    (frames)       │
                     │  Kokoro-82M        (voice)        │
                     │  numpy synth       (music bed)    │
                     │  Pillow + libass   (elements)     │
                     │  ffmpeg            (motion, mux)  │
                     └───────────────────────────────────┘
                                        ▼
                          1080×1920 MP4 + .vtt + .srt
                            + provenance record

03 — BUILD LOG

The decisions that shaped the system, and why each one was made.

  1. Every stage says whether a model actually ran

    Each generative stage carries a badge reading model, live or template, and a banner reports how many stages fell back and why. The same detail travels in plan.stages on the API response. This is the decision the whole project is built around — a pipeline that silently substitutes a template is lying to the person who will publish the result.

  2. Provider-agnostic by two environment variables

    LLM_BASE_URL and LLM_MODEL drive Ollama, vLLM, llama.cpp, TGI and LM Studio identically, with Hugging Face Inference Providers as the no-hardware path and HF_FALLBACK_MODELS behind it. The client remembers the first model that answers, so later calls skip the probing.

  3. A character sheet, because text-to-image has no memory

    Rendering a kids clip scene by scene returns six different creatures — the image model has no recollection of the previous call. Each script therefore carries a character field describing one original design, and the renderer prefixes it to every scene prompt. This is the single fix that makes a multi-scene character video coherent.

  4. The copyright guard separates two things usually conflated

    Public-domain source material is not the same as a modern adaptation of it. The guard allows mahabharat and names a specific public-domain translation to draw from, while blocking br chopra and ramanand sagar; commercial franchises are refused outright. Crucially it rewrites image prompts rather than flagging them — stripping “in the style of” clauses and replacing protected names, because an image model will otherwise draw exactly what it was asked for.

  5. The music is synthesised sample by sample, on purpose

    The bed is a four-chord loop with an arpeggio, a soft bass and a light backbeat, written into a WAV by numpy. No sample, no recording, no music model — chord progressions are not copyrightable and the arpeggio is generated from chord tones rather than written as a tune. That is what makes the audio unambiguously clear to publish, which a licensed music library cannot promise.

  6. Voice degrades through three providers instead of failing

    Kokoro-82M running locally first — 82M parameters, fits in CPU RAM, no token, no network, and roughly 0.8× realtime on an 11th-gen i3, faster than the audio it produces. Then the same family via the Hugging Face router, then the OS speech engine, which is always present and clearly synthetic. Only if all three are unavailable is the clip silent.

  7. Subtitles are timed against measured speech, not declared duration

    Scenes are padded with silence to hit a target length. Timing cues against that padding stretches a four-word caption across twenty seconds. Each scene's audio duration is instead shared out in proportion to cue length — close enough to a single synthesised utterance to read naturally, with no forced aligner in the stack.

  8. One hundred concepts cheap, full scripts on demand

    A keyword produces up to 200 screened concepts across parallel batches, each batch working a different angle family so the library does not collapse into ten restatements of one idea. Near-duplicate titles are dropped. Opening a concept writes and caches the full script — so a 100-result search costs one cheap pass, not 100 full generations.

  9. Two URLs for the same video, because one cannot do both jobs

    Serving the player URL with Content-Disposition: attachment makes browsers refuse to render it inline, so playback and download are split. On the client the download fetches to a blob first: the download attribute is ignored for cross-origin URLs, and the API sits on a different port from the dev server, so a plain link would navigate away instead of saving. Range requests return 206, so seeking works.

  10. Storage reconciles disk against database

    A failed render, a hand-deleted file or a reset database puts the filesystem and the Clip table out of step. GET /v1/storage reports both sides and names the disagreements — orphan files with no row, and rows whose video is gone. Cleanup defaults to a dry run and only deletes when called with ?delete=true.

  11. Model output is a contract, not a suggestion

    The system prompt forbids inventing statistics, studies, dates, names and quotes, and requires factual claims to be phrased so the creator can verify them. Research passes live headlines as evidence of what is being discussed, never as established fact, and returns a fact checklist alongside a risk list. Responses are validated against Pydantic contracts before reaching the UI, so a malformed response degrades to the template instead of rendering garbage.

04 — WALKTHROUGH

Running it end to end.

○ Not deployed Runs locally today — the render pipeline needs ffmpeg, disk for the clip library and either a local model or an HF_TOKEN. Steps below.
  1. Start the frontend

    npm install
    npm run dev            # http://localhost:5173
  2. Start the API

    cd apps/api
    python -m pip install -r requirements.txt
    python -m uvicorn app.main:app --reload --port 8000
  3. Point it at a model

    Pick either path. Setting the two local variables takes priority over the Hugging Face router. On a low-RAM machine prefer a 3B model — a 7B needs roughly 5 GB free.

    # Path A — no local hardware
    HF_TOKEN=hf_...
    HF_MODEL=Qwen/Qwen2.5-72B-Instruct
    
    # Path B — any OpenAI-compatible runtime
    LLM_BASE_URL=http://localhost:11434/v1
    LLM_MODEL=llama3.2:3b
  4. Install the local voice (optional)

    ~338 MB, downloaded once and then used entirely offline. apps/api/models/ is gitignored.

    python -m pip install kokoro-onnx soundfile
    # place kokoro-v1.0.onnx and voices-v1.0.bin in apps/api/models/kokoro
  5. Check what is actually live

    The header badge shows the active model and its round-trip latency, and turns amber when nothing is reachable.

    GET /v1/providers/llm/health
    GET /v1/providers/voice
  6. Run the whole thing

    Or stream it as SSE, one frame per stage, to watch the pipeline advance.

    POST /v1/workflows/creator-plan
    POST /v1/workflows/creator-plan/stream
  7. Or go straight to video

    A keyword becomes a screened concept library; opening one writes the script; generating renders a real 1080×1920 MP4 with progress reported per stage.

    POST /v1/scripts/library          # keyword → concepts
    POST /v1/scripts/concepts/{id}/expand
    POST /v1/clips/generate           # 202, then poll
    GET  /v1/clips/{id}

05 — STACK

PythonFastAPISQLAlchemyPostgreSQLRedisPydanticReactViteTypeScriptFLUX.1-schnellKokoro-82MWhisperffmpegPillowNumPyDocker

The source for this project is not public. Get in touch to discuss it.