The idea
Rotoscoping, cutting a subject out of video, frame by frame, is one of the most tedious jobs in visual effects. Studios pay people to trace outlines for days. When Meta released SAM2 and zero-shot detection matured with Grounding DINO, it struck me that the entire craft could collapse into a sentence: describe what you want isolated, and let the models do the tracing.
RotoAI is that sentence turned into a product. You upload a clip, type a natural-language prompt, “black dog”, “boy with orange backpack”, pick an effect, and get back a rendered video where the subject is isolated and the background transformed. No masks, no keyframes, no After Effects.
Six effects
Once the subject is segmented, the render pipeline applies one of six professional effects: Bokeh Blur for a shallow depth-of-field look, Chroma Key for green-screen compositing, B&W Color Pop and cinematic Color Pop for that isolated-subject aesthetic, Neon Glow for a cyberpunk outline, and Luminous Edge for a light-painting contour.
The architecture trick
The interesting product decision is where the computation lives. SAM2 plus Grounding DINO need a serious GPU, which most people don't have. So RotoAI splits in two: a responsive Next.js frontend runs on your machine, while the inference server runs on a free Google Colab T4, exposed through a secure Ngrok tunnel. You paste one URL, the status light turns green, and your laptop suddenly has a datacenter GPU.
Free hardware demands resilience. Videos are processed in 5-second chunks so a long clip never blows the 16 GB of VRAM, resolution auto-scales when the memory budget gets tight (never below 50%, never above 90%), and model weights are cached in Google Drive so a fresh Colab session boots in minutes instead of re-downloading 800 MB of models.
That architecture surfaces in two practical moments: the main studio keeps the Colab connection, prompt and effect selection in one place; Test Detection validates the subject on a single frame before a full render, so a wrong target costs seconds, not GPU minutes.
What it taught me
RotoAI was a lesson in productising constraints. Every limitation (ephemeral Colab sessions, tunnel timeouts, VRAM ceilings) had to become a designed behaviour instead of a crash: drive-cached weights, chunked rendering, auto-scaling, a Test Mode to fail fast on a single frame before spending GPU-minutes on a full clip. Making cutting-edge AI feel dependable on free infrastructure was the real product challenge, and the part I enjoyed most.
