LongCat AI Auto-Director Workflow in ComfyUI

LongCat AI Auto-Director: Build Fully Automated Video Workflows

Tired of manually extending AI videos clip by clip? Frustrated with writing endless prompts for each new scene? This guide shows you how to build a fully automated LongCat workflow that uses Large Language Models (LLMs) to generate complete storyboards and loop systems to create long-form videos automatically.

TL;DR

This tutorial covers building an automated LongCat Video workflow that:

• Uses LLM prompt engineering to automatically generate scene-by-scene storyboards
• Implements loop systems in ComfyUI for continuous video extension
• Transforms single images into multi-minute cinematic narratives
• Supports text-to-video with automatic story generation
• Runs on RunningHub cloud platform with free credits

Reading Time: 15 minutes | Difficulty: Advanced | VRAM Required: 12GB+

Key Takeaways

1. LongCat Model Integration: The LongCat model is integrated into Kijai's WanVideo extension after approximately two weeks of development effort, supporting text-to-video, image-to-video, and video continuation.

2. Acceleration LoRA: Using the acceleration LoRA reduces sampling steps from 30+ down to just 12 while maintaining quality—similar to LightX2V acceleration.

3. Overlap Frames: Setting 13 frames of overlap between video segments ensures consistency during extension, preventing jarring transitions.

4. 4N+1 Frame Rule: Video frame length should follow the 4N+1 rule (e.g., 93 frames) for optimal generation quality.

5. LLM Prompt Engineering: Connecting an LLM to automatically generate scene prompts eliminates manual prompt writing and creates more creative, fantastical scenarios than humans typically imagine.

6. Loop-Based Extension: Using ComfyUI's loop system allows unlimited video extension with standardized workflows, though quality degrades after 5+ extensions.

Understanding the LongCat Model

LongCat Video from Meituan is a major model that boasts powerful generation capabilities:

What makes LongCat special is its ability to "continue the story"—seamlessly extending your video clip by clip to create truly long-form narratives with smooth transitions between segments.

The Automated Workflow Architecture

Standard vs Automated Approach

Standard Workflow Problems:

• Write 5 separate prompts for 5 video segments
• Manually connect each extension node
• Limited creative imagination for complex scenes
• Time-consuming iteration process

Automated Workflow Solution:

• Single image or theme input
• LLM generates complete storyboard automatically
• Loop system handles all extensions
• Press play and watch the magic happen

Basic Workflow Components

The workflow requires these essential components:

1. LongCat FP8 Main Model
2. Acceleration LoRA (reduces 30+ steps to 12)
3. Wan 2.1 VAE
4. UMT5 Text Encoder (Wan 2.1 encoder)
5. LLM for prompt engineering (optional but recommended)

Image-to-Video Workflow Setup

Model Loading Configuration

Main Model: LongCat FP8
LoRA: Acceleration LoRA (for 12-step sampling)
VAE: Wan 2.1 VAE
Encoder: UMT5 (Wan 2.1)
Chunk Swap: 20 chunks enabled

Resolution Settings

Resolution must follow specific guidelines. Use resolutions from the official table:

Sampler Configuration

Sampler: LongCat-specific scheduler
Steps: 12 (with acceleration LoRA)
Shift: 12 (fixed value)
Overlap Frames: 13
Frame Length: 93 (follows 4N+1 rule)

First Generation vs Extension

Understanding the difference between initial generation and extension is crucial:

First Generation (Initial Video)

• Input: Single source image
• Output: Start image + first video segment
• Source Image: Not available (only image input)
• Node: Standard sampling pipeline

Extension Steps

• Input: Previous video segment
• Output: Extended video with overlap blending
• Source Image: Video from previous step
• Additional Node: "Image batch extend with overlap" for seamless connection

The extension workflow adds a critical node that connects the original video from step 1 with the newly extended video, creating the seamless extended output.

LLM-Powered Prompt Engineering

This is where the magic happens. Instead of manually writing prompts for each segment, we use an LLM to automatically generate a complete storyboard.

Prompt Engineering Setup

The LLM receives:

1. The initial reference image
2. Instructions to advance the video plot
3. Request to write a prompt for each scene
4. Number of shots to generate

Example LLM Output

For an image of a classical Chinese beauty with text "Post-80s programmer":

Shot 1: The girl starts to turn around and puts down the brush
Shot 2: She walks up to a large glass door, lighting changes dramatically
Shot 3: She attempts to open the glass door
Shot 4: She pushes through the door with a camera tracking shot

Why LLM Prompts Are Better

• Creates fantastical scenarios humans wouldn't imagine
• Maintains narrative coherence across scenes
• Generates detailed, specific descriptions
• Saves hours of manual prompt writing

Building the Loop System

The automated workflow uses ComfyUI's loop system for unlimited extension:

Loop Configuration

Initial Video: Video from first generation (video_0)
Loop Start: Connected to initial video
Loop Count: Number of extensions (e.g., 2 for 3 total shots)
Prompt Index: loop_index + 1 (starts from 1 since 0 is used in first generation)

Loop Flow

1. First Generation: Creates initial video segment
2. Loop Iteration 1: Extends video, updates initial_value_1
3. Loop Iteration 2: Further extends, combines all segments
4. Final Output: Complete multi-segment video

Prompt Line Node

The "Prompt Line" node extracts specific prompts from the LLM-generated list:

• Index 0: Used for first generation
• Index 1+: Used in loop iterations
• Ensures correct prompt matches each extension step

Text-to-Video Adaptation

Converting the image-to-video workflow to text-to-video is straightforward:

What to Remove

• Image input section
• Extra latent (reference image condition)

What to Modify

Prompt Engineering Changes:

• Instead of providing an image, provide a theme
• LLM generates prompts based on text description only

Example Theme

Theme: "A female secret agent attempting to open a safe"
Number of shots: 3

LLM-Generated Storyboard:

1. Female spy approaches an old safe
2. She works on the combination lock
3. Surprise explosion scene

Quality Considerations

Extension Degradation

As video extension continues, quality issues emerge:

Recommendations

• Limit to 3 shots for best face quality
• Watch for unnatural artifacts similar to Flux-generated images
• Use higher resolution if VRAM allows
• Consider post-processing for longer videos

RunningHub Cloud Deployment

All workflows are available on RunningHub, an excellent online ComfyUI workbench:

Available Workflows

RunningHub Benefits

• New models and extensions available immediately
• 1,000 free credits on registration
• 100 bonus credits daily
• No local GPU required

Troubleshooting

Common Issues

Prompt Control Weakening:

• As extension continues, prompt influence decreases
• Solution: Use stronger action verbs, limit extensions

Face Distortion:

• Appears after 5+ extensions
• Solution: Limit to 3 shots, use face restoration post-processing

Resolution Errors:

• Using non-standard resolutions causes issues
• Solution: Always use resolutions from the official table

Frequently Asked Questions

Q1: How long can automated videos be?

A: Technically unlimited, but quality degrades after 5 extensions. For best results, limit to 3 shots (approximately 15-20 seconds).

Q2: Can I use any LLM for prompt engineering?

A: Yes, any capable LLM works (GPT-4, Claude, local models). The key is proper prompt engineering to generate scene-by-scene storyboards.

Q3: Why use the 4N+1 frame rule?

A: This rule ensures optimal generation quality by aligning with the model's internal processing. Frame lengths like 93 (4×23+1) work best.

Q4: How does the overlap system work?

A: 13 frames overlap between segments ensures smooth transitions. The overlap is blended during extension to prevent jarring cuts.

Related Resources

1. LongCat Video Complete Guide - Comprehensive overview of all LongCat Video features

2. ComfyUI LongCat Tutorial Series Ep 69 - Step-by-step tutorial for beginners

3. LongCat FP8 Workflow: Memory-Efficient Video - Optimize for lower VRAM systems

4. LongCat Video Extra Long Generation - Advanced techniques for multi-minute videos

5. Stable Video Infinity WAN 2.1 Tutorial - Alternative infinite video approach

This tutorial is based on a comprehensive deep dive video with 2.1K views, demonstrating automated LongCat workflows using LLM prompt engineering and ComfyUI loop systems.