video-processing

Video Processing

Purpose

Turn long-form video into processed outputs: transcripts, short clips, vertical format with captions, thumbnails. Uses FFmpeg, yt-dlp, and transcription services. All operations via shell commands.

When to Use

• User requests video clipping, transcription, or processing.
• An evaluation or research task involves video content.
• Content creation requires extracting highlights from longer video.
• A surplus compute task involves video analysis.

Prerequisites

Required tools (install if missing):

• ffmpeg and ffprobe — video processing
• yt-dlp — video downloading from 1000+ sites
• Transcription: YouTube auto-subs (free), or Groq/OpenAI Whisper API

Check availability:

which ffmpeg ffprobe yt-dlp 2>/dev/null

Pipeline

Phase 1: Intake

From URL:

yt-dlp --dump-json "URL" 2>/dev/null | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'Title: {d[\"title\"]}')
print(f'Duration: {d[\"duration\"]}s')
print(f'Resolution: {d.get(\"width\",\"?\")}x{d.get(\"height\",\"?\")}')
"

From local file:

ffprobe -v quiet -print_format json -show_format -show_streams "file.mp4"

If duration > 2 hours, ask user to specify a segment range.

Phase 2: Download

# Best quality up to 1080p with audio
yt-dlp -f "bv[height<=1080]+ba/b[height<=1080]" -o "source.mp4" "URL"

# Also grab auto-subtitles if available (avoids transcription entirely)
yt-dlp --write-auto-subs --sub-lang en --sub-format json3 \
  --skip-download -o "source" "URL"

If source.en.json3 exists, skip to Phase 4 (transcription already done).

Phase 3: Transcription

Priority order — use the first available:

1. YouTube auto-subs (already downloaded in Phase 2) — free, instant
2. Groq Whisper API — fast cloud, free tier available

   curl -s https://api.groq.com/openai/v1/audio/transcriptions \
     -H "Authorization: Bearer $API_KEY_GROQ" \
     -F [email protected] -F model=whisper-large-v3 \
     -F response_format=verbose_json -F timestamp_granularities[]=word
   

3. OpenAI Whisper API — reliable, paid
4. Local Whisper — if installed, slowest but free

   whisper source.mp4 --model small --output_format json \
     --output_dir . --language en
   

Extract audio first if sending to API:

ffmpeg -i source.mp4 -vn -acodec libmp3lame -q:a 2 audio.mp3

Phase 4: Segment Selection

This is the core value step. Analyze the transcript and select 3-5 segments (30-90 seconds each) based on:

Selection criteria:

• Hook in first 3 seconds — starts with something attention-grabbing
• Self-contained — makes sense without watching the full video
• Emotional peak — surprise, humor, insight, controversy
• High insight density — says something valuable concisely
• Clean ending — ends on a punchline, conclusion, or cliffhanger

Rules:

• Start mid-sentence for stronger hooks when appropriate
• End on punchlines or key statements, not trailing off
• Avoid segments that require heavy visual context to understand
• Spread selections across the video (don't cluster)
• Each segment gets: exact timestamps, suggested title (<60 chars), one-sentence virality reasoning

Phase 5: Extract and Process

For each selected segment:

Extract clip:

ffmpeg -ss [start] -to [end] -i source.mp4 \
  -c:v libx264 -c:a aac -preset fast -crf 23 clip_N.mp4

Vertical crop (9:16 for shorts/reels):

# Center crop (loses sides)
ffmpeg -i clip_N.mp4 -vf "crop=ih*9/16:ih:(iw-ih*9/16)/2:0,scale=1080:1920" \
  -c:a copy clip_N_vertical.mp4

# Letterbox (keeps everything, adds black bars)
ffmpeg -i clip_N.mp4 -vf "scale=1080:-2,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" \
  -c:a copy clip_N_vertical.mp4

Generate SRT captions from transcript:

• 8-12 words per subtitle line
• 2-3 seconds per subtitle
• Break at natural pauses and sentence boundaries
• Max 42 characters per line (mobile readability)

Burn captions into video:

ffmpeg -i clip_N_vertical.mp4 \
  -vf "subtitles=clip_N.srt:force_style='FontSize=22,FontName=Arial,\
PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,Outline=2,\
Shadow=1,MarginV=60,Alignment=2'" \
  -c:a copy clip_N_captioned.mp4

Generate thumbnail:

# Frame at 2 seconds in
ffmpeg -ss 2 -i clip_N.mp4 -frames:v 1 -q:v 2 clip_N_thumb.jpg

Phase 6: Report

# Video Processing Report

**Source:** [title or filename]
**Duration:** [total duration]
**Clips generated:** N

| # | Title | Duration | File | Size |
|---|-------|----------|------|------|
| 1 | [title] | [duration] | clip_1_captioned.mp4 | [size] |

## Segment Reasoning
1. **[title]** ([start]-[end]): [why this segment was selected]

File Size Limits

If output exceeds platform limits, re-encode:

# Target ~45MB for Telegram (50MB limit)
ffmpeg -i input.mp4 -c:v libx264 -b:v 1500k -c:a aac -b:a 128k output.mp4

Output Format

job_id: <CLIP-YYYY-MM-DD-NNN>
source: <URL or filepath>
source_duration: <seconds>
clips:
  - number: <1-N>
    title: <short title>
    start: <HH:MM:SS>
    end: <HH:MM:SS>
    duration: <seconds>
    file: <output filepath>
    size_mb: <file size>
    format: <horizontal | vertical>
    captioned: <true | false>
    virality_reasoning: <one sentence>
transcription_method: <youtube_auto | groq_whisper | openai_whisper | local_whisper | none>

References

• docs/reference/gemini-routing.md — For video content analysis via Gemini API
• FFmpeg documentation: https://ffmpeg.org/documentation.html
• yt-dlp documentation: https://github.com/yt-dlp/yt-dlp