🌹TL;DR: VidaForge is a practical research-infrastructure project for turning raw videos into clean, structured, and reusable training data assets for video foundation model pretraining.
Earlier this year, I finished an internship with the Hailuo video team at Minimax. That was the first time I fully felt how much of a video foundation model is shaped by its data pipeline. Model architectures and training code are much easier to put into papers and open-source repositories. The data pipeline is often hidden behind the system.
Before that internship, I had also read several papers on video generation models, self-supervised video models, and multimodal foundation models. Most of them mention filtering, deduplication, annotation, and large-scale video data. But the path from raw videos to training-ready data is rarely written down in detail. Industry reports usually give a high-level summary. Academic projects often release processed datasets, video URLs, or relatively simple download scripts. Because of copyright constraints, scale, and engineering cost, the final dataset is usually visible, while the construction process disappears.
That left me with a more concrete question:
If I start from video pretraining itself, what should a data pipeline look like if it needs to turn raw videos into data that training code can consume, while still letting a researcher inspect each intermediate state?
Fig. 1. Public detail on data processing in recent video and visual foundation model reports. The y-axis shows my approximate English word counts for arXiv paper sections describing data preparation or processing. Points above 1000 words are capped at the top tick.
The count in Fig. 1 is rough, but it made me ask a slightly uncomfortable question. As these models get stronger generation after generation, is data processing really getting simpler? Did the data recipe stay almost unchanged while the model quality kept improving, and public reports simply got better at writing “we carefully filter the data”? A more plausible explanation is that the closer a part is to core capability, the harder it becomes to describe in a public report.
I also looked at existing open-source data processing systems, including NeMo Curator and Data-Juicer. They are powerful, and they already cover video data processing. My first reaction, however, was that the entry point felt too large. They cover many modalities, many operators, and many concepts. Text data processing still takes a large share of attention in this space. If I started there, my first step would be learning a large general-purpose system.
I wanted to start from a narrower question.
Video data feels increasingly important to me. There is a huge amount of it, and new video keeps being produced. It carries visual, temporal, semantic, and physical-world information at the same time. If future models need to move from screens toward the real world, video will likely become one of the most important pretraining sources. So I wanted to think through video data directly: raw video comes in, what stages should it go through, and how should it become something a training system can actually read?
This is why I started building VidaForge from scratch. The name comes from Video Data Forge. The focus is video data, and the process of forging it step by step into training assets. Existing frameworks can solve many similar problems, but building the pipeline myself forced me to answer basic design questions. What exactly are the inputs and outputs? Which intermediate states should be saved? What belongs in formal metadata, and what should stay as files on disk? Should failed samples and rejected samples be kept? How should a long pipeline be split into stages and steps? How can differences between data recipes be brought back into training experiments?
With coding agents becoming useful, the implementation itself is no longer the hardest part. The hard part is deciding what should be built. Codex can help me write code, change code, and run tests. It cannot decide what a video pretraining data pipeline should look like.
There is also something slightly strange about doing this in an academic lab. The natural research loop in a lab is often to change the model, change the loss, run an ablation, and look at metrics. VidaForge started from problems that look much less elegant on paper: where raw videos live, how broken videos are skipped, how transcode failures are recorded, how parquet schemas are shaped, how hundreds of thousands of clips are processed in parallel, whether rejected samples are kept, and what format the training framework actually reads.
These problems sound like industrial infrastructure. After working on them for a while, I realized that they directly affect the research questions. Does cleaning make training more stable? Does caption granularity change convergence? What should dedup remove, and what should it leave behind? If these intermediate decisions are not recorded, training results become much harder to explain. This is where the “academic lab” part matters: I wanted to build an industrial-looking data pipeline with the budget, iteration speed, and uncertainty of a lab project. The scale can start smaller, but the design has to be clear. Every change to a data recipe should be traceable all the way to the training result.
One conversation during my internship also pushed me in this direction. I once asked a senior leader on the Hailuo video team what kinds of academic papers he paid attention to. He said there were two kinds: work with strong originality and a view several years into the future, like the kind of work people associate with Kaiming He; and infrastructure work. That answer stayed with me. It made me feel that infrastructure can also have a research position, as long as it turns a vague research question into something that can be run, inspected, and compared.
VidaForge started from that thought. It is a video pretraining data pipeline from raw videos to training assets: standardized clips, metadata, quality signals, duplicate groups, captions, tags, selection decisions, and data formats that training frameworks can consume. I want to use it to answer one question:
For video foundation model pretraining, what should a practical, inspectable, iterative data pipeline look like if it also needs to be validated by real training experiments?