Lipdub Videos to Any Language

Hacker News

TLDR: Translate, dub, and lip-sync videos into 29+ languages. Here's an example: https://app.vivalabs.ai/?modal=videoShared&shareID=c955c97b-... I was creating a few educational math videos for kids in Latin America last summer, and I noticed they weren't easily following subtitles. I wanted to make the video content more engaging for them, but I'm still a ways away from Spanish proficiency. Went down the rabbit hole of audio + video AI models... and now we have the initial version of Viva Labs! Given a video, we first translate the audio to the target language using a pipeline of ASR models, translation APIs, & LLMs; users can edit these translations to fix any errors. Then, we dub the audio with similar voices or voice clones. Finally, we sync the speakers lip movements to match the dubbed audio. Our early users have surprised us with some of the common use cases like dubbing online course content, product explainers + marketing promos, podcasts, and international newscasts. Still a lot of work to do to create full immersion. The following areas of technical exploration we're pursuing: 1) matching the output audio's tone, prosody, & emotion with that of the input voice using speech to speech models, 2) training lip sync models with more robust lip sync for extreme poses and occluding objects, 3) using LLMs to produce translations that are more colloquial and more closely match the pacing of original audio. Here's a sample of Mira Murati's GPT4o announcement lip dubbed to Russian: https://app.vivalabs.ai/?modal=videoShared&shareID=c955c97b-... Looking forward to seeing what videos you dub! You can dub 3 minutes of video for free at https://app.vivalabs.ai or you can ask our twitter bot https://x.com/VivaDubs to audio dub a video on twitter.

Visit product View on Hacker NewsJun 17, 2024

Share card

Actual performance

4points

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Indie HackersFits the IH revenue-focused audience · Missing: supports, reddit linkedin, podcasting

91%91% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Product HuntOn track for Day 1 leaderboard · Strong signals: model, user, new · Missing: mac, agents, macos

86%86% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Strong signals: video, users, way · Missing: mobile apps, ios, personal

63%63% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsMay not resonate with HN audience · Strong signals: ide, pipe, io · Missing: https docs, excited, just released

41%41% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoMay struggle as an AppSumo deal · Strong signals: users · Missing: plus, platform, intuitive

40%40% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Strong signals: training · Missing: arr, mrr, revenue

22%22% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Strong signals: audio · Missing: web3, chat, crypto

2%2% predicted probability of success on BetaList, based on ML models trained on real launch data.

Correct prediction on native model

Similar products

A toy interpreter for a minimalistic imperative language in 1.3k LoC63%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A toy interpreter for a minimalistic imperative language in 1.3k LoC

Hacker News17

Implementing a Lispy Language in Haskell: Part 165%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Implementing a Lispy Language in Haskell: Part 1

Hacker News6

The Lily Programmming Language60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The Lily Programmming Language

Hacker News16

YDML – YAML DOM Markup Language59%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

YDML – YAML DOM Markup Language

Hacker News5

Language proposal - dave60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Language proposal - dave

Hacker News24

The Termcat Markup Language60%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

The Termcat Markup Language

Hacker News1

2048 in PDP-8 Assembly language56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

2048 in PDP-8 Assembly language

Hacker News2

C# language kernel for IPython55%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

C# language kernel for IPython

Hacker News2

JedLang, my first attempt at a language57%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

JedLang, my first attempt at a language

Hacker News59

A brainfuck language kernel for IPython and Jupyter45%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

A brainfuck language kernel for IPython and Jupyter

Hacker News4

Browse all products Analyze your launch