Skip to content

Search a scene in a video using natural language with X-CLIP

License

Notifications You must be signed in to change notification settings

Davegdd/Semantic-video-search

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Semantic video search

Search a scene in a video using natural language with the multimodal model X-CLIP which combines NLP and CV.

(click on GIF below to see in higher quality)

Readme GIF

This app allows you to find a scene in a video by just describing it in natural language. It uses X-CLIP, an extension of the model CLIP (Contrastive Language-Image Pre-Training) which forms parts of the current trend of multimodal models and more specifically the trend in using natural language as a training signal for learning about a domain other than language. Using text as the supervision allows for impressive zero-shot performance and generalization.

Just input a YouTube video URL and a description and click search. The time window is the duration of the chunks the video will be split into for analysis (the 10 seconds default usually works fine), the higher this value the shorter the runtime and the less RAM is used but also maybe the lower the performance. The results will be ordered from most to least relevant and will cover the whole video, so what you are looking for is probably on the top 5 or so results, being less related the lower you go. The model has several limitations that are really well explained in the corresponding papers and it won’t get everything right. Nevertheless it does show an amazing level of specificity and usually offers relevant results and it is an example of the fascinating near future in multimodal AI research.

Install

Option 1 Docker

Download the docker-compose file in a local folder and run (make sure you have Docker installed):

docker-compose up

Wait a couple minutes so that all dependencies and the model are downloaded and you stop seeing the "http connection error..." and go to localhost http://localhost:8501/

Option 2 Colab (with GPU acceleration)

There is a huge difference in runtime from running with CPU vs GPU (for a 20 minute video it takes just 2-3 minutes in a GPU vs 30 minutes in a CPU!) so using GPU is strongly recommended and depending on the use case even the only practical option. Open the Colab notebook below to run with free GPU in 3 clicks:

Open In Colab

About

Search a scene in a video using natural language with X-CLIP

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published