I trained my own Vision-Language Model under 1B parameters
Introduction What is it actually like to build and train your own Vision-Language Model nowadays? 👀 This is the question that motivated me to try it out myself. I took a base language model, I plugged in a vision backbone and, training for some hours on a rented GPU on runpod, I tried to get out of it some image descriptions. Nowadays it is pretty simple to download small, good vision encoders and language models; what is usually missing is the recipe to glue them together....