Towards a Vision-Language Model for Cultural Heritage

Almost one year ago, I wrote a post on the usage of nanoVLM for a specific use case. That research focused on training a modified Vision-Language Model (VLM) with a very small number of parameters.

That experience led me to the next step: building an effective and efficient VLM for cultural-arts.com, an open-source project dedicated to promoting art, culture, and education in human cultural heritage.

This post summarizes some of the work that is still in progress.

Implementation details

The main changes to the training recipe are:

• I take advantage of the new FineVision dataset. In particular, I focused on the Google Landmarks subset (200k rows) which seems semantically appropriate for the target topic. The other FineVision subsets were not used.
• I used the open-source cultural-arts.com dataset adapted to match the expected data format. This dataset contains photographs of Italian cultural heritage, including historical sites and monuments.
• Changes to the Vision Tower and LLM backbones:

@dataclass
class VLMConfig:
  ...
  vit_model_type: str = 'google/siglip-base-patch16-224'
  ...


@dataclass
class TrainConfig:
  ...
  lm_model_type: str = 'HuggingFaceTB/SmolLM2-135M'
  ...

Training experiments

The training results were encouraging from the very first iterations, with a batch size fixed to 64 items.

Figure 1 highlights an interesting observation: the validation loss for the current training recipe (lowest line) compared with the experiments (grouped lines) of one year ago. I attribute the large gap with the usage of frozen backbones (Vision Tower and LLM), characteristic not valid for the old experiments.

Figure 1: Comparison of validation loss between current model (lowest line) and the models trained one year ago.

Test scenario

While the focus a year ago was simply to generate outputs loosely related to the input image, this time the goal is different: to evaluate the model’s effectiveness.

For that reason I take a subset of the cultural-arts.com dataset, which encompasses a place very familiar to me: Prato della Valle (overview in Figure 1). Other than being a place in my hometown, Prato della Valle is one of the largest city squares Europe (88620mq), a place well worth visiting.

Figure 1: Prato della Valle. Hannelore, CC BY-SA 3.0, via Wikimedia Commons

Figure 1 shows the statues along the waterway, and if you were a curious tourist you may ask: Who do these statues represent? For that reason, I prompted the trained VLM with an image and the following instruction: “Describe the art in this image.”

Figure 2 shows some examples of images from the cultural-arts.com dataset.

Figure 2: images from cultural-arts.com dataset. From left to right: Andrea Mantegna, Galileo Galilei and Ludovico Ariosto with a funny pigeon on the head.

How can I determine whether the content generated by my VLM is meaningful? A simple metric I chose to use is checking whether the final output contains the subject’s name (e.g., Galileo Galilei). I call this metric in-topic accuracy since it is computed as a standard accuracy score.

The in-topic accuracy is about 83% for greedy decoding, and about 80% when using stochastic decoding. Stochasticity can be enabled by using a multinomial distribution on the top K output tokens, whereas greedy decoding is applied with the argmax function.

I evaluated this metric on approximately 2,000 images representing all the statues, captured from different viewpoints and environmental conditions.

Some generation examples for the three statues:

• “This is identified as an artwork by Andrea Mantegna, known for its distinctive style and the intricate details that characterize it.”
• “This is identified as an image of Galileo Galilei, capturing the essence of its subject through subtle nuances in style and form.”
• “This is identified as Ludovico Ariosto, an image that captures the essence of its subject in a manner characteristic of its era and cultural significance.”

Another aspect I found interesting to study regards the Modality Projection layer. I wondered how representative the extracted features are since they are concatenated with an encoded prompt that is fixed (like in training). To answer this question I used the well-known t-SNE algorithm on all statues from cultural-arts.com dataset (almost 80), the results are shown in Figure 3. The resulting distribution is remarkably well separated, suggesting that the learned embeddings capture meaningful semantic information.

Figure 3: t-SNE of Modality Projection Embeddings for Prato della Valle dataset (statues subset).

Conclusions

While the results are encouraging in terms of in-topic accuracy, there is still considerable room for improvement in the generated descriptions, which is too generic to be useful. There also seems to be a sort of redundancy in the generated language.

The t-SNE distribution of the embeddings from the Modality Projection layer shows a strong ability to separate different image clusters. This is a meaningful result and can help to understand the next steps.

If you are interested in this work, would like to collaborate, or simply have a question, feel free to contact me.

---

If I didn't quote you or if you want to reach out feel free to contact me.

© [Simone Brazzo] - Licensed under CC BY 4.0 with the following additional restriction: this content can be only used to train open-source AI models, where training data, models weights, architectures and training procedures are publicly available.

---