Original_caption

[fbj2333]

In your paper‘s section 3.2.1, you mention that utilizing original caption can enhance the model's answer. But in your prompt i haven't found any content about the original caption. So GPT4V only uses images to produce data, including caption and instruction? I'd like to know if the original caption used in your data synthesis or model inference procedure.

[g-h-chen]

Hi @fbj2333 ,

Thanks for your interest on our work. A quick answer is: For LAION images, GPT-4V only uses image to generate caption, instruction and answer. The original LAION caption is NOT used. It seems that we did not mention "the original caption" in section 3.2.1.
Please correct me if we miss anything.

Best,
Guiming

[fbj2333]

I see, in Section 3.2.1, you mention additional caption, and i misunderstood it as original caption. By the way, the Chinese edition's allava has about thousands of wrong data. For example:
{
"id": "allava_laion_cap_70294",
"conversations": [
{
"from": "human",
"value": "现在需要帮我将下面关于的问答内容进行翻译，使其成为一个中文的问答。Q: ' Could you please interpret the image and write a detailed caption?"
},
{
"from": "gpt",
"value": "以上是我需要的翻译。"
}
],
"original_caption": "Khal and Khaleesi Mugs",
"url": "http://media2.onsugar.com/files/2011/12/51/2/192/1922507/b1365c2a2c7c0971_e33fee29c7a6a726_Screen_shot_2011-12-16_at_4.17.xxxlarge/i/Khal-Khaleesi-Mugs.jpg"
}
Will you update to filter data like this?

[g-h-chen]

Hi @fbj2333 ,

Thanks for reporting the issue. We have temporarily made the HF repo private and will update the data soon.

Best,
Guiming