作者: Sayak Paul
创建日期 2021/08/08
最后修改日期 2025/01/03
描述: 训练用于预测蕴含的多模态模型。
在本示例中,我们将构建和训练一个用于预测多模态蕴含的模型。我们将使用 Google Research 最近引入的多模态蕴含数据集。
在社交媒体平台上,为了审核和管理内容,我们可能需要近乎实时地找到以下问题的答案
在自然语言处理中,此任务称为分析文本蕴含。然而,这仅限于信息来自文本内容的情况。在实践中,可用的信息通常不仅来自文本内容,还来自文本、图像、音频、视频等多模态组合。多模态蕴含只是文本蕴含对各种新输入模态的扩展。
本示例需要 TensorFlow 2.5 或更高版本。此外,BERT 模型(Devlin 等人)需要 TensorFlow Hub 和 TensorFlow Text。这些库可以使用以下命令安装
!pip install -q tensorflow_text
[[34;49mnotice[1;39;49m][39;49m A new release of pip is available: [31;49m24.0[39;49m -> [32;49m24.3.1
[[34;49mnotice[1;39;49m][39;49m To update, run: [32;49mpip install --upgrade pip
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import random
import math
from skimage.io import imread
from skimage.transform import resize
from PIL import Image
import os
os.environ["KERAS_BACKEND"] = "jax" # or tensorflow, or torch
import keras
import keras_hub
from keras.utils import PyDataset
label_map = {"Contradictory": 0, "Implies": 1, "NoEntailment": 2}
原始数据集可在此处获得。它包含托管在 Twitter 照片存储系统(称为 Photo Blob Storage(简称 PBS))上的图片 URL。我们将使用下载的图片以及原始数据集附带的附加数据。感谢 Nilabhra Roy Chowdhury 协助准备了图片数据。
image_base_path = keras.utils.get_file(
"tweet_images",
"https://github.com/sayakpaul/Multimodal-Entailment-Baseline/releases/download/v1.0.0/tweet_images.tar.gz",
untar=True,
)
df = pd.read_csv(
"https://github.com/sayakpaul/Multimodal-Entailment-Baseline/raw/main/csvs/tweets.csv"
).iloc[
0:1000
] # Resources conservation since these are examples and not SOTA
df.sample(10)
id_1 | text_1 | image_1 | id_2 | text_2 | image_2 | 标签 | |
---|---|---|---|---|---|---|---|
815 | 1370730009921343490 | Sticky bombs are a threat as they have magnets... | http://pbs.twimg.com/media/EwXOFrgVIAEkfjR.jpg | 1370731764906295307 | Sticky bombs are a threat as they have magnets... | http://pbs.twimg.com/media/EwXRK_3XEAA6Q6F.jpg | 无蕴含 |
615 | 1364119737446395905 | Daily Horoscope for #Cancer 2.23.21 ♊️❤️✨ #Hor... | http://pbs.twimg.com/media/Eu5Te44VgAIo1jZ.jpg | 1365218087906078720 | Daily Horoscope for #Cancer 2.26.21 ♊️❤️✨ #Hor... | http://pbs.twimg.com/media/EvI6nW4WQAA4_E_.jpg | 无蕴含 |
624 | 1335542260923068417 | The Reindeer Run is back and this year's run i... | http://pbs.twimg.com/media/Eoi99DyXEAE0AFV.jpg | 1335872932267122689 | Get your red nose and antlers on for the 2020 ... | http://pbs.twimg.com/media/Eon5Wk7XUAE-CxN.jpg | 无蕴含 |
970 | 1345058844439949312 | Participants needed for online survey!\n\nTopi... | http://pbs.twimg.com/media/Eqqb4_MXcAA-Pvu.jpg | 1361211461792632835 | Participants needed for top-ranked study on Su... | http://pbs.twimg.com/media/EuPz0GwXMAMDklt.jpg | 无蕴含 |
456 | 1379831489043521545 | comission for @NanoBiteTSF \nenjoyed bros and ... | http://pbs.twimg.com/media/EyVf0_VXMAMtRaL.jpg | 1380660763749142531 | another comission for @NanoBiteTSF \nhope you ... | http://pbs.twimg.com/media/EykW0iXXAAA2SBC.jpg | 无蕴含 |
917 | 1336180735191891968 | (2/10)\n(Seoul Jung-gu) Market cluster ->\n... | http://pbs.twimg.com/media/EosRFpGVQAIeuYG.jpg | 1356113330536996866 | (3/11)\n(Seoul Dongdaemun-gu) Goshitel cluster... | http://pbs.twimg.com/media/EtHhj7QVcAAibvF.jpg | 无蕴含 |
276 | 1339270210029834241 | Today the message of freedom goes to Kisoro, R... | http://pbs.twimg.com/media/EpVK3pfXcAAZ5Du.jpg | 1340881971132698625 | Today the message of freedom is going to the p... | http://pbs.twimg.com/media/EpvDorkXYAEyz4g.jpg | 蕴含 |
35 | 1360186999836200961 | Bitcoin in Argentina - Google Trends https://t... | http://pbs.twimg.com/media/EuBa3UxXYAMb99_.jpg | 1382778703055228929 | Argentina wants #Bitcoin https://#/9lNxJdxX... | http://pbs.twimg.com/media/EzCbUFNXMAABwPD.jpg | 蕴含 |
762 | 1370824756400959491 | $HSBA.L: The long term trend is positive and t... | http://pbs.twimg.com/media/EwYl2hPWYAE2niq.png | 1374347458126475269 | Although the technical rating is only medium, ... | http://pbs.twimg.com/media/ExKpuwrWgAAktg4.png | 无蕴含 |
130 | 1373789433607172097 | I've just watched episode S01 | E05 of Ted Las... | http://pbs.twimg.com/media/ExCuNbDXAAQaPiL.jpg | 1374913509662806016 | I've just watched episode S01 | E06 of Ted Las... | http://pbs.twimg.com/media/ExSsjRQWgAUVRPz.jpg | 矛盾 |
我们感兴趣的列如下所示
text_1
image_1
text_2
image_2
标签
蕴含任务表述如下
给定 (text_1
, image_1
) 和 (text_2
, image_2
) 对,它们之间是否存在蕴含(或无蕴含或矛盾)关系?
我们已经下载了图片。image_1
下载时文件名为 id1
,image_2
下载时文件名为 id2
。下一步,我们将在 df
中添加两列 - image_1
和 image_2
的文件路径。
images_one_paths = []
images_two_paths = []
for idx in range(len(df)):
current_row = df.iloc[idx]
id_1 = current_row["id_1"]
id_2 = current_row["id_2"]
extentsion_one = current_row["image_1"].split(".")[-1]
extentsion_two = current_row["image_2"].split(".")[-1]
image_one_path = os.path.join(image_base_path, str(id_1) + f".{extentsion_one}")
image_two_path = os.path.join(image_base_path, str(id_2) + f".{extentsion_two}")
images_one_paths.append(image_one_path)
images_two_paths.append(image_two_path)
df["image_1_path"] = images_one_paths
df["image_2_path"] = images_two_paths
# Create another column containing the integer ids of
# the string labels.
df["label_idx"] = df["label"].apply(lambda x: label_map[x])
def visualize(idx):
current_row = df.iloc[idx]
image_1 = plt.imread(current_row["image_1_path"])
image_2 = plt.imread(current_row["image_2_path"])
text_1 = current_row["text_1"]
text_2 = current_row["text_2"]
label = current_row["label"]
plt.subplot(1, 2, 1)
plt.imshow(image_1)
plt.axis("off")
plt.title("Image One")
plt.subplot(1, 2, 2)
plt.imshow(image_1)
plt.axis("off")
plt.title("Image Two")
plt.show()
print(f"Text one: {text_1}")
print(f"Text two: {text_2}")
print(f"Label: {label}")
random_idx = random.choice(range(len(df)))
visualize(random_idx)
random_idx = random.choice(range(len(df)))
visualize(random_idx)
Text one: World #water day reminds that we should follow the #guidelines to save water for us. This Day is an #opportunity to learn more about water related issues, be #inspired to tell others and take action to make a difference. Just remember, every #drop counts.
#WorldWaterDay2021 https://#/bQ9Hp53qUj
Text two: Water is an extremely precious resource without which life would be impossible. We need to ensure that water is used judiciously, this #WorldWaterDay, let us pledge to reduce water wastage and conserve it.
#WorldWaterDay2021 https://#/0KWnd8Kn8r
Label: NoEntailment
Text one: 🎧 𝗘𝗣𝗜𝗦𝗢𝗗𝗘 𝟯𝟬: 𝗗𝗬𝗟𝗔𝗡 𝗙𝗜𝗧𝗭𝗦𝗜𝗠𝗢𝗡𝗦
Dylan Fitzsimons is a young passionate greyhound supporter.
He and @Drakesport enjoy a great chat about everything greyhounds!
Listen: https://#/B2XgMp0yaO
#GoGreyhoundRacing #ThisRunsDeep #TalkingDogs https://#/crBiSqHUvp
Text two: 🎧 𝗘𝗣𝗜𝗦𝗢𝗗𝗘 𝟯𝟳: 𝗣𝗜𝗢 𝗕𝗔𝗥𝗥𝗬 🎧
Well known within greyhound circles, Pio Barry shares some wonderful greyhound racing stories with @Drakesport in this podcast episode.
A great chat.
Listen: https://#/mJTVlPHzp0
#TalkingDogs #GoGreyhoundRacing #ThisRunsDeep https://#/QbxtCpLcGm
Label: NoEntailment
数据集存在类别不平衡问题。我们可以在以下单元格中确认这一点。
df["label"].value_counts()
label
NoEntailment 819
Contradictory 92
Implies 89
Name: count, dtype: int64
考虑到这一点,我们将进行分层划分。
# 10% for test
train_df, test_df = train_test_split(
df, test_size=0.1, stratify=df["label"].values, random_state=42
)
# 5% for validation
train_df, val_df = train_test_split(
train_df, test_size=0.05, stratify=train_df["label"].values, random_state=42
)
print(f"Total training examples: {len(train_df)}")
print(f"Total validation examples: {len(val_df)}")
print(f"Total test examples: {len(test_df)}")
Total training examples: 855
Total validation examples: 45
Total test examples: 100
Keras Hub 提供了多种 BERT 系列模型。每个模型都附带相应的预处理层。您可以从此资源了解更多关于这些模型及其预处理层的信息。
为了让本示例的运行时间相对较短,我们将使用原始 BERT 模型的 base_unacased 变体。
使用 KerasHub 进行文本预处理
text_preprocessor = keras_hub.models.BertTextClassifierPreprocessor.from_preset(
"bert_base_en_uncased",
sequence_length=128,
)
idx = random.choice(range(len(train_df)))
row = train_df.iloc[idx]
sample_text_1, sample_text_2 = row["text_1"], row["text_2"]
print(f"Text 1: {sample_text_1}")
print(f"Text 2: {sample_text_2}")
test_text = [sample_text_1, sample_text_2]
text_preprocessed = text_preprocessor(test_text)
print("Keys : ", list(text_preprocessed.keys()))
print("Shape Token Ids : ", text_preprocessed["token_ids"].shape)
print("Token Ids : ", text_preprocessed["token_ids"][0, :16])
print(" Shape Padding Mask : ", text_preprocessed["padding_mask"].shape)
print("Padding Mask : ", text_preprocessed["padding_mask"][0, :16])
print("Shape Segment Ids : ", text_preprocessed["segment_ids"].shape)
print("Segment Ids : ", text_preprocessed["segment_ids"][0, :16])
An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.
Text 1: The RPF Lohardaga and Hatia Post of Ranchi Division have recovered 02 bags on 20.02.2021 at Station platform and in T/No.08310 Spl. respectively and handed over to their actual owner correctly. @RPF_INDIA https://#/bdEBl2egIc
Text 2: The RPF Lohardaga and Hatia Post of Ranchi Division have recovered 02 bags on 20.02.2021 at Station platform and in T/No.08310 (JAT-SBP) Spl. respectively and handed over to their actual owner correctly. @RPF_INDIA https://#/Q5l2AtA4uq
Keys : ['token_ids', 'padding_mask', 'segment_ids']
Shape Token Ids : (2, 128)
Token Ids : [ 101 1996 1054 14376 8840 11783 16098 1998 6045 2401 2695 1997
8086 2072 2407 2031]
Shape Padding Mask : (2, 128)
Padding Mask : [ True True True True True True True True True True True True
True True True True]
Shape Segment Ids : (2, 128)
Segment Ids : [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
现在我们将从数据帧创建 tf.data.Dataset
对象。
请注意,文本输入将作为数据输入管道的一部分进行预处理。但预处理模块也可以是其相应 BERT 模型的一部分。这有助于减少训练/服务偏差,并使我们的模型能够处理原始文本输入。按照此教程了解如何将预处理模块直接集成到模型中。
def dataframe_to_dataset(dataframe):
columns = ["image_1_path", "image_2_path", "text_1", "text_2", "label_idx"]
ds = UnifiedPyDataset(
dataframe,
batch_size=32,
workers=4,
)
return ds
bert_input_features = ["padding_mask", "segment_ids", "token_ids"]
def preprocess_text(text_1, text_2):
output = text_preprocessor([text_1, text_2])
output = {
feature: keras.ops.reshape(output[feature], [-1])
for feature in bert_input_features
}
return output
class UnifiedPyDataset(PyDataset):
"""A Keras-compatible dataset that processes a DataFrame for TensorFlow, JAX, and PyTorch."""
def __init__(
self,
df,
batch_size=32,
workers=4,
use_multiprocessing=False,
max_queue_size=10,
**kwargs,
):
"""
Args:
df: pandas DataFrame with data
batch_size: Batch size for dataset
workers: Number of workers to use for parallel loading (Keras)
use_multiprocessing: Whether to use multiprocessing
max_queue_size: Maximum size of the data queue for parallel loading
"""
super().__init__(**kwargs)
self.dataframe = df
columns = ["image_1_path", "image_2_path", "text_1", "text_2"]
# image files
self.image_x_1 = self.dataframe["image_1_path"]
self.image_x_2 = self.dataframe["image_1_path"]
self.image_y = self.dataframe["label_idx"]
# text files
self.text_x_1 = self.dataframe["text_1"]
self.text_x_2 = self.dataframe["text_2"]
self.text_y = self.dataframe["label_idx"]
# general
self.batch_size = batch_size
self.workers = workers
self.use_multiprocessing = use_multiprocessing
self.max_queue_size = max_queue_size
def __getitem__(self, index):
"""
Fetches a batch of data from the dataset at the given index.
"""
# Return x, y for batch idx.
low = index * self.batch_size
# Cap upper bound at array length; the last batch may be smaller
# if the total number of items is not a multiple of batch size.
high_image_1 = min(low + self.batch_size, len(self.image_x_1))
high_image_2 = min(low + self.batch_size, len(self.image_x_2))
high_text_1 = min(low + self.batch_size, len(self.text_x_1))
high_text_2 = min(low + self.batch_size, len(self.text_x_1))
# images files
batch_image_x_1 = self.image_x_1[low:high_image_1]
batch_image_y_1 = self.image_y[low:high_image_1]
batch_image_x_2 = self.image_x_2[low:high_image_2]
batch_image_y_2 = self.image_y[low:high_image_2]
# text files
batch_text_x_1 = self.text_x_1[low:high_text_1]
batch_text_y_1 = self.text_y[low:high_text_1]
batch_text_x_2 = self.text_x_2[low:high_text_2]
batch_text_y_2 = self.text_y[low:high_text_2]
# image number 1 inputs
image_1 = [
resize(imread(file_name), (128, 128)) for file_name in batch_image_x_1
]
image_1 = [
( # exeperienced some shapes which were different from others.
np.array(Image.fromarray((img.astype(np.uint8))).convert("RGB"))
if img.shape[2] == 4
else img
)
for img in image_1
]
image_1 = np.array(image_1)
# Both text inputs to the model, return a dict for inputs to BertBackbone
text = {
key: np.array(
[
d[key]
for d in [
preprocess_text(file_path1, file_path2)
for file_path1, file_path2 in zip(
batch_text_x_1, batch_text_x_2
)
]
]
)
for key in ["padding_mask", "token_ids", "segment_ids"]
}
# Image number 2 model inputs
image_2 = [
resize(imread(file_name), (128, 128)) for file_name in batch_image_x_2
]
image_2 = [
( # exeperienced some shapes which were different from others
np.array(Image.fromarray((img.astype(np.uint8))).convert("RGB"))
if img.shape[2] == 4
else img
)
for img in image_2
]
# Stack the list comprehension to an nd.array
image_2 = np.array(image_2)
return (
{
"image_1": image_1,
"image_2": image_2,
"padding_mask": text["padding_mask"],
"segment_ids": text["segment_ids"],
"token_ids": text["token_ids"],
},
# Target lables
np.array(batch_image_y_1),
)
def __len__(self):
"""
Returns the number of batches in the dataset.
"""
return math.ceil(len(self.dataframe) / self.batch_size)
创建训练集、验证集和测试集
def prepare_dataset(dataframe):
ds = dataframe_to_dataset(dataframe)
return ds
train_ds = prepare_dataset(train_df)
validation_ds = prepare_dataset(val_df)
test_ds = prepare_dataset(test_df)
我们的最终模型将接受两张图片及其对应的文本。图片将直接输入模型,而文本输入将首先进行预处理,然后输入模型。下面是这种方法的视觉说明
模型包含以下元素
提取个体嵌入后,它们将被投影到相同的空间中。最后,它们的投影将被连接起来并输入最终的分类层。
这是一个涉及以下类别的多类别分类问题
project_embeddings()
、create_vision_encoder()
和 create_text_encoder()
工具函数参考了此示例。
投影工具
def project_embeddings(
embeddings, num_projection_layers, projection_dims, dropout_rate
):
projected_embeddings = keras.layers.Dense(units=projection_dims)(embeddings)
for _ in range(num_projection_layers):
x = keras.ops.nn.gelu(projected_embeddings)
x = keras.layers.Dense(projection_dims)(x)
x = keras.layers.Dropout(dropout_rate)(x)
x = keras.layers.Add()([projected_embeddings, x])
projected_embeddings = keras.layers.LayerNormalization()(x)
return projected_embeddings
视觉编码器工具
def create_vision_encoder(
num_projection_layers, projection_dims, dropout_rate, trainable=False
):
# Load the pre-trained ResNet50V2 model to be used as the base encoder.
resnet_v2 = keras.applications.ResNet50V2(
include_top=False, weights="imagenet", pooling="avg"
)
# Set the trainability of the base encoder.
for layer in resnet_v2.layers:
layer.trainable = trainable
# Receive the images as inputs.
image_1 = keras.Input(shape=(128, 128, 3), name="image_1")
image_2 = keras.Input(shape=(128, 128, 3), name="image_2")
# Preprocess the input image.
preprocessed_1 = keras.applications.resnet_v2.preprocess_input(image_1)
preprocessed_2 = keras.applications.resnet_v2.preprocess_input(image_2)
# Generate the embeddings for the images using the resnet_v2 model
# concatenate them.
embeddings_1 = resnet_v2(preprocessed_1)
embeddings_2 = resnet_v2(preprocessed_2)
embeddings = keras.layers.Concatenate()([embeddings_1, embeddings_2])
# Project the embeddings produced by the model.
outputs = project_embeddings(
embeddings, num_projection_layers, projection_dims, dropout_rate
)
# Create the vision encoder model.
return keras.Model([image_1, image_2], outputs, name="vision_encoder")
文本编码器工具
def create_text_encoder(
num_projection_layers, projection_dims, dropout_rate, trainable=False
):
# Load the pre-trained BERT BackBone using KerasHub.
bert = keras_hub.models.BertBackbone.from_preset(
"bert_base_en_uncased", num_classes=3
)
# Set the trainability of the base encoder.
bert.trainable = trainable
# Receive the text as inputs.
bert_input_features = ["padding_mask", "segment_ids", "token_ids"]
inputs = {
feature: keras.Input(shape=(256,), dtype="int32", name=feature)
for feature in bert_input_features
}
# Generate embeddings for the preprocessed text using the BERT model.
embeddings = bert(inputs)["pooled_output"]
# Project the embeddings produced by the model.
outputs = project_embeddings(
embeddings, num_projection_layers, projection_dims, dropout_rate
)
# Create the text encoder model.
return keras.Model(inputs, outputs, name="text_encoder")
多模态模型工具
def create_multimodal_model(
num_projection_layers=1,
projection_dims=256,
dropout_rate=0.1,
vision_trainable=False,
text_trainable=False,
):
# Receive the images as inputs.
image_1 = keras.Input(shape=(128, 128, 3), name="image_1")
image_2 = keras.Input(shape=(128, 128, 3), name="image_2")
# Receive the text as inputs.
bert_input_features = ["padding_mask", "segment_ids", "token_ids"]
text_inputs = {
feature: keras.Input(shape=(256,), dtype="int32", name=feature)
for feature in bert_input_features
}
text_inputs = list(text_inputs.values())
# Create the encoders.
vision_encoder = create_vision_encoder(
num_projection_layers, projection_dims, dropout_rate, vision_trainable
)
text_encoder = create_text_encoder(
num_projection_layers, projection_dims, dropout_rate, text_trainable
)
# Fetch the embedding projections.
vision_projections = vision_encoder([image_1, image_2])
text_projections = text_encoder(text_inputs)
# Concatenate the projections and pass through the classification layer.
concatenated = keras.layers.Concatenate()([vision_projections, text_projections])
outputs = keras.layers.Dense(3, activation="softmax")(concatenated)
return keras.Model([image_1, image_2, *text_inputs], outputs)
multimodal_model = create_multimodal_model()
keras.utils.plot_model(multimodal_model, show_shapes=True)
您也可以通过将 plot_model()
的 expand_nested
参数设置为 True
来检查各个编码器的结构。建议您尝试使用构建此模型时涉及的不同超参数,并观察最终性能受到的影响。
multimodal_model.compile(
optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]
)
history = multimodal_model.fit(train_ds, validation_data=validation_ds, epochs=1)
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
1/27 [37m━━━━━━━━━━━━━━━━━━━━ 45:45 106s/step - accuracy: 0.0625 - loss: 1.6335
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
2/27 ━ [37m━━━━━━━━━━━━━━━━━━━ 42:14 101s/step - accuracy: 0.2422 - loss: 1.9508
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
3/27 ━━ [37m━━━━━━━━━━━━━━━━━━ 38:49 97s/step - accuracy: 0.3524 - loss: 2.0126
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
4/27 ━━ [37m━━━━━━━━━━━━━━━━━━ 37:09 97s/step - accuracy: 0.4284 - loss: 1.9870
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
5/27 ━━━ [37m━━━━━━━━━━━━━━━━━ 35:08 96s/step - accuracy: 0.4815 - loss: 1.9855
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
6/27 ━━━━ [37m━━━━━━━━━━━━━━━━ 31:56 91s/step - accuracy: 0.5210 - loss: 1.9939
7/27 ━━━━━ [37m━━━━━━━━━━━━━━━ 29:30 89s/step - accuracy: 0.5512 - loss: 1.9980
8/27 ━━━━━ [37m━━━━━━━━━━━━━━━ 27:12 86s/step - accuracy: 0.5750 - loss: 2.0061
9/27 ━━━━━━ [37m━━━━━━━━━━━━━━ 25:15 84s/step - accuracy: 0.5956 - loss: 1.9959
10/27 ━━━━━━━ [37m━━━━━━━━━━━━━ 23:33 83s/step - accuracy: 0.6120 - loss: 1.9738
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
11/27 ━━━━━━━━ [37m━━━━━━━━━━━━ 22:09 83s/step - accuracy: 0.6251 - loss: 1.9579
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
12/27 ━━━━━━━━ [37m━━━━━━━━━━━━ 20:59 84s/step - accuracy: 0.6357 - loss: 1.9524
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
13/27 ━━━━━━━━━ [37m━━━━━━━━━━━ 19:44 85s/step - accuracy: 0.6454 - loss: 1.9439
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
14/27 ━━━━━━━━━━ [37m━━━━━━━━━━ 18:22 85s/step - accuracy: 0.6540 - loss: 1.9346
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(23, 256))', 'Tensor(shape=(23, 256))', 'Tensor(shape=(23, 256))']
warnings.warn(msg)
15/27 ━━━━━━━━━━━ [37m━━━━━━━━━ 16:52 84s/step - accuracy: 0.6621 - loss: 1.9213
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
16/27 ━━━━━━━━━━━ [37m━━━━━━━━━ 15:29 85s/step - accuracy: 0.6693 - loss: 1.9101
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
17/27 ━━━━━━━━━━━━ [37m━━━━━━━━ 14:08 85s/step - accuracy: 0.6758 - loss: 1.9021
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
18/27 ━━━━━━━━━━━━━ [37m━━━━━━━ 12:45 85s/step - accuracy: 0.6819 - loss: 1.8916
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
19/27 ━━━━━━━━━━━━━━ [37m━━━━━━ 11:24 86s/step - accuracy: 0.6874 - loss: 1.8851
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
20/27 ━━━━━━━━━━━━━━ [37m━━━━━━ 10:00 86s/step - accuracy: 0.6925 - loss: 1.8791
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
21/27 ━━━━━━━━━━━━━━━ [37m━━━━━ 8:36 86s/step - accuracy: 0.6976 - loss: 1.8699
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
22/27 ━━━━━━━━━━━━━━━━ [37m━━━━ 7:11 86s/step - accuracy: 0.7020 - loss: 1.8623
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
23/27 ━━━━━━━━━━━━━━━━━ [37m━━━ 5:46 87s/step - accuracy: 0.7061 - loss: 1.8573
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
24/27 ━━━━━━━━━━━━━━━━━ [37m━━━ 4:20 87s/step - accuracy: 0.7100 - loss: 1.8534
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
25/27 ━━━━━━━━━━━━━━━━━━ [37m━━ 2:54 87s/step - accuracy: 0.7136 - loss: 1.8494
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
26/27 ━━━━━━━━━━━━━━━━━━━ [37m━ 1:27 87s/step - accuracy: 0.7170 - loss: 1.8449
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
27/27 ━━━━━━━━━━━━━━━━━━━━ 0s 88s/step - accuracy: 0.7201 - loss: 1.8414
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/PIL/Image.py:1054: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/PIL/Image.py:1054: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(13, 256))', 'Tensor(shape=(13, 256))', 'Tensor(shape=(13, 256))']
warnings.warn(msg)
27/27 ━━━━━━━━━━━━━━━━━━━━ 2508s 92s/step - accuracy: 0.7231 - loss: 1.8382 - val_accuracy: 0.8222 - val_loss: 1.7304
_, acc = multimodal_model.evaluate(test_ds)
print(f"Accuracy on the test set: {round(acc * 100, 2)}%.")
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/PIL/Image.py:1054: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/PIL/Image.py:1054: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
warnings.warn(
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))', 'Tensor(shape=(32, 256))']
warnings.warn(msg)
1/4 ━━━━━ [37m━━━━━━━━━━━━━━━ 5:32 111s/step - accuracy: 0.7812 - loss: 1.9384
2/4 ━━━━━━━━━━ [37m━━━━━━━━━━ 2:10 65s/step - accuracy: 0.7969 - loss: 1.8931
3/4 ━━━━━━━━━━━━━━━ [37m━━━━━ 1:05 65s/step - accuracy: 0.8056 - loss: 1.8200
/home/humbulani/tensorflow-env/env/lib/python3.11/site-packages/keras/src/models/functional.py:248: UserWarning: The structure of `inputs` doesn't match the expected structure.
Expected: {'padding_mask': 'padding_mask', 'segment_ids': 'segment_ids', 'token_ids': 'token_ids'}
Received: inputs=['Tensor(shape=(4, 256))', 'Tensor(shape=(4, 256))', 'Tensor(shape=(4, 256))']
warnings.warn(msg)
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 49s/step - accuracy: 0.8092 - loss: 1.8075
4/4 ━━━━━━━━━━━━━━━━━━━━ 256s 49s/step - accuracy: 0.8113 - loss: 1.8000
Accuracy on the test set: 82.0%.
加入正则化:
训练日志表明模型开始过拟合,可能受益于正则化。Dropout(Srivastava 等人)是一种简单而强大的正则化技术,我们可以在模型中使用。但我们应该如何在这里应用它呢?
我们总可以在模型的不同层之间引入 Dropout (keras.layers.Dropout
)。但这里还有另一种方法。我们的模型期望接收来自两种不同数据模态的输入。如果在推理过程中其中一种模态不存在怎么办?为了解决这个问题,我们可以在个体投影连接之前引入 Dropout
vision_projections = keras.layers.Dropout(rate)(vision_projections)
text_projections = keras.layers.Dropout(rate)(text_projections)
concatenated = keras.layers.Concatenate()([vision_projections, text_projections])
关注重要部分:
图片的所有部分都与其对应的文本同样重要吗?情况可能并非如此。为了让我们的模型只关注与文本对应部分关系最密切的图片重要部分,我们可以使用“交叉注意力”
# Embeddings.
vision_projections = vision_encoder([image_1, image_2])
text_projections = text_encoder(text_inputs)
# Cross-attention (Luong-style).
query_value_attention_seq = keras.layers.Attention(use_scale=True, dropout=0.2)(
[vision_projections, text_projections]
)
# Concatenate.
concatenated = keras.layers.Concatenate()([vision_projections, text_projections])
contextual = keras.layers.Concatenate()([concatenated, query_value_attention_seq])
要查看实际应用,请参考此 notebook。
处理类别不平衡:
数据集存在类别不平衡问题。对上述模型的混淆矩阵进行分析表明,它在少数类别上表现不佳。如果使用了加权损失,训练会更有针对性。您可以查看此 notebook,它在模型训练期间考虑了类别不平衡问题。
仅使用文本输入:
另外,如果我们在蕴含任务中只使用了文本输入会怎样?由于社交媒体平台上遇到的文本输入的性质,仅使用文本输入会损害最终性能。在类似的训练设置下,仅使用文本输入,在相同的测试集上我们获得了 67.14% 的 top-1 准确率。有关详细信息,请参阅此 notebook。
最后,这里是一个比较蕴含任务不同方法的表格
类型 | 标准 交叉熵 |
损失加权 交叉熵 |
Focal Loss |
---|---|---|---|
多模态 | 77.86% | 67.86% | 86.43% |
仅文本 | 67.14% | 11.43% | 37.86% |
您可以查看此仓库,了解更多关于如何进行实验以获得这些数字的信息。
您可以使用托管在Hugging Face Hub上的训练好的模型,并在Hugging Face Spaces上试用演示。