用於摘要、情感分類和翻譯的 T5-Base 模型

**作者：** Pendo Abbo、Joe Cummings

概觀

本教學課程示範如何使用預先訓練的 T5 模型來執行摘要、情感分類和翻譯任務。我們將示範如何使用 torchtext 函式庫來

1. 為 T5 模型建立文字預處理流程

2. 使用基本設定實例化預先訓練的 T5 模型

3. 讀取 CNNDM、IMDB 和 Multi30k 資料集，並預先處理其文字以準備模型

4. 執行文字摘要、情感分類和翻譯

資料轉換

T5 模型無法處理原始文字。相反地，它需要將文字轉換為數值形式，才能執行訓練和推論。T5 模型需要以下轉換

1. 文字標記化

2. 將標記轉換為（整數）ID

3. 將序列截斷為指定的長度上限

4. 新增序列結束（EOS）和填補標記 ID

T5 使用 SentencePiece 模型進行文字標記化。以下，我們使用預先訓練的 SentencePiece 模型，使用 torchtext 的 T5Transform 來建立文字預處理流程。請注意，轉換支援批次和非批次文字輸入（例如，可以傳遞單一句子或句子清單），但是 T5 模型預期輸入是批次的。

from torchtext.models import T5Transform

padding_idx = 0
eos_idx = 1
max_seq_len = 512
t5_sp_model_path = "https://download.pytorch.org/models/text/t5_tokenizer_base.model"

transform = T5Transform(
    sp_model_path=t5_sp_model_path,
    max_seq_len=max_seq_len,
    eos_idx=eos_idx,
    padding_idx=padding_idx,
)

或者，我們也可以使用預先訓練模型附帶的轉換，它可以立即完成上述所有操作

from torchtext.models import T5_BASE_GENERATION
transform = T5_BASE_GENERATION.transform()

模型準備

torchtext 提供了可以直接用於 NLP 任務或在下游任務上進行微調的 SOTA 預先訓練模型。以下我們使用具有標準基本設定的預先訓練 T5 模型來執行文字摘要、情感分類和翻譯。有關可用預先訓練模型的其他詳細資訊，請參閱torchtext 文件

from torchtext.models import T5_BASE_GENERATION


t5_base = T5_BASE_GENERATION
transform = t5_base.transform()
model = t5_base.get_model()
model.eval()

GenerationUtils

我們可以使用 torchtext 的 GenerationUtils根據提供的輸入序列產生輸出序列。這會呼叫模型的編碼器和解碼器，並迭代地展開解碼序列，直到為批次中的所有序列產生序列結束標記。下面顯示的 generate方法使用貪婪搜尋來產生序列。也支援 Beam 搜尋和其他解碼策略。

from torchtext.prototype.generate import GenerationUtils

sequence_generator = GenerationUtils(model)

資料集

torchtext 提供了幾個標準 NLP 資料集。如需完整清單，請參閱https://pytorch.dev.org.tw/text/stable/datasets.html的說明文件。這些資料集是使用可組合的 torchdata 資料管道建立的，因此支援使用使用者定義函數和轉換進行標準流程控制和映射/轉換。

以下我們示範如何預先處理 CNNDM 資料集，以包含模型識別其正在執行的任務所需的前綴。CNNDM 資料集具有訓練、驗證和測試分割。以下我們在測試分割上進行示範。

T5 模型使用前綴「summarize」進行文字摘要。如需任務前綴的詳細資訊，請造訪T5 論文的附錄 D

備註

使用資料管道目前仍有一些需要注意的地方。如果您想擴展此範例以包含 shuffling、多重處理或分散式學習，請參閱此說明以取得進一步指示。

from functools import partial

from torch.utils.data import DataLoader
from torchtext.datasets import CNNDM

cnndm_batch_size = 5
cnndm_datapipe = CNNDM(split="test")
task = "summarize"


def apply_prefix(task, x):
    return f"{task}: " + x[0], x[1]


cnndm_datapipe = cnndm_datapipe.map(partial(apply_prefix, task))
cnndm_datapipe = cnndm_datapipe.batch(cnndm_batch_size)
cnndm_datapipe = cnndm_datapipe.rows2columnar(["article", "abstract"])
cnndm_dataloader = DataLoader(cnndm_datapipe, shuffle=True, batch_size=None)

或者，我們也可以使用批次 API，例如，將前綴應用於整個批次

def batch_prefix(task, x):
 return {
     "article": [f'{task}: ' + y for y in x["article"]],
     "abstract": x["abstract"]
 }

cnndm_batch_size = 5
cnndm_datapipe = CNNDM(split="test")
task = 'summarize'

cnndm_datapipe = cnndm_datapipe.batch(cnndm_batch_size).rows2columnar(["article", "abstract"])
cnndm_datapipe = cnndm_datapipe.map(partial(batch_prefix, task))
cnndm_dataloader = DataLoader(cnndm_datapipe, batch_size=None)

我們也可以載入 IMDB 資料集，這將用於示範使用 T5 模型進行情感分類。此資料集具有訓練和測試分割。以下我們在測試分割上進行示範。

T5 模型使用前綴「sst2 sentence」在 SST2 資料集（torchtext 中也有提供）上進行情感分類訓練。因此，我們將使用此前綴對 IMDB 資料集執行情感分類。

from torchtext.datasets import IMDB

imdb_batch_size = 3
imdb_datapipe = IMDB(split="test")
task = "sst2 sentence"
labels = {"1": "negative", "2": "positive"}


def process_labels(labels, x):
    return x[1], labels[str(x[0])]


imdb_datapipe = imdb_datapipe.map(partial(process_labels, labels))
imdb_datapipe = imdb_datapipe.map(partial(apply_prefix, task))
imdb_datapipe = imdb_datapipe.batch(imdb_batch_size)
imdb_datapipe = imdb_datapipe.rows2columnar(["text", "label"])
imdb_dataloader = DataLoader(imdb_datapipe, batch_size=None)

最後，我們也可以載入 Multi30k 資料集，以示範使用 T5 模型將英文翻譯成德文。此資料集具有訓練、驗證和測試分割。以下我們在測試分割上進行示範。

T5 模型使用前綴「translate English to German」執行此任務。

from torchtext.datasets import Multi30k

multi_batch_size = 5
language_pair = ("en", "de")
multi_datapipe = Multi30k(split="test", language_pair=language_pair)
task = "translate English to German"

multi_datapipe = multi_datapipe.map(partial(apply_prefix, task))
multi_datapipe = multi_datapipe.batch(multi_batch_size)
multi_datapipe = multi_datapipe.rows2columnar(["english", "german"])
multi_dataloader = DataLoader(multi_datapipe, batch_size=None)

產生摘要

我們可以將所有元件組合在一起，使用 Beam 大小為 1 來產生 CNNDM 測試集中第一批文章的摘要。

batch = next(iter(cnndm_dataloader))
input_text = batch["article"]
target = batch["abstract"]
beam_size = 1

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(cnndm_batch_size):
    print(f"Example {i+1}:\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

摘要輸出（可能會有所不同，因為我們會打亂資料載入器的順序）

Example 1:

prediction: the 24-year-old has been tattooed for over a decade . he has landed in australia
to start work on a new campaign . he says he is 'taking it in your stride' to be honest .

target: London-based model Stephen James Hendry famed for his full body tattoo . The supermodel
is in Sydney for a new modelling campaign . Australian fans understood to have already located
him at his hotel . The 24-year-old heartthrob is recently single .


Example 2:

prediction: a stray pooch has used up at least three of her own after being hit by a
car and buried in a field . the dog managed to stagger to a nearby farm, dirt-covered
and emaciated, where she was found . she suffered a dislocated jaw, leg injuries and a
caved-in sinus cavity -- and still requires surgery to help her breathe .

target: Theia, a bully breed mix, was apparently hit by a car, whacked with a hammer
and buried in a field . "She's a true miracle dog and she deserves a good life," says
Sara Mellado, who is looking for a home for Theia .


Example 3:

prediction: mohammad Javad Zarif arrived in Iran on a sunny friday morning . he has gone
a long way to bring Iran in from the cold and allow it to rejoin the international
community . but there are some facts about him that are less well-known .

target: Mohammad Javad Zarif has spent more time with John Kerry than any other
foreign minister . He once participated in a takeover of the Iranian Consulate in San
Francisco . The Iranian foreign minister tweets in English .


Example 4:

prediction: five americans were monitored for three weeks after being exposed to Ebola in
west africa . one of the five had a heart-related issue and has been discharged but hasn't
left the area . they are clinicians for Partners in Health, a Boston-based aid group .

target: 17 Americans were exposed to the Ebola virus while in Sierra Leone in March .
Another person was diagnosed with the disease and taken to hospital in Maryland .
National Institutes of Health says the patient is in fair condition after weeks of
treatment .


Example 5:

prediction: the student was identified during an investigation by campus police and
the office of student affairs . he admitted to placing the noose on the tree early
Wednesday morning . the incident is one of several recent racist events to affect
college students .

target: Student is no longer on Duke University campus and will face disciplinary
review . School officials identified student during investigation and the person
admitted to hanging the noose, Duke says . The noose, made of rope, was discovered on
campus about 2 a.m.

產生情感分類

同樣地，我們可以使用模型在 IMDB 測試集中第一批評論上產生情感分類，Beam 大小為 1。

batch = next(iter(imdb_dataloader))
input_text = batch["text"]
target = batch["label"]
beam_size = 1

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(imdb_batch_size):
    print(f"Example {i+1}:\n")
    print(f"input_text: {input_text[i]}\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

情感輸出

Example 1:

input_text: sst2 sentence: I love sci-fi and am willing to put up with a lot. Sci-fi
movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like
this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original).
Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the
background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi'
setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV.
It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character
development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may
treat important issues, yet not as a serious philosophy. It's really difficult to care about
the characters here as they are not simply foolish, just missing a spark of life. Their
actions and reactions are wooden and predictable, often painful to watch. The makers of Earth
KNOW it's rubbish as they have to always say "Gene Roddenberry's Earth..." otherwise people
would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull,
cheap, poorly edited (watching it without advert breaks really brings this home) trudging
Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring
him back as another actor. Jeeez. Dallas all over again.

prediction: negative

target: negative


Example 2:

input_text: sst2 sentence: Worth the entertainment value of a rental, especially if you like
action movies. This one features the usual car chases, fights with the great Van Damme kick
style, shooting battles with the 40 shell load shotgun, and even terrorist style bombs. All
of this is entertaining and competently handled but there is nothing that really blows you
away if you've seen your share before.<br /><br />The plot is made interesting by the
inclusion of a rabbit, which is clever but hardly profound. Many of the characters are
heavily stereotyped -- the angry veterans, the terrified illegal aliens, the crooked cops,
the indifferent feds, the bitchy tough lady station head, the crooked politician, the fat
federale who looks like he was typecast as the Mexican in a Hollywood movie from the 1940s.
All passably acted but again nothing special.<br /><br />I thought the main villains were
pretty well done and fairly well acted. By the end of the movie you certainly knew who the
good guys were and weren't. There was an emotional lift as the really bad ones got their just
deserts. Very simplistic, but then you weren't expecting Hamlet, right? The only thing I found
really annoying was the constant cuts to VDs daughter during the last fight scene.<br /><br />
Not bad. Not good. Passable 4.

prediction: positive

target: negative


Example 3:

input_text: sst2 sentence: its a totally average film with a few semi-alright action sequences
that make the plot seem a little better and remind the viewer of the classic van dam films.
parts of the plot don't make sense and seem to be added in to use up time. the end plot is that
of a very basic type that doesn't leave the viewer guessing and any twists are obvious from the
beginning. the end scene with the flask backs don't make sense as they are added in and seem to
have little relevance to the history of van dam's character. not really worth watching again,
bit disappointed in the end production, even though it is apparent it was shot on a low budget
certain shots and sections in the film are of poor directed quality.

prediction: negative

target: negative

產生翻譯

最後，我們也可以使用模型在 Multi30k 測試集中第一批範例上產生英文到德文的翻譯。

batch = next(iter(multi_dataloader))
input_text = batch["english"]
target = batch["german"]

model_input = transform(input_text)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())

for i in range(multi_batch_size):
    print(f"Example {i+1}:\n")
    print(f"input_text: {input_text[i]}\n")
    print(f"prediction: {output_text[i]}\n")
    print(f"target: {target[i]}\n\n")

翻譯輸出

Example 1:

input_text: translate English to German: A man in an orange hat starring at something.

prediction: Ein Mann in einem orangen Hut, der an etwas schaut.

target: Ein Mann mit einem orangefarbenen Hut, der etwas anstarrt.


Example 2:

input_text: translate English to German: A Boston Terrier is running on lush green grass in front of a white fence.

prediction: Ein Boston Terrier läuft auf üppigem grünem Gras vor einem weißen Zaun.

target: Ein Boston Terrier läuft über saftig-grünes Gras vor einem weißen Zaun.


Example 3:

input_text: translate English to German: A girl in karate uniform breaking a stick with a front kick.

prediction: Ein Mädchen in Karate-Uniform bricht einen Stöck mit einem Frontkick.

target: Ein Mädchen in einem Karateanzug bricht ein Brett mit einem Tritt.


Example 4:

input_text: translate English to German: Five people wearing winter jackets and helmets stand in the snow, with snowmobiles in the background.

prediction: Fünf Menschen mit Winterjacken und Helmen stehen im Schnee, mit Schneemobilen im Hintergrund.

target: Fünf Leute in Winterjacken und mit Helmen stehen im Schnee mit Schneemobilen im Hintergrund.


Example 5:

input_text: translate English to German: People are fixing the roof of a house.

prediction: Die Leute fixieren das Dach eines Hauses.

target: Leute Reparieren das Dach eines Hauses.