Transformer-based Language Model - GPT2

Contents

Transformer-based Language Model - GPT2#

This notebook runs on Google Colab.
Codes from A Comprehensive Guide to Build Your Own Language Model in Python
Use the OpenAI GPT-2 language model (based on Transformers) to:
- Generate text sequences based on seed texts
- Convert text sequences into numerical representations

!pip install transformers

# Import required libraries
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load pre-trained model tokenizer (vocabulary)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# Encode a text inputs
text = "What is the fastest car in the"
indexed_tokens = tokenizer.encode(text)

# Convert indexed tokens in a PyTorch tensor
tokens_tensor = torch.tensor([indexed_tokens])

# Load pre-trained model (weights)
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Set the model in evaluation mode to deactivate the DropOut modules
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor = tokens_tensor.to('cuda')
model.to('cuda')

# Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor)
    predictions = outputs[0]

# Get the predicted next sub-word
predicted_index = torch.argmax(predictions[0, -1, :]).item()
predicted_text = tokenizer.decode(indexed_tokens + [predicted_index])

# Print the predicted word
print(predicted_text)

!git clone https://github.com/huggingface/transformers.git

!ls transformers/examples

Text Generation Using DPT2#

Write with Transformer

# !python transformers/examples/text-generation/run_generation.py \
#     --model_type=gpt2 \
#     --model_name_or_path=gpt2 \
#     --length=100

Text Generation Using GPT2#

from transformers import pipeline, set_seed
generator = pipeline('text-generation', model='gpt2')
set_seed(42)
generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)

generator("Once upon a time, ", max_length=30, num_return_sequences=5)

Transforming Texts into Features#

# from transformers import GPT2Tokenizer, GPT2Model
# tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
# model = GPT2Model.from_pretrained('gpt2')
# text = "Replace me by any text you'd like."
# encoded_input = tokenizer(text, return_tensors='pt') # return tensorflow tensors
# output = model(encoded_input)


from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
print(encoded_input)