Neural Language Model: A Start#

  • In this tutorial, we will look at a naive example of neural language model.

  • Given a corpus, we can build a neural language model, which will learn to predict the next word given a specified limited context.

  • Depending on the size of the limited context, we can implement different types of neural language model:

    • Bigram-based neural language model: The model uses one preceding word for the next-word prediction.

    • Trigram-based neural language model: The model uses two preceding words for the next-word prediction.

    • Line-based neural language model: The model uses all the existing fore-going words in the “sequence” for the next-word prediction.

    • Discourse-based neural language model: The model uses inter-sentential information for next-word prediction (e.g., BERT).

  • This tutorial will demonstarte how to build a bigram-based language model.

  • In the Assignments, you need to extend the same rationale to other types of language models.

Workflow of Neural Language Model#

Bigram Model#

  • A bigram-based language model assumes that the next word (to be predicted) depends only on one preceding word.

## Dependencies
import numpy as np
import tensorflow.keras as keras
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM, Embedding

Tokenization#

  • A quick reminder of important parameters for Tokenzier():

    • num_words: the maximum number of words to keep, based on word frequency. Only the most common num_words-1 words will be kept.

    • filters: a string where each element is a character that will be filtered from the texts. The default includes all punctuations, plus tabs and line breaks (except for the ' character).

    • lower: boolean. Whether to convert the texts to lowercase.

    • split: string. Separator for word splitting.

    • char_level: if True, every character will be treated as a token.

    • oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls

# source text
data = """ Jack and Jill went up the hill\n
		To fetch a pail of water\n
		Jack fell down and broke his crown\n
		And Jill came tumbling after\n """

data = [l.strip() for l in data.split('\n') if l != ""]

# integer encode text
tokenizer = Tokenizer()
tokenizer.fit_on_texts(data)

# now the data consists of a sequence of word index integers
encoded = tokenizer.texts_to_sequences(data)

# determine the vocabulary size
vocab_size = len(tokenizer.word_index) + 1
print('Vocabulary Size: %d' % vocab_size)
print(tokenizer.word_index)
Vocabulary Size: 22
{'and': 1, 'jack': 2, 'jill': 3, 'went': 4, 'up': 5, 'the': 6, 'hill': 7, 'to': 8, 'fetch': 9, 'a': 10, 'pail': 11, 'of': 12, 'water': 13, 'fell': 14, 'down': 15, 'broke': 16, 'his': 17, 'crown': 18, 'came': 19, 'tumbling': 20, 'after': 21}

Text-to-Sequences and Training-Testing Sets#

  • Principles for bigrams extraction

    • When we create bigrams as the input sequences for network training, we need to make sure that we do not include unmeaningful bigrams, such as bigrams spanning the text boundaries, or sentence boundaries.

# create bigrams sequences

## bigrams holder
sequences = list()


## Extract bigrams from each text
for e in encoded:
    for i in range(1, len(e)):
        sequence = e[i - 1:i + 1]
        sequences.append(sequence)
print('Total Sequences: %d' % len(sequences))
sequences = np.array(sequences)
Total Sequences: 21
sequences[:5]
array([[2, 1],
       [1, 3],
       [3, 4],
       [4, 5],
       [5, 6]])
  • A sequence contains both our input and also output of the network.

  • That is, for bigram-based LM, the first word is the input X and the second word is the expected output y.

# split into X and y elements
X, y = sequences[:, 0], sequences[:, 1]
print(sequences[:5])
print(X[:5])
print(y[:5])
[[2 1]
 [1 3]
 [3 4]
 [4 5]
 [5 6]]
[2 1 3 4 5]
[1 3 4 5 6]

One-hot Representation of the Next-Word#

  • Because the neural language model is going to be a multi-class classifier (for word prediction), we need to convert our y into one-hot encoding.

# one hot encode outputs
y = to_categorical(y, num_classes=vocab_size)
y.shape
(21, 22)
print(y[:5])
[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

Define RNN Language Model#

# define model
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=10, input_length=1))
model.add(LSTM(50))  # LSTM Complexity
model.add(Dense(vocab_size, activation='softmax'))
print(model.summary())
Hide code cell output
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 1, 10)             220       
_________________________________________________________________
lstm (LSTM)                  (None, 50)                12200     
_________________________________________________________________
dense (Dense)                (None, 22)                1122      
=================================================================
Total params: 13,542
Trainable params: 13,542
Non-trainable params: 0
_________________________________________________________________
None
# compile network
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
# fit network
model.fit(X, y, epochs=500, verbose=2)
Hide code cell output
Epoch 1/500
1/1 - 2s - loss: 3.0898 - accuracy: 0.2381
Epoch 2/500
1/1 - 0s - loss: 3.0889 - accuracy: 0.2381
Epoch 3/500
1/1 - 0s - loss: 3.0881 - accuracy: 0.2381
Epoch 4/500
1/1 - 0s - loss: 3.0872 - accuracy: 0.2381
Epoch 5/500
1/1 - 0s - loss: 3.0863 - accuracy: 0.2857
Epoch 6/500
1/1 - 0s - loss: 3.0855 - accuracy: 0.2857
Epoch 7/500
1/1 - 0s - loss: 3.0846 - accuracy: 0.2857
Epoch 8/500
1/1 - 0s - loss: 3.0837 - accuracy: 0.3333
Epoch 9/500
1/1 - 0s - loss: 3.0828 - accuracy: 0.3333
Epoch 10/500
1/1 - 0s - loss: 3.0819 - accuracy: 0.3333
Epoch 11/500
1/1 - 0s - loss: 3.0810 - accuracy: 0.3333
Epoch 12/500
1/1 - 0s - loss: 3.0800 - accuracy: 0.3333
Epoch 13/500
1/1 - 0s - loss: 3.0791 - accuracy: 0.3333
Epoch 14/500
1/1 - 0s - loss: 3.0781 - accuracy: 0.3333
Epoch 15/500
1/1 - 0s - loss: 3.0771 - accuracy: 0.3333
Epoch 16/500
1/1 - 0s - loss: 3.0760 - accuracy: 0.3333
Epoch 17/500
1/1 - 0s - loss: 3.0750 - accuracy: 0.3333
Epoch 18/500
1/1 - 0s - loss: 3.0739 - accuracy: 0.3333
Epoch 19/500
1/1 - 0s - loss: 3.0728 - accuracy: 0.3333
Epoch 20/500
1/1 - 0s - loss: 3.0717 - accuracy: 0.3333
Epoch 21/500
1/1 - 0s - loss: 3.0706 - accuracy: 0.3333
Epoch 22/500
1/1 - 0s - loss: 3.0694 - accuracy: 0.2857
Epoch 23/500
1/1 - 0s - loss: 3.0682 - accuracy: 0.2381
Epoch 24/500
1/1 - 0s - loss: 3.0669 - accuracy: 0.2381
Epoch 25/500
1/1 - 0s - loss: 3.0657 - accuracy: 0.2381
Epoch 26/500
1/1 - 0s - loss: 3.0644 - accuracy: 0.2381
Epoch 27/500
1/1 - 0s - loss: 3.0630 - accuracy: 0.2381
Epoch 28/500
1/1 - 0s - loss: 3.0617 - accuracy: 0.2381
Epoch 29/500
1/1 - 0s - loss: 3.0602 - accuracy: 0.2381
Epoch 30/500
1/1 - 0s - loss: 3.0588 - accuracy: 0.2381
Epoch 31/500
1/1 - 0s - loss: 3.0573 - accuracy: 0.2381
Epoch 32/500
1/1 - 0s - loss: 3.0557 - accuracy: 0.2381
Epoch 33/500
1/1 - 0s - loss: 3.0542 - accuracy: 0.2381
Epoch 34/500
1/1 - 0s - loss: 3.0525 - accuracy: 0.2381
Epoch 35/500
1/1 - 0s - loss: 3.0509 - accuracy: 0.2381
Epoch 36/500
1/1 - 0s - loss: 3.0491 - accuracy: 0.2381
Epoch 37/500
1/1 - 0s - loss: 3.0474 - accuracy: 0.2381
Epoch 38/500
1/1 - 0s - loss: 3.0455 - accuracy: 0.2381
Epoch 39/500
1/1 - 0s - loss: 3.0436 - accuracy: 0.2381
Epoch 40/500
1/1 - 0s - loss: 3.0417 - accuracy: 0.2381
Epoch 41/500
1/1 - 0s - loss: 3.0397 - accuracy: 0.2381
Epoch 42/500
1/1 - 0s - loss: 3.0377 - accuracy: 0.2381
Epoch 43/500
1/1 - 0s - loss: 3.0355 - accuracy: 0.2381
Epoch 44/500
1/1 - 0s - loss: 3.0334 - accuracy: 0.2381
Epoch 45/500
1/1 - 0s - loss: 3.0311 - accuracy: 0.1905
Epoch 46/500
1/1 - 0s - loss: 3.0288 - accuracy: 0.1905
Epoch 47/500
1/1 - 0s - loss: 3.0264 - accuracy: 0.2381
Epoch 48/500
1/1 - 0s - loss: 3.0240 - accuracy: 0.2381
Epoch 49/500
1/1 - 0s - loss: 3.0215 - accuracy: 0.2381
Epoch 50/500
1/1 - 0s - loss: 3.0189 - accuracy: 0.2381
Epoch 51/500
1/1 - 0s - loss: 3.0162 - accuracy: 0.2381
Epoch 52/500
1/1 - 0s - loss: 3.0135 - accuracy: 0.2381
Epoch 53/500
1/1 - 0s - loss: 3.0106 - accuracy: 0.2381
Epoch 54/500
1/1 - 0s - loss: 3.0077 - accuracy: 0.2381
Epoch 55/500
1/1 - 0s - loss: 3.0047 - accuracy: 0.2381
Epoch 56/500
1/1 - 0s - loss: 3.0017 - accuracy: 0.2381
Epoch 57/500
1/1 - 0s - loss: 2.9985 - accuracy: 0.2381
Epoch 58/500
1/1 - 0s - loss: 2.9952 - accuracy: 0.2381
Epoch 59/500
1/1 - 0s - loss: 2.9919 - accuracy: 0.2381
Epoch 60/500
1/1 - 0s - loss: 2.9884 - accuracy: 0.2381
Epoch 61/500
1/1 - 0s - loss: 2.9849 - accuracy: 0.2381
Epoch 62/500
1/1 - 0s - loss: 2.9812 - accuracy: 0.2381
Epoch 63/500
1/1 - 0s - loss: 2.9774 - accuracy: 0.2381
Epoch 64/500
1/1 - 0s - loss: 2.9736 - accuracy: 0.2381
Epoch 65/500
1/1 - 0s - loss: 2.9696 - accuracy: 0.2381
Epoch 66/500
1/1 - 0s - loss: 2.9655 - accuracy: 0.2381
Epoch 67/500
1/1 - 0s - loss: 2.9613 - accuracy: 0.2381
Epoch 68/500
1/1 - 0s - loss: 2.9570 - accuracy: 0.2381
Epoch 69/500
1/1 - 0s - loss: 2.9526 - accuracy: 0.2381
Epoch 70/500
1/1 - 0s - loss: 2.9480 - accuracy: 0.2857
Epoch 71/500
1/1 - 0s - loss: 2.9433 - accuracy: 0.2857
Epoch 72/500
1/1 - 0s - loss: 2.9385 - accuracy: 0.2857
Epoch 73/500
1/1 - 0s - loss: 2.9336 - accuracy: 0.2857
Epoch 74/500
1/1 - 0s - loss: 2.9285 - accuracy: 0.2857
Epoch 75/500
1/1 - 0s - loss: 2.9233 - accuracy: 0.2857
Epoch 76/500
1/1 - 0s - loss: 2.9179 - accuracy: 0.2857
Epoch 77/500
1/1 - 0s - loss: 2.9125 - accuracy: 0.2857
Epoch 78/500
1/1 - 0s - loss: 2.9068 - accuracy: 0.2857
Epoch 79/500
1/1 - 0s - loss: 2.9010 - accuracy: 0.2857
Epoch 80/500
1/1 - 0s - loss: 2.8951 - accuracy: 0.2857
Epoch 81/500
1/1 - 0s - loss: 2.8890 - accuracy: 0.2857
Epoch 82/500
1/1 - 0s - loss: 2.8828 - accuracy: 0.2857
Epoch 83/500
1/1 - 0s - loss: 2.8764 - accuracy: 0.2857
Epoch 84/500
1/1 - 0s - loss: 2.8698 - accuracy: 0.2857
Epoch 85/500
1/1 - 0s - loss: 2.8631 - accuracy: 0.2857
Epoch 86/500
1/1 - 0s - loss: 2.8562 - accuracy: 0.2857
Epoch 87/500
1/1 - 0s - loss: 2.8492 - accuracy: 0.2857
Epoch 88/500
1/1 - 0s - loss: 2.8420 - accuracy: 0.2857
Epoch 89/500
1/1 - 0s - loss: 2.8346 - accuracy: 0.2857
Epoch 90/500
1/1 - 0s - loss: 2.8270 - accuracy: 0.2857
Epoch 91/500
1/1 - 0s - loss: 2.8193 - accuracy: 0.2857
Epoch 92/500
1/1 - 0s - loss: 2.8114 - accuracy: 0.2857
Epoch 93/500
1/1 - 0s - loss: 2.8033 - accuracy: 0.2857
Epoch 94/500
1/1 - 0s - loss: 2.7951 - accuracy: 0.2857
Epoch 95/500
1/1 - 0s - loss: 2.7866 - accuracy: 0.2857
Epoch 96/500
1/1 - 0s - loss: 2.7780 - accuracy: 0.2857
Epoch 97/500
1/1 - 0s - loss: 2.7692 - accuracy: 0.2857
Epoch 98/500
1/1 - 0s - loss: 2.7602 - accuracy: 0.2857
Epoch 99/500
1/1 - 0s - loss: 2.7510 - accuracy: 0.2857
Epoch 100/500
1/1 - 0s - loss: 2.7416 - accuracy: 0.2857
Epoch 101/500
1/1 - 0s - loss: 2.7321 - accuracy: 0.2857
Epoch 102/500
1/1 - 0s - loss: 2.7224 - accuracy: 0.2857
Epoch 103/500
1/1 - 0s - loss: 2.7124 - accuracy: 0.2857
Epoch 104/500
1/1 - 0s - loss: 2.7023 - accuracy: 0.2857
Epoch 105/500
1/1 - 0s - loss: 2.6920 - accuracy: 0.2857
Epoch 106/500
1/1 - 0s - loss: 2.6816 - accuracy: 0.2857
Epoch 107/500
1/1 - 0s - loss: 2.6709 - accuracy: 0.2857
Epoch 108/500
1/1 - 0s - loss: 2.6601 - accuracy: 0.2857
Epoch 109/500
1/1 - 0s - loss: 2.6491 - accuracy: 0.2857
Epoch 110/500
1/1 - 0s - loss: 2.6379 - accuracy: 0.2857
Epoch 111/500
1/1 - 0s - loss: 2.6266 - accuracy: 0.2857
Epoch 112/500
1/1 - 0s - loss: 2.6151 - accuracy: 0.2857
Epoch 113/500
1/1 - 0s - loss: 2.6034 - accuracy: 0.2857
Epoch 114/500
1/1 - 0s - loss: 2.5915 - accuracy: 0.2857
Epoch 115/500
1/1 - 0s - loss: 2.5795 - accuracy: 0.2857
Epoch 116/500
1/1 - 0s - loss: 2.5673 - accuracy: 0.2857
Epoch 117/500
1/1 - 0s - loss: 2.5550 - accuracy: 0.2857
Epoch 118/500
1/1 - 0s - loss: 2.5426 - accuracy: 0.2857
Epoch 119/500
1/1 - 0s - loss: 2.5300 - accuracy: 0.2857
Epoch 120/500
1/1 - 0s - loss: 2.5172 - accuracy: 0.2857
Epoch 121/500
1/1 - 0s - loss: 2.5044 - accuracy: 0.3333
Epoch 122/500
1/1 - 0s - loss: 2.4914 - accuracy: 0.3333
Epoch 123/500
1/1 - 0s - loss: 2.4782 - accuracy: 0.3333
Epoch 124/500
1/1 - 0s - loss: 2.4650 - accuracy: 0.3333
Epoch 125/500
1/1 - 0s - loss: 2.4517 - accuracy: 0.3810
Epoch 126/500
1/1 - 0s - loss: 2.4382 - accuracy: 0.3810
Epoch 127/500
1/1 - 0s - loss: 2.4247 - accuracy: 0.4286
Epoch 128/500
1/1 - 0s - loss: 2.4111 - accuracy: 0.4286
Epoch 129/500
1/1 - 0s - loss: 2.3974 - accuracy: 0.4286
Epoch 130/500
1/1 - 0s - loss: 2.3836 - accuracy: 0.4286
Epoch 131/500
1/1 - 0s - loss: 2.3697 - accuracy: 0.4762
Epoch 132/500
1/1 - 0s - loss: 2.3558 - accuracy: 0.4762
Epoch 133/500
1/1 - 0s - loss: 2.3419 - accuracy: 0.4762
Epoch 134/500
1/1 - 0s - loss: 2.3278 - accuracy: 0.5238
Epoch 135/500
1/1 - 0s - loss: 2.3138 - accuracy: 0.5238
Epoch 136/500
1/1 - 0s - loss: 2.2997 - accuracy: 0.5238
Epoch 137/500
1/1 - 0s - loss: 2.2856 - accuracy: 0.5238
Epoch 138/500
1/1 - 0s - loss: 2.2714 - accuracy: 0.5238
Epoch 139/500
1/1 - 0s - loss: 2.2572 - accuracy: 0.5238
Epoch 140/500
1/1 - 0s - loss: 2.2431 - accuracy: 0.5238
Epoch 141/500
1/1 - 0s - loss: 2.2289 - accuracy: 0.5238
Epoch 142/500
1/1 - 0s - loss: 2.2147 - accuracy: 0.5238
Epoch 143/500
1/1 - 0s - loss: 2.2005 - accuracy: 0.5238
Epoch 144/500
1/1 - 0s - loss: 2.1863 - accuracy: 0.5238
Epoch 145/500
1/1 - 0s - loss: 2.1721 - accuracy: 0.5238
Epoch 146/500
1/1 - 0s - loss: 2.1579 - accuracy: 0.5238
Epoch 147/500
1/1 - 0s - loss: 2.1438 - accuracy: 0.5238
Epoch 148/500
1/1 - 0s - loss: 2.1296 - accuracy: 0.5238
Epoch 149/500
1/1 - 0s - loss: 2.1155 - accuracy: 0.5238
Epoch 150/500
1/1 - 0s - loss: 2.1014 - accuracy: 0.5238
Epoch 151/500
1/1 - 0s - loss: 2.0874 - accuracy: 0.5238
Epoch 152/500
1/1 - 0s - loss: 2.0733 - accuracy: 0.5238
Epoch 153/500
1/1 - 0s - loss: 2.0593 - accuracy: 0.5238
Epoch 154/500
1/1 - 0s - loss: 2.0454 - accuracy: 0.5238
Epoch 155/500
1/1 - 0s - loss: 2.0314 - accuracy: 0.5238
Epoch 156/500
1/1 - 0s - loss: 2.0175 - accuracy: 0.5238
Epoch 157/500
1/1 - 0s - loss: 2.0037 - accuracy: 0.5238
Epoch 158/500
1/1 - 0s - loss: 1.9898 - accuracy: 0.5714
Epoch 159/500
1/1 - 0s - loss: 1.9760 - accuracy: 0.5714
Epoch 160/500
1/1 - 0s - loss: 1.9622 - accuracy: 0.5714
Epoch 161/500
1/1 - 0s - loss: 1.9485 - accuracy: 0.5714
Epoch 162/500
1/1 - 0s - loss: 1.9348 - accuracy: 0.5714
Epoch 163/500
1/1 - 0s - loss: 1.9211 - accuracy: 0.5714
Epoch 164/500
1/1 - 0s - loss: 1.9075 - accuracy: 0.5714
Epoch 165/500
1/1 - 0s - loss: 1.8939 - accuracy: 0.5714
Epoch 166/500
1/1 - 0s - loss: 1.8803 - accuracy: 0.5714
Epoch 167/500
1/1 - 0s - loss: 1.8668 - accuracy: 0.5714
Epoch 168/500
1/1 - 0s - loss: 1.8533 - accuracy: 0.6190
Epoch 169/500
1/1 - 0s - loss: 1.8398 - accuracy: 0.6190
Epoch 170/500
1/1 - 0s - loss: 1.8263 - accuracy: 0.6190
Epoch 171/500
1/1 - 0s - loss: 1.8129 - accuracy: 0.6190
Epoch 172/500
1/1 - 0s - loss: 1.7995 - accuracy: 0.6190
Epoch 173/500
1/1 - 0s - loss: 1.7861 - accuracy: 0.6190
Epoch 174/500
1/1 - 0s - loss: 1.7728 - accuracy: 0.6190
Epoch 175/500
1/1 - 0s - loss: 1.7595 - accuracy: 0.6190
Epoch 176/500
1/1 - 0s - loss: 1.7462 - accuracy: 0.6667
Epoch 177/500
1/1 - 0s - loss: 1.7329 - accuracy: 0.6667
Epoch 178/500
1/1 - 0s - loss: 1.7196 - accuracy: 0.6667
Epoch 179/500
1/1 - 0s - loss: 1.7064 - accuracy: 0.6667
Epoch 180/500
1/1 - 0s - loss: 1.6932 - accuracy: 0.6667
Epoch 181/500
1/1 - 0s - loss: 1.6800 - accuracy: 0.6667
Epoch 182/500
1/1 - 0s - loss: 1.6668 - accuracy: 0.6667
Epoch 183/500
1/1 - 0s - loss: 1.6537 - accuracy: 0.6667
Epoch 184/500
1/1 - 0s - loss: 1.6406 - accuracy: 0.6667
Epoch 185/500
1/1 - 0s - loss: 1.6275 - accuracy: 0.6667
Epoch 186/500
1/1 - 0s - loss: 1.6144 - accuracy: 0.6667
Epoch 187/500
1/1 - 0s - loss: 1.6013 - accuracy: 0.6667
Epoch 188/500
1/1 - 0s - loss: 1.5883 - accuracy: 0.7143
Epoch 189/500
1/1 - 0s - loss: 1.5753 - accuracy: 0.7619
Epoch 190/500
1/1 - 0s - loss: 1.5623 - accuracy: 0.7619
Epoch 191/500
1/1 - 0s - loss: 1.5493 - accuracy: 0.7619
Epoch 192/500
1/1 - 0s - loss: 1.5364 - accuracy: 0.7619
Epoch 193/500
1/1 - 0s - loss: 1.5235 - accuracy: 0.7619
Epoch 194/500
1/1 - 0s - loss: 1.5106 - accuracy: 0.7619
Epoch 195/500
1/1 - 0s - loss: 1.4977 - accuracy: 0.7619
Epoch 196/500
1/1 - 0s - loss: 1.4849 - accuracy: 0.7619
Epoch 197/500
1/1 - 0s - loss: 1.4721 - accuracy: 0.7619
Epoch 198/500
1/1 - 0s - loss: 1.4593 - accuracy: 0.7619
Epoch 199/500
1/1 - 0s - loss: 1.4466 - accuracy: 0.7619
Epoch 200/500
1/1 - 0s - loss: 1.4338 - accuracy: 0.8095
Epoch 201/500
1/1 - 0s - loss: 1.4212 - accuracy: 0.8095
Epoch 202/500
1/1 - 0s - loss: 1.4085 - accuracy: 0.8095
Epoch 203/500
1/1 - 0s - loss: 1.3959 - accuracy: 0.8095
Epoch 204/500
1/1 - 0s - loss: 1.3833 - accuracy: 0.8095
Epoch 205/500
1/1 - 0s - loss: 1.3708 - accuracy: 0.8095
Epoch 206/500
1/1 - 0s - loss: 1.3583 - accuracy: 0.8095
Epoch 207/500
1/1 - 0s - loss: 1.3459 - accuracy: 0.8095
Epoch 208/500
1/1 - 0s - loss: 1.3335 - accuracy: 0.8095
Epoch 209/500
1/1 - 0s - loss: 1.3211 - accuracy: 0.8095
Epoch 210/500
1/1 - 0s - loss: 1.3088 - accuracy: 0.8095
Epoch 211/500
1/1 - 0s - loss: 1.2966 - accuracy: 0.8095
Epoch 212/500
1/1 - 0s - loss: 1.2844 - accuracy: 0.8095
Epoch 213/500
1/1 - 0s - loss: 1.2722 - accuracy: 0.8095
Epoch 214/500
1/1 - 0s - loss: 1.2601 - accuracy: 0.8095
Epoch 215/500
1/1 - 0s - loss: 1.2481 - accuracy: 0.8095
Epoch 216/500
1/1 - 0s - loss: 1.2361 - accuracy: 0.8095
Epoch 217/500
1/1 - 0s - loss: 1.2242 - accuracy: 0.8095
Epoch 218/500
1/1 - 0s - loss: 1.2123 - accuracy: 0.8095
Epoch 219/500
1/1 - 0s - loss: 1.2005 - accuracy: 0.8095
Epoch 220/500
1/1 - 0s - loss: 1.1888 - accuracy: 0.8095
Epoch 221/500
1/1 - 0s - loss: 1.1772 - accuracy: 0.8095
Epoch 222/500
1/1 - 0s - loss: 1.1656 - accuracy: 0.8095
Epoch 223/500
1/1 - 0s - loss: 1.1541 - accuracy: 0.8095
Epoch 224/500
1/1 - 0s - loss: 1.1427 - accuracy: 0.8095
Epoch 225/500
1/1 - 0s - loss: 1.1313 - accuracy: 0.8571
Epoch 226/500
1/1 - 0s - loss: 1.1200 - accuracy: 0.8571
Epoch 227/500
1/1 - 0s - loss: 1.1088 - accuracy: 0.8571
Epoch 228/500
1/1 - 0s - loss: 1.0977 - accuracy: 0.8571
Epoch 229/500
1/1 - 0s - loss: 1.0867 - accuracy: 0.8571
Epoch 230/500
1/1 - 0s - loss: 1.0757 - accuracy: 0.8571
Epoch 231/500
1/1 - 0s - loss: 1.0649 - accuracy: 0.8571
Epoch 232/500
1/1 - 0s - loss: 1.0541 - accuracy: 0.8571
Epoch 233/500
1/1 - 0s - loss: 1.0434 - accuracy: 0.8571
Epoch 234/500
1/1 - 0s - loss: 1.0328 - accuracy: 0.8571
Epoch 235/500
1/1 - 0s - loss: 1.0223 - accuracy: 0.8571
Epoch 236/500
1/1 - 0s - loss: 1.0119 - accuracy: 0.8571
Epoch 237/500
1/1 - 0s - loss: 1.0015 - accuracy: 0.8571
Epoch 238/500
1/1 - 0s - loss: 0.9913 - accuracy: 0.8571
Epoch 239/500
1/1 - 0s - loss: 0.9812 - accuracy: 0.8571
Epoch 240/500
1/1 - 0s - loss: 0.9711 - accuracy: 0.8571
Epoch 241/500
1/1 - 0s - loss: 0.9612 - accuracy: 0.8571
Epoch 242/500
1/1 - 0s - loss: 0.9513 - accuracy: 0.8571
Epoch 243/500
1/1 - 0s - loss: 0.9416 - accuracy: 0.8571
Epoch 244/500
1/1 - 0s - loss: 0.9319 - accuracy: 0.8571
Epoch 245/500
1/1 - 0s - loss: 0.9223 - accuracy: 0.8571
Epoch 246/500
1/1 - 0s - loss: 0.9129 - accuracy: 0.8571
Epoch 247/500
1/1 - 0s - loss: 0.9035 - accuracy: 0.8571
Epoch 248/500
1/1 - 0s - loss: 0.8943 - accuracy: 0.8571
Epoch 249/500
1/1 - 0s - loss: 0.8851 - accuracy: 0.8571
Epoch 250/500
1/1 - 0s - loss: 0.8760 - accuracy: 0.8571
Epoch 251/500
1/1 - 0s - loss: 0.8671 - accuracy: 0.8571
Epoch 252/500
1/1 - 0s - loss: 0.8582 - accuracy: 0.8571
Epoch 253/500
1/1 - 0s - loss: 0.8495 - accuracy: 0.8571
Epoch 254/500
1/1 - 0s - loss: 0.8408 - accuracy: 0.8571
Epoch 255/500
1/1 - 0s - loss: 0.8322 - accuracy: 0.8571
Epoch 256/500
1/1 - 0s - loss: 0.8238 - accuracy: 0.8571
Epoch 257/500
1/1 - 0s - loss: 0.8154 - accuracy: 0.8571
Epoch 258/500
1/1 - 0s - loss: 0.8071 - accuracy: 0.8571
Epoch 259/500
1/1 - 0s - loss: 0.7990 - accuracy: 0.8571
Epoch 260/500
1/1 - 0s - loss: 0.7909 - accuracy: 0.8571
Epoch 261/500
1/1 - 0s - loss: 0.7829 - accuracy: 0.8571
Epoch 262/500
1/1 - 0s - loss: 0.7750 - accuracy: 0.8571
Epoch 263/500
1/1 - 0s - loss: 0.7673 - accuracy: 0.8571
Epoch 264/500
1/1 - 0s - loss: 0.7596 - accuracy: 0.8571
Epoch 265/500
1/1 - 0s - loss: 0.7520 - accuracy: 0.8571
Epoch 266/500
1/1 - 0s - loss: 0.7445 - accuracy: 0.8571
Epoch 267/500
1/1 - 0s - loss: 0.7371 - accuracy: 0.8571
Epoch 268/500
1/1 - 0s - loss: 0.7298 - accuracy: 0.8571
Epoch 269/500
1/1 - 0s - loss: 0.7226 - accuracy: 0.8571
Epoch 270/500
1/1 - 0s - loss: 0.7155 - accuracy: 0.8571
Epoch 271/500
1/1 - 0s - loss: 0.7085 - accuracy: 0.8571
Epoch 272/500
1/1 - 0s - loss: 0.7016 - accuracy: 0.8571
Epoch 273/500
1/1 - 0s - loss: 0.6948 - accuracy: 0.8571
Epoch 274/500
1/1 - 0s - loss: 0.6880 - accuracy: 0.8571
Epoch 275/500
1/1 - 0s - loss: 0.6814 - accuracy: 0.8571
Epoch 276/500
1/1 - 0s - loss: 0.6748 - accuracy: 0.8571
Epoch 277/500
1/1 - 0s - loss: 0.6683 - accuracy: 0.8571
Epoch 278/500
1/1 - 0s - loss: 0.6620 - accuracy: 0.8571
Epoch 279/500
1/1 - 0s - loss: 0.6557 - accuracy: 0.8571
Epoch 280/500
1/1 - 0s - loss: 0.6494 - accuracy: 0.8571
Epoch 281/500
1/1 - 0s - loss: 0.6433 - accuracy: 0.8571
Epoch 282/500
1/1 - 0s - loss: 0.6373 - accuracy: 0.8571
Epoch 283/500
1/1 - 0s - loss: 0.6313 - accuracy: 0.8571
Epoch 284/500
1/1 - 0s - loss: 0.6255 - accuracy: 0.8571
Epoch 285/500
1/1 - 0s - loss: 0.6197 - accuracy: 0.8571
Epoch 286/500
1/1 - 0s - loss: 0.6140 - accuracy: 0.8571
Epoch 287/500
1/1 - 0s - loss: 0.6084 - accuracy: 0.8571
Epoch 288/500
1/1 - 0s - loss: 0.6028 - accuracy: 0.8571
Epoch 289/500
1/1 - 0s - loss: 0.5974 - accuracy: 0.8571
Epoch 290/500
1/1 - 0s - loss: 0.5920 - accuracy: 0.8571
Epoch 291/500
1/1 - 0s - loss: 0.5867 - accuracy: 0.8571
Epoch 292/500
1/1 - 0s - loss: 0.5815 - accuracy: 0.8571
Epoch 293/500
1/1 - 0s - loss: 0.5763 - accuracy: 0.8571
Epoch 294/500
1/1 - 0s - loss: 0.5712 - accuracy: 0.8571
Epoch 295/500
1/1 - 0s - loss: 0.5663 - accuracy: 0.8571
Epoch 296/500
1/1 - 0s - loss: 0.5613 - accuracy: 0.8571
Epoch 297/500
1/1 - 0s - loss: 0.5565 - accuracy: 0.8571
Epoch 298/500
1/1 - 0s - loss: 0.5517 - accuracy: 0.8571
Epoch 299/500
1/1 - 0s - loss: 0.5470 - accuracy: 0.8571
Epoch 300/500
1/1 - 0s - loss: 0.5424 - accuracy: 0.8571
Epoch 301/500
1/1 - 0s - loss: 0.5378 - accuracy: 0.8571
Epoch 302/500
1/1 - 0s - loss: 0.5333 - accuracy: 0.8571
Epoch 303/500
1/1 - 0s - loss: 0.5289 - accuracy: 0.8571
Epoch 304/500
1/1 - 0s - loss: 0.5245 - accuracy: 0.8571
Epoch 305/500
1/1 - 0s - loss: 0.5203 - accuracy: 0.8571
Epoch 306/500
1/1 - 0s - loss: 0.5160 - accuracy: 0.8571
Epoch 307/500
1/1 - 0s - loss: 0.5119 - accuracy: 0.8571
Epoch 308/500
1/1 - 0s - loss: 0.5078 - accuracy: 0.8571
Epoch 309/500
1/1 - 0s - loss: 0.5038 - accuracy: 0.8571
Epoch 310/500
1/1 - 0s - loss: 0.4998 - accuracy: 0.8571
Epoch 311/500
1/1 - 0s - loss: 0.4959 - accuracy: 0.8571
Epoch 312/500
1/1 - 0s - loss: 0.4921 - accuracy: 0.8571
Epoch 313/500
1/1 - 0s - loss: 0.4883 - accuracy: 0.8571
Epoch 314/500
1/1 - 0s - loss: 0.4846 - accuracy: 0.8571
Epoch 315/500
1/1 - 0s - loss: 0.4809 - accuracy: 0.8571
Epoch 316/500
1/1 - 0s - loss: 0.4773 - accuracy: 0.8571
Epoch 317/500
1/1 - 0s - loss: 0.4738 - accuracy: 0.8571
Epoch 318/500
1/1 - 0s - loss: 0.4703 - accuracy: 0.8571
Epoch 319/500
1/1 - 0s - loss: 0.4668 - accuracy: 0.8571
Epoch 320/500
1/1 - 0s - loss: 0.4635 - accuracy: 0.8571
Epoch 321/500
1/1 - 0s - loss: 0.4602 - accuracy: 0.8571
Epoch 322/500
1/1 - 0s - loss: 0.4569 - accuracy: 0.8571
Epoch 323/500
1/1 - 0s - loss: 0.4537 - accuracy: 0.8571
Epoch 324/500
1/1 - 0s - loss: 0.4505 - accuracy: 0.8571
Epoch 325/500
1/1 - 0s - loss: 0.4474 - accuracy: 0.8571
Epoch 326/500
1/1 - 0s - loss: 0.4443 - accuracy: 0.8571
Epoch 327/500
1/1 - 0s - loss: 0.4413 - accuracy: 0.8571
Epoch 328/500
1/1 - 0s - loss: 0.4384 - accuracy: 0.8571
Epoch 329/500
1/1 - 0s - loss: 0.4354 - accuracy: 0.8571
Epoch 330/500
1/1 - 0s - loss: 0.4326 - accuracy: 0.8571
Epoch 331/500
1/1 - 0s - loss: 0.4298 - accuracy: 0.8571
Epoch 332/500
1/1 - 0s - loss: 0.4270 - accuracy: 0.8571
Epoch 333/500
1/1 - 0s - loss: 0.4243 - accuracy: 0.8571
Epoch 334/500
1/1 - 0s - loss: 0.4216 - accuracy: 0.8571
Epoch 335/500
1/1 - 0s - loss: 0.4189 - accuracy: 0.8571
Epoch 336/500
1/1 - 0s - loss: 0.4164 - accuracy: 0.8571
Epoch 337/500
1/1 - 0s - loss: 0.4138 - accuracy: 0.8571
Epoch 338/500
1/1 - 0s - loss: 0.4113 - accuracy: 0.8571
Epoch 339/500
1/1 - 0s - loss: 0.4088 - accuracy: 0.8571
Epoch 340/500
1/1 - 0s - loss: 0.4064 - accuracy: 0.8571
Epoch 341/500
1/1 - 0s - loss: 0.4040 - accuracy: 0.8571
Epoch 342/500
1/1 - 0s - loss: 0.4017 - accuracy: 0.8571
Epoch 343/500
1/1 - 0s - loss: 0.3994 - accuracy: 0.8571
Epoch 344/500
1/1 - 0s - loss: 0.3971 - accuracy: 0.8571
Epoch 345/500
1/1 - 0s - loss: 0.3948 - accuracy: 0.8571
Epoch 346/500
1/1 - 0s - loss: 0.3927 - accuracy: 0.8571
Epoch 347/500
1/1 - 0s - loss: 0.3905 - accuracy: 0.8571
Epoch 348/500
1/1 - 0s - loss: 0.3884 - accuracy: 0.8571
Epoch 349/500
1/1 - 0s - loss: 0.3863 - accuracy: 0.8571
Epoch 350/500
1/1 - 0s - loss: 0.3842 - accuracy: 0.8571
Epoch 351/500
1/1 - 0s - loss: 0.3822 - accuracy: 0.8571
Epoch 352/500
1/1 - 0s - loss: 0.3802 - accuracy: 0.8571
Epoch 353/500
1/1 - 0s - loss: 0.3783 - accuracy: 0.8571
Epoch 354/500
1/1 - 0s - loss: 0.3763 - accuracy: 0.8571
Epoch 355/500
1/1 - 0s - loss: 0.3744 - accuracy: 0.8571
Epoch 356/500
1/1 - 0s - loss: 0.3726 - accuracy: 0.8571
Epoch 357/500
1/1 - 0s - loss: 0.3707 - accuracy: 0.8571
Epoch 358/500
1/1 - 0s - loss: 0.3689 - accuracy: 0.8571
Epoch 359/500
1/1 - 0s - loss: 0.3672 - accuracy: 0.8571
Epoch 360/500
1/1 - 0s - loss: 0.3654 - accuracy: 0.8571
Epoch 361/500
1/1 - 0s - loss: 0.3637 - accuracy: 0.8571
Epoch 362/500
1/1 - 0s - loss: 0.3620 - accuracy: 0.8571
Epoch 363/500
1/1 - 0s - loss: 0.3604 - accuracy: 0.8571
Epoch 364/500
1/1 - 0s - loss: 0.3587 - accuracy: 0.8571
Epoch 365/500
1/1 - 0s - loss: 0.3571 - accuracy: 0.8571
Epoch 366/500
1/1 - 0s - loss: 0.3556 - accuracy: 0.8571
Epoch 367/500
1/1 - 0s - loss: 0.3540 - accuracy: 0.8571
Epoch 368/500
1/1 - 0s - loss: 0.3525 - accuracy: 0.8571
Epoch 369/500
1/1 - 0s - loss: 0.3510 - accuracy: 0.8571
Epoch 370/500
1/1 - 0s - loss: 0.3495 - accuracy: 0.8571
Epoch 371/500
1/1 - 0s - loss: 0.3480 - accuracy: 0.8571
Epoch 372/500
1/1 - 0s - loss: 0.3466 - accuracy: 0.8571
Epoch 373/500
1/1 - 0s - loss: 0.3452 - accuracy: 0.8571
Epoch 374/500
1/1 - 0s - loss: 0.3438 - accuracy: 0.8571
Epoch 375/500
1/1 - 0s - loss: 0.3424 - accuracy: 0.8571
Epoch 376/500
1/1 - 0s - loss: 0.3411 - accuracy: 0.8571
Epoch 377/500
1/1 - 0s - loss: 0.3398 - accuracy: 0.8571
Epoch 378/500
1/1 - 0s - loss: 0.3385 - accuracy: 0.8571
Epoch 379/500
1/1 - 0s - loss: 0.3372 - accuracy: 0.8571
Epoch 380/500
1/1 - 0s - loss: 0.3359 - accuracy: 0.8571
Epoch 381/500
1/1 - 0s - loss: 0.3347 - accuracy: 0.8571
Epoch 382/500
1/1 - 0s - loss: 0.3335 - accuracy: 0.8571
Epoch 383/500
1/1 - 0s - loss: 0.3322 - accuracy: 0.8571
Epoch 384/500
1/1 - 0s - loss: 0.3311 - accuracy: 0.8571
Epoch 385/500
1/1 - 0s - loss: 0.3299 - accuracy: 0.8571
Epoch 386/500
1/1 - 0s - loss: 0.3287 - accuracy: 0.8571
Epoch 387/500
1/1 - 0s - loss: 0.3276 - accuracy: 0.8571
Epoch 388/500
1/1 - 0s - loss: 0.3265 - accuracy: 0.8571
Epoch 389/500
1/1 - 0s - loss: 0.3254 - accuracy: 0.8571
Epoch 390/500
1/1 - 0s - loss: 0.3243 - accuracy: 0.8571
Epoch 391/500
1/1 - 0s - loss: 0.3233 - accuracy: 0.8571
Epoch 392/500
1/1 - 0s - loss: 0.3222 - accuracy: 0.8571
Epoch 393/500
1/1 - 0s - loss: 0.3212 - accuracy: 0.8571
Epoch 394/500
1/1 - 0s - loss: 0.3202 - accuracy: 0.8571
Epoch 395/500
1/1 - 0s - loss: 0.3192 - accuracy: 0.8571
Epoch 396/500
1/1 - 0s - loss: 0.3182 - accuracy: 0.8571
Epoch 397/500
1/1 - 0s - loss: 0.3172 - accuracy: 0.8571
Epoch 398/500
1/1 - 0s - loss: 0.3162 - accuracy: 0.8571
Epoch 399/500
1/1 - 0s - loss: 0.3153 - accuracy: 0.8571
Epoch 400/500
1/1 - 0s - loss: 0.3144 - accuracy: 0.8571
Epoch 401/500
1/1 - 0s - loss: 0.3134 - accuracy: 0.8571
Epoch 402/500
1/1 - 0s - loss: 0.3125 - accuracy: 0.8571
Epoch 403/500
1/1 - 0s - loss: 0.3117 - accuracy: 0.8571
Epoch 404/500
1/1 - 0s - loss: 0.3108 - accuracy: 0.8571
Epoch 405/500
1/1 - 0s - loss: 0.3099 - accuracy: 0.8571
Epoch 406/500
1/1 - 0s - loss: 0.3091 - accuracy: 0.8571
Epoch 407/500
1/1 - 0s - loss: 0.3082 - accuracy: 0.8571
Epoch 408/500
1/1 - 0s - loss: 0.3074 - accuracy: 0.8571
Epoch 409/500
1/1 - 0s - loss: 0.3066 - accuracy: 0.8571
Epoch 410/500
1/1 - 0s - loss: 0.3058 - accuracy: 0.8571
Epoch 411/500
1/1 - 0s - loss: 0.3050 - accuracy: 0.8571
Epoch 412/500
1/1 - 0s - loss: 0.3042 - accuracy: 0.8571
Epoch 413/500
1/1 - 0s - loss: 0.3034 - accuracy: 0.8571
Epoch 414/500
1/1 - 0s - loss: 0.3027 - accuracy: 0.8571
Epoch 415/500
1/1 - 0s - loss: 0.3019 - accuracy: 0.8571
Epoch 416/500
1/1 - 0s - loss: 0.3012 - accuracy: 0.8571
Epoch 417/500
1/1 - 0s - loss: 0.3004 - accuracy: 0.8571
Epoch 418/500
1/1 - 0s - loss: 0.2997 - accuracy: 0.8571
Epoch 419/500
1/1 - 0s - loss: 0.2990 - accuracy: 0.8571
Epoch 420/500
1/1 - 0s - loss: 0.2983 - accuracy: 0.8571
Epoch 421/500
1/1 - 0s - loss: 0.2976 - accuracy: 0.8571
Epoch 422/500
1/1 - 0s - loss: 0.2969 - accuracy: 0.8571
Epoch 423/500
1/1 - 0s - loss: 0.2963 - accuracy: 0.8571
Epoch 424/500
1/1 - 0s - loss: 0.2956 - accuracy: 0.8571
Epoch 425/500
1/1 - 0s - loss: 0.2950 - accuracy: 0.8571
Epoch 426/500
1/1 - 0s - loss: 0.2943 - accuracy: 0.8571
Epoch 427/500
1/1 - 0s - loss: 0.2937 - accuracy: 0.8571
Epoch 428/500
1/1 - 0s - loss: 0.2931 - accuracy: 0.8571
Epoch 429/500
1/1 - 0s - loss: 0.2924 - accuracy: 0.8571
Epoch 430/500
1/1 - 0s - loss: 0.2918 - accuracy: 0.8571
Epoch 431/500
1/1 - 0s - loss: 0.2912 - accuracy: 0.8571
Epoch 432/500
1/1 - 0s - loss: 0.2906 - accuracy: 0.8571
Epoch 433/500
1/1 - 0s - loss: 0.2900 - accuracy: 0.8571
Epoch 434/500
1/1 - 0s - loss: 0.2895 - accuracy: 0.8571
Epoch 435/500
1/1 - 0s - loss: 0.2889 - accuracy: 0.8571
Epoch 436/500
1/1 - 0s - loss: 0.2883 - accuracy: 0.8571
Epoch 437/500
1/1 - 0s - loss: 0.2878 - accuracy: 0.8571
Epoch 438/500
1/1 - 0s - loss: 0.2872 - accuracy: 0.8571
Epoch 439/500
1/1 - 0s - loss: 0.2867 - accuracy: 0.8571
Epoch 440/500
1/1 - 0s - loss: 0.2861 - accuracy: 0.8571
Epoch 441/500
1/1 - 0s - loss: 0.2856 - accuracy: 0.8571
Epoch 442/500
1/1 - 0s - loss: 0.2851 - accuracy: 0.8571
Epoch 443/500
1/1 - 0s - loss: 0.2846 - accuracy: 0.8571
Epoch 444/500
1/1 - 0s - loss: 0.2840 - accuracy: 0.8571
Epoch 445/500
1/1 - 0s - loss: 0.2835 - accuracy: 0.8571
Epoch 446/500
1/1 - 0s - loss: 0.2830 - accuracy: 0.8571
Epoch 447/500
1/1 - 0s - loss: 0.2826 - accuracy: 0.8571
Epoch 448/500
1/1 - 0s - loss: 0.2821 - accuracy: 0.8571
Epoch 449/500
1/1 - 0s - loss: 0.2816 - accuracy: 0.8571
Epoch 450/500
1/1 - 0s - loss: 0.2811 - accuracy: 0.8571
Epoch 451/500
1/1 - 0s - loss: 0.2806 - accuracy: 0.8571
Epoch 452/500
1/1 - 0s - loss: 0.2802 - accuracy: 0.8571
Epoch 453/500
1/1 - 0s - loss: 0.2797 - accuracy: 0.8571
Epoch 454/500
1/1 - 0s - loss: 0.2793 - accuracy: 0.8571
Epoch 455/500
1/1 - 0s - loss: 0.2788 - accuracy: 0.8571
Epoch 456/500
1/1 - 0s - loss: 0.2784 - accuracy: 0.8571
Epoch 457/500
1/1 - 0s - loss: 0.2780 - accuracy: 0.8571
Epoch 458/500
1/1 - 0s - loss: 0.2775 - accuracy: 0.8571
Epoch 459/500
1/1 - 0s - loss: 0.2771 - accuracy: 0.8571
Epoch 460/500
1/1 - 0s - loss: 0.2767 - accuracy: 0.8571
Epoch 461/500
1/1 - 0s - loss: 0.2763 - accuracy: 0.8571
Epoch 462/500
1/1 - 0s - loss: 0.2759 - accuracy: 0.8571
Epoch 463/500
1/1 - 0s - loss: 0.2754 - accuracy: 0.8571
Epoch 464/500
1/1 - 0s - loss: 0.2750 - accuracy: 0.8571
Epoch 465/500
1/1 - 0s - loss: 0.2747 - accuracy: 0.8571
Epoch 466/500
1/1 - 0s - loss: 0.2743 - accuracy: 0.8571
Epoch 467/500
1/1 - 0s - loss: 0.2739 - accuracy: 0.8571
Epoch 468/500
1/1 - 0s - loss: 0.2735 - accuracy: 0.8571
Epoch 469/500
1/1 - 0s - loss: 0.2731 - accuracy: 0.8571
Epoch 470/500
1/1 - 0s - loss: 0.2727 - accuracy: 0.8571
Epoch 471/500
1/1 - 0s - loss: 0.2724 - accuracy: 0.8571
Epoch 472/500
1/1 - 0s - loss: 0.2720 - accuracy: 0.8571
Epoch 473/500
1/1 - 0s - loss: 0.2716 - accuracy: 0.8571
Epoch 474/500
1/1 - 0s - loss: 0.2713 - accuracy: 0.8571
Epoch 475/500
1/1 - 0s - loss: 0.2709 - accuracy: 0.8571
Epoch 476/500
1/1 - 0s - loss: 0.2706 - accuracy: 0.8571
Epoch 477/500
1/1 - 0s - loss: 0.2702 - accuracy: 0.8571
Epoch 478/500
1/1 - 0s - loss: 0.2699 - accuracy: 0.8571
Epoch 479/500
1/1 - 0s - loss: 0.2696 - accuracy: 0.8571
Epoch 480/500
1/1 - 0s - loss: 0.2692 - accuracy: 0.8571
Epoch 481/500
1/1 - 0s - loss: 0.2689 - accuracy: 0.8571
Epoch 482/500
1/1 - 0s - loss: 0.2686 - accuracy: 0.8571
Epoch 483/500
1/1 - 0s - loss: 0.2682 - accuracy: 0.8571
Epoch 484/500
1/1 - 0s - loss: 0.2679 - accuracy: 0.8571
Epoch 485/500
1/1 - 0s - loss: 0.2676 - accuracy: 0.8571
Epoch 486/500
1/1 - 0s - loss: 0.2673 - accuracy: 0.8571
Epoch 487/500
1/1 - 0s - loss: 0.2670 - accuracy: 0.8571
Epoch 488/500
1/1 - 0s - loss: 0.2667 - accuracy: 0.8571
Epoch 489/500
1/1 - 0s - loss: 0.2664 - accuracy: 0.8571
Epoch 490/500
1/1 - 0s - loss: 0.2661 - accuracy: 0.8571
Epoch 491/500
1/1 - 0s - loss: 0.2658 - accuracy: 0.8571
Epoch 492/500
1/1 - 0s - loss: 0.2655 - accuracy: 0.8571
Epoch 493/500
1/1 - 0s - loss: 0.2652 - accuracy: 0.8571
Epoch 494/500
1/1 - 0s - loss: 0.2649 - accuracy: 0.8571
Epoch 495/500
1/1 - 0s - loss: 0.2646 - accuracy: 0.8571
Epoch 496/500
1/1 - 0s - loss: 0.2643 - accuracy: 0.8571
Epoch 497/500
1/1 - 0s - loss: 0.2640 - accuracy: 0.8571
Epoch 498/500
1/1 - 0s - loss: 0.2638 - accuracy: 0.8571
Epoch 499/500
1/1 - 0s - loss: 0.2635 - accuracy: 0.8571
Epoch 500/500
1/1 - 0s - loss: 0.2632 - accuracy: 0.8571
<tensorflow.python.keras.callbacks.History at 0x7ff300c072e8>
plot_model(model)
../_images/71d3336deba2135b2510e970845e93e6e3fb3d9c506cdae89203305a3392466b.png

Text Generation Using the Model#

  • After we trained the bigram-based LM, we can use the model for text generation (e.g., a one-to-many sequence-to-sequence application).

  • We can implement a simple text generator: the model always outputs the next-word that has the highest predicted probability value from the neural LM.

  • At every time step, the model will use the newly predicted word as the input for another next-word prediction.

# generate a sequence from the model
def generate_seq(model, tokenizer, seed_text, n_words):
    in_text, result = seed_text, seed_text
    # generate a fixed number of words
    for _ in range(n_words):
        # encode the text as integer
        encoded = tokenizer.texts_to_sequences([in_text])[0]
        encoded = np.array(encoded)
        
        # predict a word in the vocabulary
        yhat=np.argmax(model.predict(encoded), axis=-1)
        
        # map predicted word index to word
        out_word = ''
        for word, index in tokenizer.word_index.items():
            if index == yhat:
                out_word = word
                break
        # append to input
        in_text, result = out_word, result + ' ' + out_word
    return result
  • In the above generate_seq(), we use a greedy search, which selects the most likely word at each time step in the output sequence.

  • While this approach features its efficiency, the quality of the final output sequences may not necessarily be optimal.

# evaluate
print(generate_seq(model, tokenizer, 'Jill', 10))
Jill came tumbling after and jill came tumbling after and jill

Sampling Strategies for Text Generation#

  • Given a trained language model and a seed text chunk, we can generate new text by greedy-search like we’ve seen above.

  • But we may sometimes have to add a certain degree of variation to the robotic texts for linguistic creativity.

  • Possible alternatives:

    • We can re-normalize the predicted probability distributions of all next-words to reduce probability differences between the highest and the lowest. (Please see Ch.8.1 Text Generation with LSTM in Chollet’s Deep Learning with Python. You will need this strategy for the assignment.)

    • We can use non-greedy search by keeping the top k probable candidates in the list for next-word prediction. (cf. Beam Search below) and determine the tokens by choosing the sequence of the maximum probability.

Beam Search (skipped)#

Searching in NLP#

  • In the previous demonstration, when we generate the predicted next word, we adopt a naive approach, i.e., always choosing the word of the highest probability.

  • It is common in NLP for models to output a probability distribution over words in the vocabulary.

  • This step involves searching through all the possible output sequences based on their likelihood.

  • Choosing the next word of highest probability does not guarantee us the most optimal sequence.

  • The search problem is exponential in the length of the output sequence given the large size of vocabulary.

Beam Search Decoding#

The beam search expands all possible next steps and keeps the \(k\) most likely, where \(k\) is a researcher-specified parameter and controls the number of beams or parallel searches through the sequence of probabilities.

The search process can stop for each candidate independently either by:

  • reaching a maximum length

  • reaching an end-of-sequence token

  • reaching a threshold likelihood

Please see Jason Brownlee's blog post [How to Implement a Beam Search Decoder for Natural Language Processing](https://machinelearningmastery.com/beam-search-decoder-natural-language-processing/) for the python implementation.

The following codes are based on Jason's code.

Warning

The following codes may not work properly. In Beam Search, when the model predicts None as the next character, we should set it as a stopping condition. The following codes have not be optimized with respect to this.

# generate a sequence from the model
def generate_seq_beam(model, tokenizer, seed_text, n_words, k):
    in_text = seed_text 
    sequences = [[[in_text], 0.0]]
    # prepare id_2_word map
    id_2_word = dict([(i,w) for (w, i) in tokenizer.word_index.items()])
    
    # start next-word generating
    for _ in range(n_words):
        all_candidates = list()        
        #print("Next word ", _+1)
        # temp list to hold all possible candidates
        # `sequence + next words`


        # for each existing sequence
        # take the last word of the sequence
        # find probs of all words in the next position
        # save the top k
        # all_candidates should have 3 * 22 = 66 candidates
        
        for i in range(len(sequences)):
            # for the current sequence
            seq, score = sequences[i]            
            # next word probablity distribution
            encoded = tokenizer.texts_to_sequences([seq[-1]])[0]
            encoded = np.array(encoded)
            model_pred_prob = model.predict(encoded).flatten()

            # compute all probabilities for `curent_sequence + all_possible_next_word`
            for j in range(len(model_pred_prob)):
                candidate = [seq + [id_2_word.get(j+1)], score-np.log(model_pred_prob[j])]
                all_candidates.append(candidate)

            all_candidates= [[seq, score] for seq, score in all_candidates if seq[-1] is not None]

            # order all candidates (seqence + nextword) by score
            #print("all_condidates length:", len(all_candidates))
            ordered = sorted(all_candidates, key = lambda x:x[1]) # default ascending
            # select k best
            sequences = ordered[:k] ## choose top k

    return sequences
generate_seq_beam(model, tokenizer, 'Jill', 5, k =10)
[[['Jill', 'tumbling', 'a', 'of', 'fell', 'broke'], 5.70021602883935],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'pail'], 9.975788600742817],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'pail', 'water'],
  10.006022842600942],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'came'], 10.572875507175922],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'came', 'after'], 10.62351381778717],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'tumbling'], 10.633405216038227],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'up'], 10.789379127323627],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'up', 'hill'], 10.838647928088903],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'of'], 11.02272368222475],
 [['Jill', 'tumbling', 'a', 'of', 'fell', 'of', 'fell'], 11.054580122232437]]

References#