Hyper-Parameter Tuning

Hyper-Parameter Tuning#

There are two important techniques to fine-tune the hyperparameters of the model: Grid Search and Cross Validation.

Grid Search
- Define a few parameter values and experiment all these values in modeling. Use sklearn.model_selection.GridSearchCV to find the best parameter settings.
Cross Validation
- Fine-tune the parameters using cross-validation. Common CV methods include sklearn.model_selection.StratifiedKFold, sklearn_model_selection.ShuffleSplit, LeaveOneOut.

Both mothods can make use of the pipeline in sklearn to streamline the processing of training and validation.

SVM Model#

This example is from the officient sklearn documentation
A classic SVM model with train and test split

## Normal SVM model
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm

X, y = datasets.load_iris(return_X_y=True)
X.shape, y.shape

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=0)

X_train.shape, y_train.shape

X_test.shape, y_test.shape

## fit on train sets
clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
## test on test sets
clf.score(X_test, y_test)

0.9666666666666667

Default K-fold Cross Validation#

We can easily use the cross_val_score() to do the K-fold cross validation for a specific training model (without shuffling)

Note

By default, train_test_split returns a random split.

## Cross-validation

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, X, y, cv=10, scoring='f1_macro')
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.97 (+/- 0.09)

GridSearchCV and Fine-Tuning Hyper-Parameters#

Important steps:
- Define SVM classifier
- Define a set of parameter values to experiment
- Use GridSearchCV to find the best parameter settings
- GridSearchCV by default implements cross-validation methods to find the optimal parameter settings.
- In other words, we can specify our own more sophisticated CV methods in GridSearchCV.

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters, cv=5) # 
clf.fit(iris.data, iris.target)


sorted(clf.cv_results_.keys())

#print(clf.cv_results_)

import pandas as pd

cv_results_df=pd.DataFrame(clf.cv_results_)
cv_results_df

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_C	param_kernel	params	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	0.000737	0.000221	0.000469	0.000225	1	linear	{'C': 1, 'kernel': 'linear'}	0.966667	1.000000	0.966667	0.966667	1.0	0.980000	0.016330	1
1	0.000651	0.000035	0.000331	0.000015	1	rbf	{'C': 1, 'kernel': 'rbf'}	0.966667	0.966667	0.966667	0.933333	1.0	0.966667	0.021082	4
2	0.000511	0.000027	0.000276	0.000013	10	linear	{'C': 10, 'kernel': 'linear'}	1.000000	1.000000	0.900000	0.966667	1.0	0.973333	0.038873	3
3	0.000604	0.000047	0.000344	0.000070	10	rbf	{'C': 10, 'kernel': 'rbf'}	0.966667	1.000000	0.966667	0.966667	1.0	0.980000	0.016330	1

Cross-validation, Hyper-Parameter Tuning, and Pipeline#

Common cross validation methods:
- StratifiedKFold: Split data into train and validation sets by preserving the percentage of samples of each class
- ShuffleSplit: Split data into train and validation sets by first shuffling the data and then splitting
- StratifiedShuffleSplit: Stratified + Shuffled
- LeaveOneOut: Creating train sets by taking all samples execept one, which is left out for validation
Important Steps
- Define all preprocessing methods as Tranasformer
- Create a pipeline
- Define a cross-validation method
- Use cross_val_score() to fine the best parameter settings by feeding the pipeline and the CV method

from sklearn.pipeline import make_pipeline
from sklearn import preprocessing
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import ShuffleSplit

## Create pipeline
clf = make_pipeline(preprocessing.StandardScaler(), svm.SVC(C=1))

## Define CV methods
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
shuffsplit = ShuffleSplit(n_splits=5, test_size=0.1, random_state=7)

## CV results
print('K-fold CV:', cross_val_score(clf, X, y, cv=kfold))
print('Shuffle CV:', cross_val_score(clf, X, y, cv=shuffsplit))

K-fold CV: [0.96666667 0.96666667 0.96666667 0.9        1.        ]
Shuffle CV: [0.86666667 1.         1.         1.         0.93333333]

Deep Learning Example#

This is based on GridSearchCV with keras

MNIST Dataset#

import numpy as np
import os
from keras.datasets import mnist
from keras.layers import *
from keras.models import *
from time import time
nb_classes = 10
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 784)
X_test = X_test.reshape(X_test.shape[0], 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

## Convert y labels (integers) into binary class matrix
from keras.utils import np_utils
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
print(Y_train[:5,])

[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]

Hyper-Parameter Tuning Using Grid Search#

In this example, we aim to fine-tune the following hyper-parameters of the deep neural network:

optimizer
kernel_initializer of the Dense layer

We can fine-tune other common parameters like epochs and batch-sizes.

def create_model(optimizer='rmsprop', init='glorot_uniform'):
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
    model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or "overfitting" the training data
    model.add(Dense(512, kernel_initializer=init))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10, kernel_initializer=init))
    model.add(Activation('softmax')) # This special "softmax" a
    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy']) 
    return model

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
start=time()
model = KerasClassifier(build_fn=create_model)
optimizers = ['rmsprop', 'adam']
init = ['glorot_uniform', 'normal', 'uniform']
# epochs = [50, 100, 150]
# batches = [5, 10, 20]
param_grid = dict(optimizer=optimizers, init=init)
grid = GridSearchCV(estimator=model, param_grid=param_grid)
grid_result = grid.fit(X_train, Y_train)
print("total time:",time()-start)

1500/1500 [==============================] - 4s 3ms/step - loss: 0.0417 - accuracy: 0.9264
375/375 [==============================] - 1s 2ms/step - loss: 0.0259 - accuracy: 0.9593
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9270
375/375 [==============================] - 1s 2ms/step - loss: 0.0226 - accuracy: 0.9651
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0410 - accuracy: 0.9282
375/375 [==============================] - 1s 2ms/step - loss: 0.0270 - accuracy: 0.9578
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0415 - accuracy: 0.9274
375/375 [==============================] - 1s 2ms/step - loss: 0.0289 - accuracy: 0.9567
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0419 - accuracy: 0.9270
375/375 [==============================] - 1s 2ms/step - loss: 0.0221 - accuracy: 0.9633
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9271
375/375 [==============================] - 1s 2ms/step - loss: 0.0220 - accuracy: 0.9626
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0399 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0239 - accuracy: 0.9604
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0397 - accuracy: 0.9293
375/375 [==============================] - 1s 2ms/step - loss: 0.0232 - accuracy: 0.9592
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0219 - accuracy: 0.9624
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9271
375/375 [==============================] - 1s 2ms/step - loss: 0.0213 - accuracy: 0.9628
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0421 - accuracy: 0.9262
375/375 [==============================] - 1s 2ms/step - loss: 0.0254 - accuracy: 0.9599
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0239 - accuracy: 0.9621
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0418 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0271 - accuracy: 0.9573
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0272 - accuracy: 0.9560
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0423 - accuracy: 0.9250
375/375 [==============================] - 1s 2ms/step - loss: 0.0227 - accuracy: 0.9643
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0407 - accuracy: 0.9268
375/375 [==============================] - 1s 2ms/step - loss: 0.0263 - accuracy: 0.9568
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0405 - accuracy: 0.9283
375/375 [==============================] - 1s 2ms/step - loss: 0.0219 - accuracy: 0.9612
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0409 - accuracy: 0.9275
375/375 [==============================] - 1s 2ms/step - loss: 0.0227 - accuracy: 0.9615
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9284
375/375 [==============================] - 1s 2ms/step - loss: 0.0241 - accuracy: 0.9573
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0407 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0184 - accuracy: 0.9672
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0422 - accuracy: 0.9251
375/375 [==============================] - 1s 2ms/step - loss: 0.0253 - accuracy: 0.9614
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0417 - accuracy: 0.9264
375/375 [==============================] - 1s 2ms/step - loss: 0.0238 - accuracy: 0.9621
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0418 - accuracy: 0.9266
375/375 [==============================] - 1s 2ms/step - loss: 0.0261 - accuracy: 0.9584
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0412 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0331 - accuracy: 0.9478
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0428 - accuracy: 0.9250
375/375 [==============================] - 1s 2ms/step - loss: 0.0223 - accuracy: 0.9663
1500/1500 [==============================] - 3s 2ms/step - loss: 0.0412 - accuracy: 0.9258
375/375 [==============================] - 1s 2ms/step - loss: 0.0211 - accuracy: 0.9632
1500/1500 [==============================] - 3s 2ms/step - loss: 0.0411 - accuracy: 0.9243
375/375 [==============================] - 1s 2ms/step - loss: 0.0281 - accuracy: 0.9517
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0417 - accuracy: 0.9268
375/375 [==============================] - 1s 2ms/step - loss: 0.0311 - accuracy: 0.9483
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0412 - accuracy: 0.9266
375/375 [==============================] - 1s 2ms/step - loss: 0.0247 - accuracy: 0.9566
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0421 - accuracy: 0.9253
375/375 [==============================] - 1s 2ms/step - loss: 0.0255 - accuracy: 0.9543
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0370 - accuracy: 0.9340
total time: 163.10951614379883

## The Best Model
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

## Print all experiment results
import pandas as pd
results = pd.DataFrame(grid_result.cv_results_)
results

Best: 0.961483 using {'init': 'glorot_uniform', 'optimizer': 'adam'}

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_init	param_optimizer	params	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	4.962227	0.534804	0.789589	0.021689	glorot_uniform	rmsprop	{'init': 'glorot_uniform', 'optimizer': 'rmspr...	0.959333	0.965083	0.957750	0.956667	0.963333	0.960433	0.003244	3
1	4.087789	0.024603	0.798292	0.013339	glorot_uniform	adam	{'init': 'glorot_uniform', 'optimizer': 'adam'}	0.962583	0.960417	0.959250	0.962417	0.962750	0.961483	0.001401	1
2	4.723551	0.063123	0.842619	0.107239	normal	rmsprop	{'init': 'normal', 'optimizer': 'rmsprop'}	0.959917	0.962083	0.957333	0.956000	0.964333	0.959933	0.003040	4
3	4.174438	0.124374	0.784431	0.015665	normal	adam	{'init': 'normal', 'optimizer': 'adam'}	0.956833	0.961167	0.961500	0.957333	0.967167	0.960800	0.003712	2
4	4.698246	0.033024	0.798633	0.015098	uniform	rmsprop	{'init': 'uniform', 'optimizer': 'rmsprop'}	0.961417	0.962083	0.958417	0.947833	0.966250	0.959200	0.006209	5
5	4.086391	0.034679	0.859442	0.142505	uniform	adam	{'init': 'uniform', 'optimizer': 'adam'}	0.963250	0.951667	0.948333	0.956583	0.954333	0.954833	0.005029	6

K-fold Validation#

Similar to shuffle split?

np.random.seed(7)
seed=12

def create_model():
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
    model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or "overfitting" the training data
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10))
    model.add(Activation('softmax')) # This special "softmax" a
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) 
    return model

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
start=time()

model2 = KerasClassifier(build_fn=create_model, epochs=10, batch_size=1500,verbose=1)
results = cross_val_score(model2, X_train, Y_train, cv=10)

print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

print("total time:",time()-start)

Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1047 - accuracy: 0.7911
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0458 - accuracy: 0.9188
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0334 - accuracy: 0.9411
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9559
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0207 - accuracy: 0.9641
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9712
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9795
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0108 - accuracy: 0.9817
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0095 - accuracy: 0.9835
4/4 [==============================] - 0s 4ms/step - loss: 0.0121 - accuracy: 0.9782
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1044 - accuracy: 0.7950
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0435 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0319 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9552
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0196 - accuracy: 0.9662
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0161 - accuracy: 0.9725
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0100 - accuracy: 0.9825
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9851
4/4 [==============================] - 0s 3ms/step - loss: 0.0159 - accuracy: 0.9748
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1058 - accuracy: 0.7887
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0450 - accuracy: 0.9204
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0318 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9827
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9859
4/4 [==============================] - 0s 3ms/step - loss: 0.0184 - accuracy: 0.9685
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1028 - accuracy: 0.7998
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9203
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0243 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9660
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9715
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9757
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0121 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0097 - accuracy: 0.9830
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9846
4/4 [==============================] - 0s 3ms/step - loss: 0.0130 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1050 - accuracy: 0.7889
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0456 - accuracy: 0.9189
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0325 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9563
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9643
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0165 - accuracy: 0.9717
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9763
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9829
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9852
4/4 [==============================] - 0s 3ms/step - loss: 0.0144 - accuracy: 0.9750
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1059 - accuracy: 0.7875
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0451 - accuracy: 0.9212
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9446
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0246 - accuracy: 0.9583
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9750
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0122 - accuracy: 0.9793
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9819
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9855
4/4 [==============================] - 0s 4ms/step - loss: 0.0154 - accuracy: 0.9740
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1047 - accuracy: 0.7929
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9433
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0247 - accuracy: 0.9576
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0175 - accuracy: 0.9700
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9800
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0105 - accuracy: 0.9820
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9853
4/4 [==============================] - 0s 4ms/step - loss: 0.0143 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1046 - accuracy: 0.7949
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9182
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9426
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9556
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0167 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0085 - accuracy: 0.9856
4/4 [==============================] - 0s 4ms/step - loss: 0.0181 - accuracy: 0.9720
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1042 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0440 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0324 - accuracy: 0.9429
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0248 - accuracy: 0.9578
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0166 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0138 - accuracy: 0.9764
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9820
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0081 - accuracy: 0.9863
4/4 [==============================] - 0s 3ms/step - loss: 0.0180 - accuracy: 0.9710
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1045 - accuracy: 0.7953
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0455 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9438
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0175 - accuracy: 0.9697
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9785
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0114 - accuracy: 0.9810
K-fold Results: 97.49% (0.36%)
total time: 29.779260635375977

Shuffle Split Cross Validation#

Shuffle but not stratified

## Shuffle Split
start=time()
sf = ShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=sf)
print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)

Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7965
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0454 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0335 - accuracy: 0.9410
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0169 - accuracy: 0.9713
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0126 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9839
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0092 - accuracy: 0.9839
4/4 [==============================] - 0s 3ms/step - loss: 0.0129 - accuracy: 0.9773
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7952
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9183
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0261 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0208 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9723
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9791
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0101 - accuracy: 0.9835
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9850
4/4 [==============================] - 0s 3ms/step - loss: 0.0124 - accuracy: 0.9788
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1057 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0458 - accuracy: 0.9200
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9431
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0169 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9839
4/4 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7970
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0442 - accuracy: 0.9224
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0114 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0103 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0084 - accuracy: 0.9858
4/4 [==============================] - 0s 4ms/step - loss: 0.0167 - accuracy: 0.9733
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1043 - accuracy: 0.7938
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0331 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9781
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0098 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0087 - accuracy: 0.9851
4/4 [==============================] - 0s 4ms/step - loss: 0.0168 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1014 - accuracy: 0.8043
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0443 - accuracy: 0.9218
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0323 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0247 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9724
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9759
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0113 - accuracy: 0.9806
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9821
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9860
4/4 [==============================] - 0s 4ms/step - loss: 0.0125 - accuracy: 0.9800
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1080 - accuracy: 0.7908
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0467 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0332 - accuracy: 0.9424
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9638
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9702
Epoch 7/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0141 - accuracy: 0.9760
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0120 - accuracy: 0.9789
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0101 - accuracy: 0.9831
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0091 - accuracy: 0.9847
4/4 [==============================] - 0s 4ms/step - loss: 0.0120 - accuracy: 0.9785
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1038 - accuracy: 0.7945
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0448 - accuracy: 0.9215
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0204 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9718
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9768
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0155 - accuracy: 0.9747
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1076 - accuracy: 0.7903
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9422
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9565
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0171 - accuracy: 0.9701
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0090 - accuracy: 0.9845
4/4 [==============================] - 0s 4ms/step - loss: 0.0137 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1045 - accuracy: 0.7910
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0454 - accuracy: 0.9210
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0324 - accuracy: 0.9441
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0165 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0144 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9804
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9832
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9848
4/4 [==============================] - 0s 3ms/step - loss: 0.0127 - accuracy: 0.9787
K-fold Results: 97.72% (0.20%)
total time: 31.2500159740448

Stratified K-Fold Validation#

Stratified but not shuffled
Stratified cross-validation gives poor results. No idea why?

print(Y_train[:4,])
print(y_train[:4])
from collections import Counter

Counter(y_train)

[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]
[5 0 4 1]

Counter({0: 5923,
         1: 6742,
         2: 5958,
         3: 6131,
         4: 5842,
         5: 5421,
         6: 5918,
         7: 6265,
         8: 5851,
         9: 5949})

## Stratified Shuffle
from sklearn.model_selection import StratifiedKFold
start=time()
sk = StratifiedKFold(n_splits=10)
results = cross_val_score(model2, X_train, y_train, cv=sk)
print("Stratified K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)

Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0234 - accuracy: 0.0974
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8925 - accuracy: 0.1069
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0986
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1016
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0997
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1024
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
4/4 [==============================] - 0s 3ms/step - loss: 9.8922 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0317 - accuracy: 0.0952
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8930 - accuracy: 0.1012
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1006
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0984
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0981
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1014
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0076 - accuracy: 0.0989
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8922 - accuracy: 0.1001
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0968
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1012
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0975
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0995
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1021
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1018
4/4 [==============================] - 0s 3ms/step - loss: 9.8903 - accuracy: 0.0993
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0352 - accuracy: 0.1036
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8921 - accuracy: 0.1104
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1044
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1035
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0994
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1004
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1002
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1022
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.1022
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0332 - accuracy: 0.0993
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8924 - accuracy: 0.1021
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0984
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0978
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0956
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1037
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1028
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1009
4/4 [==============================] - 0s 4ms/step - loss: 9.8922 - accuracy: 0.0992
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0069 - accuracy: 0.0996
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0977
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0980
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0998
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1025
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1010
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
4/4 [==============================] - 0s 3ms/step - loss: 9.8925 - accuracy: 0.1023
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0232 - accuracy: 0.0983
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0992
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0981
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0989
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0995
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0973
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0990
4/4 [==============================] - 0s 4ms/step - loss: 9.8925 - accuracy: 0.0975
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0450 - accuracy: 0.0947
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8924 - accuracy: 0.0943
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0983
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0987
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0977
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1024
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
4/4 [==============================] - 0s 3ms/step - loss: 9.8900 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0163 - accuracy: 0.1055
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.0834
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0926
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0991
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1032
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1022
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
4/4 [==============================] - 0s 4ms/step - loss: 9.8929 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0165 - accuracy: 0.1039
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8919 - accuracy: 0.0942
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0979
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0997
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1028
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1008
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1046
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0997
4/4 [==============================] - 0s 4ms/step - loss: 9.8933 - accuracy: 0.1022
Stratified K-fold Results: 9.97% (0.17%)
total time: 30.07964015007019

Stratified Shuffle Split#

A mix of StratifiedKFold and ShuffleSplit, which returns stratified randomized folds
The folds are made by preserving the percentage of samples for each class.

## Stratified Shuffle
from sklearn.model_selection import StratifiedShuffleSplit
start=time()
ss = StratifiedShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=ss)
print("Stratified Shuffle Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)

Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1072 - accuracy: 0.7853
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0341 - accuracy: 0.9406
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0260 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0209 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0143 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9840
4/4 [==============================] - 0s 3ms/step - loss: 0.0149 - accuracy: 0.9753
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1032 - accuracy: 0.7954
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9205
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9573
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0142 - accuracy: 0.9754
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0118 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0099 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9856
4/4 [==============================] - 0s 3ms/step - loss: 0.0151 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1054 - accuracy: 0.7976
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0474 - accuracy: 0.9161
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9434
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0253 - accuracy: 0.9564
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9706
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0136 - accuracy: 0.9767
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0124 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0105 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9745
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1044 - accuracy: 0.7927
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0452 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0320 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0250 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9710
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9762
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0167 - accuracy: 0.9722
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1065 - accuracy: 0.7891
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0462 - accuracy: 0.9185
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0330 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0214 - accuracy: 0.9632
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0144 - accuracy: 0.9744
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9794
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0107 - accuracy: 0.9810
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0094 - accuracy: 0.9841
4/4 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9780
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1051 - accuracy: 0.7913
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0464 - accuracy: 0.9176
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9447
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0254 - accuracy: 0.9560
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0143 - accuracy: 0.9752
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0104 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1052 - accuracy: 0.7921
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0445 - accuracy: 0.9216
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0325 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0254 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0201 - accuracy: 0.9650
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9705
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0140 - accuracy: 0.9766
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0138 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1029 - accuracy: 0.7947
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9211
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0316 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0239 - accuracy: 0.9595
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0198 - accuracy: 0.9657
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0157 - accuracy: 0.9731
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0119 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9842
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9854
4/4 [==============================] - 0s 4ms/step - loss: 0.0127 - accuracy: 0.9777
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1030 - accuracy: 0.7933
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9201
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9572
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0193 - accuracy: 0.9659
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9704
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0134 - accuracy: 0.9769
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0096 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9857
4/4 [==============================] - 0s 4ms/step - loss: 0.0136 - accuracy: 0.9775
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1058 - accuracy: 0.7881
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0448 - accuracy: 0.9198
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9436
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0201 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0083 - accuracy: 0.9862
4/4 [==============================] - 0s 3ms/step - loss: 0.0119 - accuracy: 0.9795
Stratified Shuffle Results: 97.63% (0.20%)
total time: 30.383320093154907

GridSearchCV and Cross-Validation#

We perform the GridSearchCV to find the optimal optimizers
We determine the best parameter values based on stratified shuffled 5-fold cross validation

def create_model():
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
    model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or "overfitting" the training data
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10))
    model.add(Activation('softmax')) # This special "softmax" a
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) 
    return model
start=time()

## Define cross-validation method
ss = StratifiedShuffleSplit(n_splits=5)

## Define Model
model3 = KerasClassifier(build_fn=create_model)

## Define Hyper-parameter values
optimizers = ['rmsprop', 'adam']
param_grid = dict(optimizer=optimizers, init=init)

## Define GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=ss)
grid_result = grid.fit(X_train, Y_train)
print("total time:",time()-start)

1688/1688 [==============================] - 6s 4ms/step - loss: 0.0402 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0275 - accuracy: 0.9605
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0395 - accuracy: 0.9323
188/188 [==============================] - 0s 2ms/step - loss: 0.0300 - accuracy: 0.9485
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9315
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9605
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0400 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0249 - accuracy: 0.9627
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0400 - accuracy: 0.9309
188/188 [==============================] - 0s 2ms/step - loss: 0.0305 - accuracy: 0.9533
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0393 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0237 - accuracy: 0.9608
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0382 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0201 - accuracy: 0.9648
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0387 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0274 - accuracy: 0.9540
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0379 - accuracy: 0.9324
188/188 [==============================] - 0s 2ms/step - loss: 0.0256 - accuracy: 0.9577
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0389 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9633
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0405 - accuracy: 0.9282
188/188 [==============================] - 0s 2ms/step - loss: 0.0236 - accuracy: 0.9647
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0397 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0259 - accuracy: 0.9587
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0405 - accuracy: 0.9285
188/188 [==============================] - 0s 2ms/step - loss: 0.0219 - accuracy: 0.9645
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0311 - accuracy: 0.9557
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0212 - accuracy: 0.9642
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0392 - accuracy: 0.9298
188/188 [==============================] - 0s 2ms/step - loss: 0.0230 - accuracy: 0.9590
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0388 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9580
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0390 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0224 - accuracy: 0.9613
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9302
188/188 [==============================] - 0s 2ms/step - loss: 0.0263 - accuracy: 0.9573
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0221 - accuracy: 0.9638
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0408 - accuracy: 0.9281
188/188 [==============================] - 0s 2ms/step - loss: 0.0235 - accuracy: 0.9618
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0408 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0246 - accuracy: 0.9605
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0404 - accuracy: 0.9306
188/188 [==============================] - 0s 2ms/step - loss: 0.0271 - accuracy: 0.9593
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0399 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0231 - accuracy: 0.9638
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0403 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0213 - accuracy: 0.9653
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0395 - accuracy: 0.9291
188/188 [==============================] - 0s 2ms/step - loss: 0.0239 - accuracy: 0.9612
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0394 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9635
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0401 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0210 - accuracy: 0.9640
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0397 - accuracy: 0.9286
188/188 [==============================] - 0s 2ms/step - loss: 0.0225 - accuracy: 0.9600
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0405 - accuracy: 0.9280
188/188 [==============================] - 0s 2ms/step - loss: 0.0193 - accuracy: 0.9663
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0384 - accuracy: 0.9313
total time: 203.81905102729797

## The Best Model
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

## Print all experiment results
import pandas as pd
results = pd.DataFrame(grid_result.cv_results_)
results

Best: 0.963000 using {'init': 'uniform', 'optimizer': 'adam'}

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_init	param_optimizer	params	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	6.656565	0.121780	0.524209	0.008416	glorot_uniform	rmsprop	{'init': 'glorot_uniform', 'optimizer': 'rmspr...	0.960500	0.948500	0.960500	0.962667	0.953333	0.957100	0.005334	6
1	6.022944	0.100696	0.534721	0.013367	glorot_uniform	adam	{'init': 'glorot_uniform', 'optimizer': 'adam'}	0.960833	0.964833	0.954000	0.957667	0.963333	0.960133	0.003912	4
2	6.614549	0.093743	0.522084	0.007600	normal	rmsprop	{'init': 'normal', 'optimizer': 'rmsprop'}	0.964667	0.958667	0.964500	0.955667	0.964167	0.961533	0.003693	3
3	5.639868	0.232617	0.496856	0.013943	normal	adam	{'init': 'normal', 'optimizer': 'adam'}	0.959000	0.958000	0.961333	0.957333	0.963833	0.959900	0.002389	5
4	6.057838	0.092201	0.491186	0.010245	uniform	rmsprop	{'init': 'uniform', 'optimizer': 'rmsprop'}	0.961833	0.960500	0.959333	0.963833	0.965333	0.962167	0.002178	2
5	5.619616	0.151503	0.484414	0.004364	uniform	adam	{'init': 'uniform', 'optimizer': 'adam'}	0.961167	0.963500	0.964000	0.960000	0.966333	0.963000	0.002224	1

References#

Requirements#

# %run ./get_modules.py
import pkg_resources
import types
def get_imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            # Split ensures you get root package, 
            # not just imported function
            name = val.__name__.split(".")[0]

        elif isinstance(val, type):
            name = val.__module__.split(".")[0]

        # Some packages are weird and have different
        # imported names vs. system/pip names. Unfortunately,
        # there is no systematic way to get pip names from
        # a package's imported name. You'll have to add
        # exceptions to this list manually!
        poorly_named_packages = {
            "PIL": "Pillow",
            "sklearn": "scikit-learn"
        }
        if name in poorly_named_packages.keys():
            name = poorly_named_packages[name]

        yield name
        
        
imports = list(set(get_imports()))

# The only way I found to get the version of the root package
# from only the name of the package is to cross-check the names 
# of installed packages vs. imported packages
requirements = []
for m in pkg_resources.working_set:
    if m.project_name in imports and m.project_name!="pip":
        requirements.append((m.project_name, m.version))

for r in requirements:
    print("{}=={}".format(*r))

tensorflow==2.3.0
scikit-learn==0.22.2.post1
pandas==1.1.2
numpy==1.18.5