Hyper-Parameter Tuning#

There are two important techniques to fine-tune the hyperparameters of the model: Grid Search and Cross Validation.

  • Grid Search

    • Define a few parameter values and experiment all these values in modeling. Use sklearn.model_selection.GridSearchCV to find the best parameter settings.

  • Cross Validation

    • Fine-tune the parameters using cross-validation. Common CV methods include sklearn.model_selection.StratifiedKFold, sklearn_model_selection.ShuffleSplit, LeaveOneOut.

Both mothods can make use of the pipeline in sklearn to streamline the processing of training and validation.

SVM Model#

  • This example is from the officient sklearn documentation

  • A classic SVM model with train and test split

## Normal SVM model
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm

X, y = datasets.load_iris(return_X_y=True)
X.shape, y.shape

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=0)

X_train.shape, y_train.shape

X_test.shape, y_test.shape

## fit on train sets
clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
## test on test sets
clf.score(X_test, y_test)
0.9666666666666667

Default K-fold Cross Validation#

  • We can easily use the cross_val_score() to do the K-fold cross validation for a specific training model (without shuffling)

Note

By default, train_test_split returns a random split.

## Cross-validation

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, X, y, cv=10, scoring='f1_macro')
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
Accuracy: 0.97 (+/- 0.09)

GridSearchCV and Fine-Tuning Hyper-Parameters#

  • Important steps:

    • Define SVM classifier

    • Define a set of parameter values to experiment

    • Use GridSearchCV to find the best parameter settings

    • GridSearchCV by default implements cross-validation methods to find the optimal parameter settings.

    • In other words, we can specify our own more sophisticated CV methods in GridSearchCV.

from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters, cv=5) # 
clf.fit(iris.data, iris.target)


sorted(clf.cv_results_.keys())

#print(clf.cv_results_)

import pandas as pd

cv_results_df=pd.DataFrame(clf.cv_results_)
cv_results_df
mean_fit_time std_fit_time mean_score_time std_score_time param_C param_kernel params split0_test_score split1_test_score split2_test_score split3_test_score split4_test_score mean_test_score std_test_score rank_test_score
0 0.000737 0.000221 0.000469 0.000225 1 linear {'C': 1, 'kernel': 'linear'} 0.966667 1.000000 0.966667 0.966667 1.0 0.980000 0.016330 1
1 0.000651 0.000035 0.000331 0.000015 1 rbf {'C': 1, 'kernel': 'rbf'} 0.966667 0.966667 0.966667 0.933333 1.0 0.966667 0.021082 4
2 0.000511 0.000027 0.000276 0.000013 10 linear {'C': 10, 'kernel': 'linear'} 1.000000 1.000000 0.900000 0.966667 1.0 0.973333 0.038873 3
3 0.000604 0.000047 0.000344 0.000070 10 rbf {'C': 10, 'kernel': 'rbf'} 0.966667 1.000000 0.966667 0.966667 1.0 0.980000 0.016330 1

Cross-validation, Hyper-Parameter Tuning, and Pipeline#

  • Common cross validation methods:

    • StratifiedKFold: Split data into train and validation sets by preserving the percentage of samples of each class

    • ShuffleSplit: Split data into train and validation sets by first shuffling the data and then splitting

    • StratifiedShuffleSplit: Stratified + Shuffled

    • LeaveOneOut: Creating train sets by taking all samples execept one, which is left out for validation

  • Important Steps

    • Define all preprocessing methods as Tranasformer

    • Create a pipeline

    • Define a cross-validation method

    • Use cross_val_score() to fine the best parameter settings by feeding the pipeline and the CV method

from sklearn.pipeline import make_pipeline
from sklearn import preprocessing
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import ShuffleSplit

## Create pipeline
clf = make_pipeline(preprocessing.StandardScaler(), svm.SVC(C=1))

## Define CV methods
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
shuffsplit = ShuffleSplit(n_splits=5, test_size=0.1, random_state=7)

## CV results
print('K-fold CV:', cross_val_score(clf, X, y, cv=kfold))
print('Shuffle CV:', cross_val_score(clf, X, y, cv=shuffsplit))
K-fold CV: [0.96666667 0.96666667 0.96666667 0.9        1.        ]
Shuffle CV: [0.86666667 1.         1.         1.         0.93333333]

Deep Learning Example#

MNIST Dataset#

import numpy as np
import os
from keras.datasets import mnist
from keras.layers import *
from keras.models import *
from time import time
nb_classes = 10
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 784)
X_test = X_test.reshape(X_test.shape[0], 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
## Convert y labels (integers) into binary class matrix
from keras.utils import np_utils
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
print(Y_train[:5,])
[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]

K-fold Validation#

  • Similar to shuffle split?

np.random.seed(7)
seed=12
def create_model():
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
    model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or "overfitting" the training data
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10))
    model.add(Activation('softmax')) # This special "softmax" a
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) 
    return model

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
start=time()

model2 = KerasClassifier(build_fn=create_model, epochs=10, batch_size=1500,verbose=1)
results = cross_val_score(model2, X_train, Y_train, cv=10)

print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1047 - accuracy: 0.7911
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0458 - accuracy: 0.9188
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0334 - accuracy: 0.9411
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9559
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0207 - accuracy: 0.9641
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9712
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9795
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0108 - accuracy: 0.9817
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0095 - accuracy: 0.9835
4/4 [==============================] - 0s 4ms/step - loss: 0.0121 - accuracy: 0.9782
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1044 - accuracy: 0.7950
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0435 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0319 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9552
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0196 - accuracy: 0.9662
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0161 - accuracy: 0.9725
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0100 - accuracy: 0.9825
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9851
4/4 [==============================] - 0s 3ms/step - loss: 0.0159 - accuracy: 0.9748
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1058 - accuracy: 0.7887
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0450 - accuracy: 0.9204
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0318 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9827
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9859
4/4 [==============================] - 0s 3ms/step - loss: 0.0184 - accuracy: 0.9685
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1028 - accuracy: 0.7998
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9203
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0243 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9660
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9715
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9757
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0121 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0097 - accuracy: 0.9830
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9846
4/4 [==============================] - 0s 3ms/step - loss: 0.0130 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1050 - accuracy: 0.7889
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0456 - accuracy: 0.9189
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0325 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9563
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9643
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0165 - accuracy: 0.9717
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9763
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9829
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9852
4/4 [==============================] - 0s 3ms/step - loss: 0.0144 - accuracy: 0.9750
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1059 - accuracy: 0.7875
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0451 - accuracy: 0.9212
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9446
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0246 - accuracy: 0.9583
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9750
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0122 - accuracy: 0.9793
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9819
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9855
4/4 [==============================] - 0s 4ms/step - loss: 0.0154 - accuracy: 0.9740
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1047 - accuracy: 0.7929
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9433
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0247 - accuracy: 0.9576
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0175 - accuracy: 0.9700
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9800
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0105 - accuracy: 0.9820
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9853
4/4 [==============================] - 0s 4ms/step - loss: 0.0143 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1046 - accuracy: 0.7949
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9182
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9426
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9556
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0167 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0085 - accuracy: 0.9856
4/4 [==============================] - 0s 4ms/step - loss: 0.0181 - accuracy: 0.9720
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1042 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0440 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0324 - accuracy: 0.9429
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0248 - accuracy: 0.9578
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0166 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0138 - accuracy: 0.9764
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9820
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0081 - accuracy: 0.9863
4/4 [==============================] - 0s 3ms/step - loss: 0.0180 - accuracy: 0.9710
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1045 - accuracy: 0.7953
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0455 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9438
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0175 - accuracy: 0.9697
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9785
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0114 - accuracy: 0.9810
K-fold Results: 97.49% (0.36%)
total time: 29.779260635375977

Shuffle Split Cross Validation#

  • Shuffle but not stratified

## Shuffle Split
start=time()
sf = ShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=sf)
print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7965
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0454 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0335 - accuracy: 0.9410
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0169 - accuracy: 0.9713
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0126 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9839
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0092 - accuracy: 0.9839
4/4 [==============================] - 0s 3ms/step - loss: 0.0129 - accuracy: 0.9773
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7952
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9183
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0261 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0208 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9723
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9791
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0101 - accuracy: 0.9835
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9850
4/4 [==============================] - 0s 3ms/step - loss: 0.0124 - accuracy: 0.9788
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1057 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0458 - accuracy: 0.9200
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9431
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0169 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9839
4/4 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7970
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0442 - accuracy: 0.9224
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0114 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0103 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0084 - accuracy: 0.9858
4/4 [==============================] - 0s 4ms/step - loss: 0.0167 - accuracy: 0.9733
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1043 - accuracy: 0.7938
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0331 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9781
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0098 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0087 - accuracy: 0.9851
4/4 [==============================] - 0s 4ms/step - loss: 0.0168 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1014 - accuracy: 0.8043
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0443 - accuracy: 0.9218
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0323 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0247 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9724
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9759
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0113 - accuracy: 0.9806
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9821
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9860
4/4 [==============================] - 0s 4ms/step - loss: 0.0125 - accuracy: 0.9800
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1080 - accuracy: 0.7908
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0467 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0332 - accuracy: 0.9424
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9638
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9702
Epoch 7/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0141 - accuracy: 0.9760
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0120 - accuracy: 0.9789
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0101 - accuracy: 0.9831
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0091 - accuracy: 0.9847
4/4 [==============================] - 0s 4ms/step - loss: 0.0120 - accuracy: 0.9785
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1038 - accuracy: 0.7945
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0448 - accuracy: 0.9215
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0204 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9718
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9768
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0155 - accuracy: 0.9747
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1076 - accuracy: 0.7903
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9422
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9565
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0171 - accuracy: 0.9701
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0090 - accuracy: 0.9845
4/4 [==============================] - 0s 4ms/step - loss: 0.0137 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1045 - accuracy: 0.7910
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0454 - accuracy: 0.9210
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0324 - accuracy: 0.9441
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0165 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0144 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9804
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9832
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9848
4/4 [==============================] - 0s 3ms/step - loss: 0.0127 - accuracy: 0.9787
K-fold Results: 97.72% (0.20%)
total time: 31.2500159740448

Stratified K-Fold Validation#

  • Stratified but not shuffled

  • Stratified cross-validation gives poor results. No idea why?

print(Y_train[:4,])
print(y_train[:4])
from collections import Counter

Counter(y_train)
[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]
[5 0 4 1]
Counter({0: 5923,
         1: 6742,
         2: 5958,
         3: 6131,
         4: 5842,
         5: 5421,
         6: 5918,
         7: 6265,
         8: 5851,
         9: 5949})
## Stratified Shuffle
from sklearn.model_selection import StratifiedKFold
start=time()
sk = StratifiedKFold(n_splits=10)
results = cross_val_score(model2, X_train, y_train, cv=sk)
print("Stratified K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0234 - accuracy: 0.0974
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8925 - accuracy: 0.1069
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0986
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1016
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0997
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1024
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
4/4 [==============================] - 0s 3ms/step - loss: 9.8922 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0317 - accuracy: 0.0952
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8930 - accuracy: 0.1012
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1006
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0984
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0981
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1014
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0076 - accuracy: 0.0989
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8922 - accuracy: 0.1001
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0968
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1012
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0975
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0995
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1021
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1018
4/4 [==============================] - 0s 3ms/step - loss: 9.8903 - accuracy: 0.0993
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0352 - accuracy: 0.1036
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8921 - accuracy: 0.1104
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1044
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1035
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0994
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1004
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1002
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1022
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.1022
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0332 - accuracy: 0.0993
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8924 - accuracy: 0.1021
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0984
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0978
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0956
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1037
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1028
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1009
4/4 [==============================] - 0s 4ms/step - loss: 9.8922 - accuracy: 0.0992
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0069 - accuracy: 0.0996
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0977
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0980
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0998
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1025
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1010
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
4/4 [==============================] - 0s 3ms/step - loss: 9.8925 - accuracy: 0.1023
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0232 - accuracy: 0.0983
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0992
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0981
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0989
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0995
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0973
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0990
4/4 [==============================] - 0s 4ms/step - loss: 9.8925 - accuracy: 0.0975
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0450 - accuracy: 0.0947
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8924 - accuracy: 0.0943
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0983
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0987
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0977
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1024
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
4/4 [==============================] - 0s 3ms/step - loss: 9.8900 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0163 - accuracy: 0.1055
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.0834
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0926
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0991
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1032
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1022
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
4/4 [==============================] - 0s 4ms/step - loss: 9.8929 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0165 - accuracy: 0.1039
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8919 - accuracy: 0.0942
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0979
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0997
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1028
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1008
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1046
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0997
4/4 [==============================] - 0s 4ms/step - loss: 9.8933 - accuracy: 0.1022
Stratified K-fold Results: 9.97% (0.17%)
total time: 30.07964015007019

Stratified Shuffle Split#

  • A mix of StratifiedKFold and ShuffleSplit, which returns stratified randomized folds

  • The folds are made by preserving the percentage of samples for each class.

## Stratified Shuffle
from sklearn.model_selection import StratifiedShuffleSplit
start=time()
ss = StratifiedShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=ss)
print("Stratified Shuffle Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1072 - accuracy: 0.7853
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0341 - accuracy: 0.9406
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0260 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0209 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0143 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9840
4/4 [==============================] - 0s 3ms/step - loss: 0.0149 - accuracy: 0.9753
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1032 - accuracy: 0.7954
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9205
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9573
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0142 - accuracy: 0.9754
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0118 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0099 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9856
4/4 [==============================] - 0s 3ms/step - loss: 0.0151 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1054 - accuracy: 0.7976
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0474 - accuracy: 0.9161
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9434
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0253 - accuracy: 0.9564
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9706
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0136 - accuracy: 0.9767
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0124 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0105 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9745
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1044 - accuracy: 0.7927
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0452 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0320 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0250 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9710
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9762
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0167 - accuracy: 0.9722
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1065 - accuracy: 0.7891
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0462 - accuracy: 0.9185
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0330 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0214 - accuracy: 0.9632
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0144 - accuracy: 0.9744
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9794
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0107 - accuracy: 0.9810
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0094 - accuracy: 0.9841
4/4 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9780
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1051 - accuracy: 0.7913
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0464 - accuracy: 0.9176
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9447
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0254 - accuracy: 0.9560
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0143 - accuracy: 0.9752
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0104 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1052 - accuracy: 0.7921
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0445 - accuracy: 0.9216
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0325 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0254 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0201 - accuracy: 0.9650
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9705
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0140 - accuracy: 0.9766
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0138 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1029 - accuracy: 0.7947
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9211
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0316 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0239 - accuracy: 0.9595
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0198 - accuracy: 0.9657
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0157 - accuracy: 0.9731
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0119 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9842
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9854
4/4 [==============================] - 0s 4ms/step - loss: 0.0127 - accuracy: 0.9777
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1030 - accuracy: 0.7933
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9201
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9572
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0193 - accuracy: 0.9659
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9704
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0134 - accuracy: 0.9769
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0096 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9857
4/4 [==============================] - 0s 4ms/step - loss: 0.0136 - accuracy: 0.9775
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1058 - accuracy: 0.7881
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0448 - accuracy: 0.9198
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9436
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0201 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0083 - accuracy: 0.9862
4/4 [==============================] - 0s 3ms/step - loss: 0.0119 - accuracy: 0.9795
Stratified Shuffle Results: 97.63% (0.20%)
total time: 30.383320093154907

GridSearchCV and Cross-Validation#

  • We perform the GridSearchCV to find the optimal optimizers

  • We determine the best parameter values based on stratified shuffled 5-fold cross validation

def create_model():
    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
    model.add(Dropout(0.2))   # Dropout helps protect the model from memorizing or "overfitting" the training data
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.2))
    model.add(Dense(10))
    model.add(Activation('softmax')) # This special "softmax" a
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy']) 
    return model
start=time()

## Define cross-validation method
ss = StratifiedShuffleSplit(n_splits=5)

## Define Model
model3 = KerasClassifier(build_fn=create_model)

## Define Hyper-parameter values
optimizers = ['rmsprop', 'adam']
param_grid = dict(optimizer=optimizers, init=init)

## Define GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=ss)
grid_result = grid.fit(X_train, Y_train)
print("total time:",time()-start)
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0402 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0275 - accuracy: 0.9605
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0395 - accuracy: 0.9323
188/188 [==============================] - 0s 2ms/step - loss: 0.0300 - accuracy: 0.9485
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9315
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9605
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0400 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0249 - accuracy: 0.9627
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0400 - accuracy: 0.9309
188/188 [==============================] - 0s 2ms/step - loss: 0.0305 - accuracy: 0.9533
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0393 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0237 - accuracy: 0.9608
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0382 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0201 - accuracy: 0.9648
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0387 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0274 - accuracy: 0.9540
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0379 - accuracy: 0.9324
188/188 [==============================] - 0s 2ms/step - loss: 0.0256 - accuracy: 0.9577
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0389 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9633
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0405 - accuracy: 0.9282
188/188 [==============================] - 0s 2ms/step - loss: 0.0236 - accuracy: 0.9647
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0397 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0259 - accuracy: 0.9587
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0405 - accuracy: 0.9285
188/188 [==============================] - 0s 2ms/step - loss: 0.0219 - accuracy: 0.9645
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0311 - accuracy: 0.9557
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0212 - accuracy: 0.9642
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0392 - accuracy: 0.9298
188/188 [==============================] - 0s 2ms/step - loss: 0.0230 - accuracy: 0.9590
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0388 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9580
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0390 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0224 - accuracy: 0.9613
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9302
188/188 [==============================] - 0s 2ms/step - loss: 0.0263 - accuracy: 0.9573
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0221 - accuracy: 0.9638
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0408 - accuracy: 0.9281
188/188 [==============================] - 0s 2ms/step - loss: 0.0235 - accuracy: 0.9618
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0408 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0246 - accuracy: 0.9605
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0404 - accuracy: 0.9306
188/188 [==============================] - 0s 2ms/step - loss: 0.0271 - accuracy: 0.9593
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0399 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0231 - accuracy: 0.9638
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0403 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0213 - accuracy: 0.9653
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0395 - accuracy: 0.9291
188/188 [==============================] - 0s 2ms/step - loss: 0.0239 - accuracy: 0.9612
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0394 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9635
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0401 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0210 - accuracy: 0.9640
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0397 - accuracy: 0.9286
188/188 [==============================] - 0s 2ms/step - loss: 0.0225 - accuracy: 0.9600
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0405 - accuracy: 0.9280
188/188 [==============================] - 0s 2ms/step - loss: 0.0193 - accuracy: 0.9663
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0384 - accuracy: 0.9313
total time: 203.81905102729797
## The Best Model
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

## Print all experiment results
import pandas as pd
results = pd.DataFrame(grid_result.cv_results_)
results
Best: 0.963000 using {'init': 'uniform', 'optimizer': 'adam'}
mean_fit_time std_fit_time mean_score_time std_score_time param_init param_optimizer params split0_test_score split1_test_score split2_test_score split3_test_score split4_test_score mean_test_score std_test_score rank_test_score
0 6.656565 0.121780 0.524209 0.008416 glorot_uniform rmsprop {'init': 'glorot_uniform', 'optimizer': 'rmspr... 0.960500 0.948500 0.960500 0.962667 0.953333 0.957100 0.005334 6
1 6.022944 0.100696 0.534721 0.013367 glorot_uniform adam {'init': 'glorot_uniform', 'optimizer': 'adam'} 0.960833 0.964833 0.954000 0.957667 0.963333 0.960133 0.003912 4
2 6.614549 0.093743 0.522084 0.007600 normal rmsprop {'init': 'normal', 'optimizer': 'rmsprop'} 0.964667 0.958667 0.964500 0.955667 0.964167 0.961533 0.003693 3
3 5.639868 0.232617 0.496856 0.013943 normal adam {'init': 'normal', 'optimizer': 'adam'} 0.959000 0.958000 0.961333 0.957333 0.963833 0.959900 0.002389 5
4 6.057838 0.092201 0.491186 0.010245 uniform rmsprop {'init': 'uniform', 'optimizer': 'rmsprop'} 0.961833 0.960500 0.959333 0.963833 0.965333 0.962167 0.002178 2
5 5.619616 0.151503 0.484414 0.004364 uniform adam {'init': 'uniform', 'optimizer': 'adam'} 0.961167 0.963500 0.964000 0.960000 0.966333 0.963000 0.002224 1

References#

Requirements#

# %run ./get_modules.py
import pkg_resources
import types
def get_imports():
    for name, val in globals().items():
        if isinstance(val, types.ModuleType):
            # Split ensures you get root package, 
            # not just imported function
            name = val.__name__.split(".")[0]

        elif isinstance(val, type):
            name = val.__module__.split(".")[0]

        # Some packages are weird and have different
        # imported names vs. system/pip names. Unfortunately,
        # there is no systematic way to get pip names from
        # a package's imported name. You'll have to add
        # exceptions to this list manually!
        poorly_named_packages = {
            "PIL": "Pillow",
            "sklearn": "scikit-learn"
        }
        if name in poorly_named_packages.keys():
            name = poorly_named_packages[name]

        yield name
        
        
imports = list(set(get_imports()))

# The only way I found to get the version of the root package
# from only the name of the package is to cross-check the names 
# of installed packages vs. imported packages
requirements = []
for m in pkg_resources.working_set:
    if m.project_name in imports and m.project_name!="pip":
        requirements.append((m.project_name, m.version))

for r in requirements:
    print("{}=={}".format(*r))
tensorflow==2.3.0
scikit-learn==0.22.2.post1
pandas==1.1.2
numpy==1.18.5