Hyper-Parameter Tuning#
There are two important techniques to fine-tune the hyperparameters of the model: Grid Search and Cross Validation.
Grid Search
Define a few parameter values and experiment all these values in modeling. Use
sklearn.model_selection.GridSearchCV
to find the best parameter settings.
Cross Validation
Fine-tune the parameters using cross-validation. Common CV methods include
sklearn.model_selection.StratifiedKFold
,sklearn_model_selection.ShuffleSplit
,LeaveOneOut
.
Both mothods can make use of the pipeline
in sklearn to streamline the processing of training and validation.
SVM Model#
This example is from the officient
sklearn
documentationA classic SVM model with train and test split
## Normal SVM model
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import datasets
from sklearn import svm
X, y = datasets.load_iris(return_X_y=True)
X.shape, y.shape
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.4, random_state=0)
X_train.shape, y_train.shape
X_test.shape, y_test.shape
## fit on train sets
clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
## test on test sets
clf.score(X_test, y_test)
0.9666666666666667
Default K-fold Cross Validation#
We can easily use the
cross_val_score()
to do the K-fold cross validation for a specific training model (without shuffling)
Note
By default, train_test_split
returns a random split.
## Cross-validation
from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, X, y, cv=10, scoring='f1_macro')
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
Accuracy: 0.97 (+/- 0.09)
GridSearchCV and Fine-Tuning Hyper-Parameters#
Important steps:
Define SVM classifier
Define a set of parameter values to experiment
Use
GridSearchCV
to find the best parameter settingsGridSearchCV
by default implements cross-validation methods to find the optimal parameter settings.In other words, we can specify our own more sophisticated CV methods in
GridSearchCV
.
from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters, cv=5) #
clf.fit(iris.data, iris.target)
sorted(clf.cv_results_.keys())
#print(clf.cv_results_)
import pandas as pd
cv_results_df=pd.DataFrame(clf.cv_results_)
cv_results_df
mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_C | param_kernel | params | split0_test_score | split1_test_score | split2_test_score | split3_test_score | split4_test_score | mean_test_score | std_test_score | rank_test_score | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.000737 | 0.000221 | 0.000469 | 0.000225 | 1 | linear | {'C': 1, 'kernel': 'linear'} | 0.966667 | 1.000000 | 0.966667 | 0.966667 | 1.0 | 0.980000 | 0.016330 | 1 |
1 | 0.000651 | 0.000035 | 0.000331 | 0.000015 | 1 | rbf | {'C': 1, 'kernel': 'rbf'} | 0.966667 | 0.966667 | 0.966667 | 0.933333 | 1.0 | 0.966667 | 0.021082 | 4 |
2 | 0.000511 | 0.000027 | 0.000276 | 0.000013 | 10 | linear | {'C': 10, 'kernel': 'linear'} | 1.000000 | 1.000000 | 0.900000 | 0.966667 | 1.0 | 0.973333 | 0.038873 | 3 |
3 | 0.000604 | 0.000047 | 0.000344 | 0.000070 | 10 | rbf | {'C': 10, 'kernel': 'rbf'} | 0.966667 | 1.000000 | 0.966667 | 0.966667 | 1.0 | 0.980000 | 0.016330 | 1 |
Cross-validation, Hyper-Parameter Tuning, and Pipeline#
Common cross validation methods:
StratifiedKFold
: Split data into train and validation sets by preserving the percentage of samples of each classShuffleSplit
: Split data into train and validation sets by first shuffling the data and then splittingStratifiedShuffleSplit
: Stratified + ShuffledLeaveOneOut
: Creating train sets by taking all samples execept one, which is left out for validation
Important Steps
Define all preprocessing methods as Tranasformer
Create a pipeline
Define a cross-validation method
Use
cross_val_score()
to fine the best parameter settings by feeding the pipeline and the CV method
from sklearn.pipeline import make_pipeline
from sklearn import preprocessing
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import ShuffleSplit
## Create pipeline
clf = make_pipeline(preprocessing.StandardScaler(), svm.SVC(C=1))
## Define CV methods
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
shuffsplit = ShuffleSplit(n_splits=5, test_size=0.1, random_state=7)
## CV results
print('K-fold CV:', cross_val_score(clf, X, y, cv=kfold))
print('Shuffle CV:', cross_val_score(clf, X, y, cv=shuffsplit))
K-fold CV: [0.96666667 0.96666667 0.96666667 0.9 1. ]
Shuffle CV: [0.86666667 1. 1. 1. 0.93333333]
Deep Learning Example#
This is based on GridSearchCV with keras
MNIST Dataset#
import numpy as np
import os
from keras.datasets import mnist
from keras.layers import *
from keras.models import *
from time import time
nb_classes = 10
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 784)
X_test = X_test.reshape(X_test.shape[0], 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
## Convert y labels (integers) into binary class matrix
from keras.utils import np_utils
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
print(Y_train[:5,])
[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
[1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]
Hyper-Parameter Tuning Using Grid Search#
In this example, we aim to fine-tune the following hyper-parameters of the deep neural network:
optimizer
kernel_initializer
of the Dense layer
We can fine-tune other common parameters like epochs and batch-sizes.
def create_model(optimizer='rmsprop', init='glorot_uniform'):
model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
model.add(Dropout(0.2)) # Dropout helps protect the model from memorizing or "overfitting" the training data
model.add(Dense(512, kernel_initializer=init))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10, kernel_initializer=init))
model.add(Activation('softmax')) # This special "softmax" a
model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
return model
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV
start=time()
model = KerasClassifier(build_fn=create_model)
optimizers = ['rmsprop', 'adam']
init = ['glorot_uniform', 'normal', 'uniform']
# epochs = [50, 100, 150]
# batches = [5, 10, 20]
param_grid = dict(optimizer=optimizers, init=init)
grid = GridSearchCV(estimator=model, param_grid=param_grid)
grid_result = grid.fit(X_train, Y_train)
print("total time:",time()-start)
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0417 - accuracy: 0.9264
375/375 [==============================] - 1s 2ms/step - loss: 0.0259 - accuracy: 0.9593
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9270
375/375 [==============================] - 1s 2ms/step - loss: 0.0226 - accuracy: 0.9651
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0410 - accuracy: 0.9282
375/375 [==============================] - 1s 2ms/step - loss: 0.0270 - accuracy: 0.9578
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0415 - accuracy: 0.9274
375/375 [==============================] - 1s 2ms/step - loss: 0.0289 - accuracy: 0.9567
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0419 - accuracy: 0.9270
375/375 [==============================] - 1s 2ms/step - loss: 0.0221 - accuracy: 0.9633
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9271
375/375 [==============================] - 1s 2ms/step - loss: 0.0220 - accuracy: 0.9626
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0399 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0239 - accuracy: 0.9604
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0397 - accuracy: 0.9293
375/375 [==============================] - 1s 2ms/step - loss: 0.0232 - accuracy: 0.9592
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0219 - accuracy: 0.9624
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9271
375/375 [==============================] - 1s 2ms/step - loss: 0.0213 - accuracy: 0.9628
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0421 - accuracy: 0.9262
375/375 [==============================] - 1s 2ms/step - loss: 0.0254 - accuracy: 0.9599
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0239 - accuracy: 0.9621
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0418 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0271 - accuracy: 0.9573
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0416 - accuracy: 0.9277
375/375 [==============================] - 1s 2ms/step - loss: 0.0272 - accuracy: 0.9560
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0423 - accuracy: 0.9250
375/375 [==============================] - 1s 2ms/step - loss: 0.0227 - accuracy: 0.9643
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0407 - accuracy: 0.9268
375/375 [==============================] - 1s 2ms/step - loss: 0.0263 - accuracy: 0.9568
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0405 - accuracy: 0.9283
375/375 [==============================] - 1s 2ms/step - loss: 0.0219 - accuracy: 0.9612
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0409 - accuracy: 0.9275
375/375 [==============================] - 1s 2ms/step - loss: 0.0227 - accuracy: 0.9615
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0406 - accuracy: 0.9284
375/375 [==============================] - 1s 2ms/step - loss: 0.0241 - accuracy: 0.9573
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0407 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0184 - accuracy: 0.9672
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0422 - accuracy: 0.9251
375/375 [==============================] - 1s 2ms/step - loss: 0.0253 - accuracy: 0.9614
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0417 - accuracy: 0.9264
375/375 [==============================] - 1s 2ms/step - loss: 0.0238 - accuracy: 0.9621
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0418 - accuracy: 0.9266
375/375 [==============================] - 1s 2ms/step - loss: 0.0261 - accuracy: 0.9584
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0412 - accuracy: 0.9276
375/375 [==============================] - 1s 2ms/step - loss: 0.0331 - accuracy: 0.9478
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0428 - accuracy: 0.9250
375/375 [==============================] - 1s 2ms/step - loss: 0.0223 - accuracy: 0.9663
1500/1500 [==============================] - 3s 2ms/step - loss: 0.0412 - accuracy: 0.9258
375/375 [==============================] - 1s 2ms/step - loss: 0.0211 - accuracy: 0.9632
1500/1500 [==============================] - 3s 2ms/step - loss: 0.0411 - accuracy: 0.9243
375/375 [==============================] - 1s 2ms/step - loss: 0.0281 - accuracy: 0.9517
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0417 - accuracy: 0.9268
375/375 [==============================] - 1s 2ms/step - loss: 0.0311 - accuracy: 0.9483
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0412 - accuracy: 0.9266
375/375 [==============================] - 1s 2ms/step - loss: 0.0247 - accuracy: 0.9566
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0421 - accuracy: 0.9253
375/375 [==============================] - 1s 2ms/step - loss: 0.0255 - accuracy: 0.9543
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0370 - accuracy: 0.9340
total time: 163.10951614379883
## The Best Model
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
## Print all experiment results
import pandas as pd
results = pd.DataFrame(grid_result.cv_results_)
results
Best: 0.961483 using {'init': 'glorot_uniform', 'optimizer': 'adam'}
mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_init | param_optimizer | params | split0_test_score | split1_test_score | split2_test_score | split3_test_score | split4_test_score | mean_test_score | std_test_score | rank_test_score | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 4.962227 | 0.534804 | 0.789589 | 0.021689 | glorot_uniform | rmsprop | {'init': 'glorot_uniform', 'optimizer': 'rmspr... | 0.959333 | 0.965083 | 0.957750 | 0.956667 | 0.963333 | 0.960433 | 0.003244 | 3 |
1 | 4.087789 | 0.024603 | 0.798292 | 0.013339 | glorot_uniform | adam | {'init': 'glorot_uniform', 'optimizer': 'adam'} | 0.962583 | 0.960417 | 0.959250 | 0.962417 | 0.962750 | 0.961483 | 0.001401 | 1 |
2 | 4.723551 | 0.063123 | 0.842619 | 0.107239 | normal | rmsprop | {'init': 'normal', 'optimizer': 'rmsprop'} | 0.959917 | 0.962083 | 0.957333 | 0.956000 | 0.964333 | 0.959933 | 0.003040 | 4 |
3 | 4.174438 | 0.124374 | 0.784431 | 0.015665 | normal | adam | {'init': 'normal', 'optimizer': 'adam'} | 0.956833 | 0.961167 | 0.961500 | 0.957333 | 0.967167 | 0.960800 | 0.003712 | 2 |
4 | 4.698246 | 0.033024 | 0.798633 | 0.015098 | uniform | rmsprop | {'init': 'uniform', 'optimizer': 'rmsprop'} | 0.961417 | 0.962083 | 0.958417 | 0.947833 | 0.966250 | 0.959200 | 0.006209 | 5 |
5 | 4.086391 | 0.034679 | 0.859442 | 0.142505 | uniform | adam | {'init': 'uniform', 'optimizer': 'adam'} | 0.963250 | 0.951667 | 0.948333 | 0.956583 | 0.954333 | 0.954833 | 0.005029 | 6 |
K-fold Validation#
Similar to shuffle split?
np.random.seed(7)
seed=12
def create_model():
model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
model.add(Dropout(0.2)) # Dropout helps protect the model from memorizing or "overfitting" the training data
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax')) # This special "softmax" a
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
return model
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
start=time()
model2 = KerasClassifier(build_fn=create_model, epochs=10, batch_size=1500,verbose=1)
results = cross_val_score(model2, X_train, Y_train, cv=10)
print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1047 - accuracy: 0.7911
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0458 - accuracy: 0.9188
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0334 - accuracy: 0.9411
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9559
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0207 - accuracy: 0.9641
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9712
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9795
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0108 - accuracy: 0.9817
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0095 - accuracy: 0.9835
4/4 [==============================] - 0s 4ms/step - loss: 0.0121 - accuracy: 0.9782
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1044 - accuracy: 0.7950
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0435 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0319 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9552
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0196 - accuracy: 0.9662
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0161 - accuracy: 0.9725
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0100 - accuracy: 0.9825
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9851
4/4 [==============================] - 0s 3ms/step - loss: 0.0159 - accuracy: 0.9748
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1058 - accuracy: 0.7887
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0450 - accuracy: 0.9204
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0318 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9827
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9859
4/4 [==============================] - 0s 3ms/step - loss: 0.0184 - accuracy: 0.9685
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1028 - accuracy: 0.7998
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9203
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0243 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9660
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9715
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9757
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0121 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0097 - accuracy: 0.9830
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9846
4/4 [==============================] - 0s 3ms/step - loss: 0.0130 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1050 - accuracy: 0.7889
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0456 - accuracy: 0.9189
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0325 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9563
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9643
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0165 - accuracy: 0.9717
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9763
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9829
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9852
4/4 [==============================] - 0s 3ms/step - loss: 0.0144 - accuracy: 0.9750
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1059 - accuracy: 0.7875
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0451 - accuracy: 0.9212
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9446
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0246 - accuracy: 0.9583
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9750
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0122 - accuracy: 0.9793
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9819
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9855
4/4 [==============================] - 0s 4ms/step - loss: 0.0154 - accuracy: 0.9740
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1047 - accuracy: 0.7929
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9433
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0247 - accuracy: 0.9576
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0175 - accuracy: 0.9700
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9800
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0105 - accuracy: 0.9820
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9853
4/4 [==============================] - 0s 4ms/step - loss: 0.0143 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1046 - accuracy: 0.7949
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9182
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9426
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9556
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0206 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0167 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0085 - accuracy: 0.9856
4/4 [==============================] - 0s 4ms/step - loss: 0.0181 - accuracy: 0.9720
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1042 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0440 - accuracy: 0.9239
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0324 - accuracy: 0.9429
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0248 - accuracy: 0.9578
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9663
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0166 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0138 - accuracy: 0.9764
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0106 - accuracy: 0.9820
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0081 - accuracy: 0.9863
4/4 [==============================] - 0s 3ms/step - loss: 0.0180 - accuracy: 0.9710
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1045 - accuracy: 0.7953
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0455 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0328 - accuracy: 0.9438
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0175 - accuracy: 0.9697
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0145 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9785
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0114 - accuracy: 0.9810
K-fold Results: 97.49% (0.36%)
total time: 29.779260635375977
Shuffle Split Cross Validation#
Shuffle but not stratified
## Shuffle Split
start=time()
sf = ShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=sf)
print("K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7965
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0454 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0335 - accuracy: 0.9410
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0169 - accuracy: 0.9713
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0142 - accuracy: 0.9758
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0126 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9839
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0092 - accuracy: 0.9839
4/4 [==============================] - 0s 3ms/step - loss: 0.0129 - accuracy: 0.9773
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7952
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9183
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0261 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0208 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9723
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9791
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0101 - accuracy: 0.9835
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9850
4/4 [==============================] - 0s 3ms/step - loss: 0.0124 - accuracy: 0.9788
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1057 - accuracy: 0.7922
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0458 - accuracy: 0.9200
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9431
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0257 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0169 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9753
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9839
4/4 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9783
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1039 - accuracy: 0.7970
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0442 - accuracy: 0.9224
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9445
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0249 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0206 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0140 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0114 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0103 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0084 - accuracy: 0.9858
4/4 [==============================] - 0s 4ms/step - loss: 0.0167 - accuracy: 0.9733
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1043 - accuracy: 0.7938
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0453 - accuracy: 0.9199
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0331 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0255 - accuracy: 0.9557
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0208 - accuracy: 0.9642
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9749
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0124 - accuracy: 0.9781
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0098 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0087 - accuracy: 0.9851
4/4 [==============================] - 0s 4ms/step - loss: 0.0168 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1014 - accuracy: 0.8043
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0443 - accuracy: 0.9218
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0323 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0247 - accuracy: 0.9577
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0164 - accuracy: 0.9724
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9759
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0113 - accuracy: 0.9806
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9821
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0084 - accuracy: 0.9860
4/4 [==============================] - 0s 4ms/step - loss: 0.0125 - accuracy: 0.9800
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1080 - accuracy: 0.7908
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0467 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0332 - accuracy: 0.9424
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0209 - accuracy: 0.9638
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9702
Epoch 7/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0141 - accuracy: 0.9760
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0120 - accuracy: 0.9789
Epoch 9/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0101 - accuracy: 0.9831
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0091 - accuracy: 0.9847
4/4 [==============================] - 0s 4ms/step - loss: 0.0120 - accuracy: 0.9785
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1038 - accuracy: 0.7945
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0448 - accuracy: 0.9215
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0322 - accuracy: 0.9439
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0204 - accuracy: 0.9649
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0163 - accuracy: 0.9718
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9768
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0121 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0155 - accuracy: 0.9747
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1076 - accuracy: 0.7903
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0463 - accuracy: 0.9187
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9422
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9565
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0199 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0171 - accuracy: 0.9701
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0136 - accuracy: 0.9765
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9802
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9823
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0090 - accuracy: 0.9845
4/4 [==============================] - 0s 4ms/step - loss: 0.0137 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1045 - accuracy: 0.7910
Epoch 2/10
36/36 [==============================] - 0s 7ms/step - loss: 0.0454 - accuracy: 0.9210
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0324 - accuracy: 0.9441
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0253 - accuracy: 0.9561
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0202 - accuracy: 0.9651
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0165 - accuracy: 0.9719
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0144 - accuracy: 0.9755
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0116 - accuracy: 0.9804
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0100 - accuracy: 0.9832
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9848
4/4 [==============================] - 0s 3ms/step - loss: 0.0127 - accuracy: 0.9787
K-fold Results: 97.72% (0.20%)
total time: 31.2500159740448
Stratified K-Fold Validation#
Stratified but not shuffled
Stratified cross-validation gives poor results. No idea why?
print(Y_train[:4,])
print(y_train[:4])
from collections import Counter
Counter(y_train)
[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
[1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]
[5 0 4 1]
Counter({0: 5923,
1: 6742,
2: 5958,
3: 6131,
4: 5842,
5: 5421,
6: 5918,
7: 6265,
8: 5851,
9: 5949})
## Stratified Shuffle
from sklearn.model_selection import StratifiedKFold
start=time()
sk = StratifiedKFold(n_splits=10)
results = cross_val_score(model2, X_train, y_train, cv=sk)
print("Stratified K-fold Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0234 - accuracy: 0.0974
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8925 - accuracy: 0.1069
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0986
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1016
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0997
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1024
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
4/4 [==============================] - 0s 3ms/step - loss: 9.8922 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0317 - accuracy: 0.0952
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8930 - accuracy: 0.1012
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1006
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0984
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0981
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1014
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0076 - accuracy: 0.0989
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8922 - accuracy: 0.1001
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0968
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1012
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0975
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.0995
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1021
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1017
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1018
4/4 [==============================] - 0s 3ms/step - loss: 9.8903 - accuracy: 0.0993
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0352 - accuracy: 0.1036
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8921 - accuracy: 0.1104
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1044
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1035
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0994
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8918 - accuracy: 0.1016
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0992
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1004
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1002
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1022
4/4 [==============================] - 0s 4ms/step - loss: 9.8903 - accuracy: 0.1022
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0332 - accuracy: 0.0993
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8924 - accuracy: 0.1021
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1002
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0984
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0978
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0956
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1037
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1028
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1009
4/4 [==============================] - 0s 4ms/step - loss: 9.8922 - accuracy: 0.0992
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0069 - accuracy: 0.0996
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0977
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0980
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1004
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0998
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.1025
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1010
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0991
4/4 [==============================] - 0s 3ms/step - loss: 9.8925 - accuracy: 0.1023
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0232 - accuracy: 0.0983
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8921 - accuracy: 0.0992
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0999
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0981
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0989
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.0995
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0973
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8916 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8916 - accuracy: 0.0990
4/4 [==============================] - 0s 4ms/step - loss: 9.8925 - accuracy: 0.0975
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0450 - accuracy: 0.0947
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8924 - accuracy: 0.0943
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.1015
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0983
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0987
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.0977
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1024
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8918 - accuracy: 0.1005
4/4 [==============================] - 0s 3ms/step - loss: 9.8900 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 10.0163 - accuracy: 0.1055
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8919 - accuracy: 0.0834
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0926
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0991
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1032
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1005
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1022
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1000
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
4/4 [==============================] - 0s 4ms/step - loss: 9.8929 - accuracy: 0.0987
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 10.0165 - accuracy: 0.1039
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8919 - accuracy: 0.0942
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0979
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.0997
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1028
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1006
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0996
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 9.8915 - accuracy: 0.1008
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.1046
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 9.8915 - accuracy: 0.0997
4/4 [==============================] - 0s 4ms/step - loss: 9.8933 - accuracy: 0.1022
Stratified K-fold Results: 9.97% (0.17%)
total time: 30.07964015007019
Stratified Shuffle Split#
A mix of
StratifiedKFold
andShuffleSplit
, which returns stratified randomized foldsThe folds are made by preserving the percentage of samples for each class.
## Stratified Shuffle
from sklearn.model_selection import StratifiedShuffleSplit
start=time()
ss = StratifiedShuffleSplit(n_splits=10)
results = cross_val_score(model2, X_train, Y_train, cv=ss)
print("Stratified Shuffle Results: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))
print("total time:",time()-start)
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1072 - accuracy: 0.7853
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0460 - accuracy: 0.9186
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0341 - accuracy: 0.9406
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0260 - accuracy: 0.9551
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0209 - accuracy: 0.9644
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0143 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0122 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0101 - accuracy: 0.9828
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9840
4/4 [==============================] - 0s 3ms/step - loss: 0.0149 - accuracy: 0.9753
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1032 - accuracy: 0.7954
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0449 - accuracy: 0.9205
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0321 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9573
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9647
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9714
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0142 - accuracy: 0.9754
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0118 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0099 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9856
4/4 [==============================] - 0s 3ms/step - loss: 0.0151 - accuracy: 0.9752
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1054 - accuracy: 0.7976
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0474 - accuracy: 0.9161
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0328 - accuracy: 0.9434
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0253 - accuracy: 0.9564
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0205 - accuracy: 0.9654
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9706
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0136 - accuracy: 0.9767
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0124 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0105 - accuracy: 0.9818
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0087 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9745
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1044 - accuracy: 0.7927
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0452 - accuracy: 0.9197
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0320 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0250 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0198 - accuracy: 0.9658
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0167 - accuracy: 0.9710
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0139 - accuracy: 0.9762
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0117 - accuracy: 0.9803
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0102 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0167 - accuracy: 0.9722
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1065 - accuracy: 0.7891
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0462 - accuracy: 0.9185
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0330 - accuracy: 0.9418
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0252 - accuracy: 0.9568
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0214 - accuracy: 0.9632
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0144 - accuracy: 0.9744
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9794
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0107 - accuracy: 0.9810
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0094 - accuracy: 0.9841
4/4 [==============================] - 0s 5ms/step - loss: 0.0141 - accuracy: 0.9780
Epoch 1/10
36/36 [==============================] - 0s 5ms/step - loss: 0.1051 - accuracy: 0.7913
Epoch 2/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0464 - accuracy: 0.9176
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0320 - accuracy: 0.9447
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0254 - accuracy: 0.9560
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0204 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0170 - accuracy: 0.9709
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0143 - accuracy: 0.9752
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0117 - accuracy: 0.9796
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0104 - accuracy: 0.9824
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0088 - accuracy: 0.9849
4/4 [==============================] - 0s 3ms/step - loss: 0.0143 - accuracy: 0.9770
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1052 - accuracy: 0.7921
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0445 - accuracy: 0.9216
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0325 - accuracy: 0.9442
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0254 - accuracy: 0.9570
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0201 - accuracy: 0.9650
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0168 - accuracy: 0.9705
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0140 - accuracy: 0.9766
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0118 - accuracy: 0.9799
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0098 - accuracy: 0.9833
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0089 - accuracy: 0.9847
4/4 [==============================] - 0s 3ms/step - loss: 0.0138 - accuracy: 0.9763
Epoch 1/10
36/36 [==============================] - 0s 7ms/step - loss: 0.1029 - accuracy: 0.7947
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9211
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0316 - accuracy: 0.9444
Epoch 4/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0239 - accuracy: 0.9595
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0198 - accuracy: 0.9657
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0157 - accuracy: 0.9731
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0139 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0119 - accuracy: 0.9797
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0093 - accuracy: 0.9842
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0086 - accuracy: 0.9854
4/4 [==============================] - 0s 4ms/step - loss: 0.0127 - accuracy: 0.9777
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1030 - accuracy: 0.7933
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0447 - accuracy: 0.9201
Epoch 3/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0322 - accuracy: 0.9437
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0250 - accuracy: 0.9572
Epoch 5/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0193 - accuracy: 0.9659
Epoch 6/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0168 - accuracy: 0.9704
Epoch 7/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0134 - accuracy: 0.9769
Epoch 8/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0123 - accuracy: 0.9788
Epoch 9/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0096 - accuracy: 0.9834
Epoch 10/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0086 - accuracy: 0.9857
4/4 [==============================] - 0s 4ms/step - loss: 0.0136 - accuracy: 0.9775
Epoch 1/10
36/36 [==============================] - 0s 6ms/step - loss: 0.1058 - accuracy: 0.7881
Epoch 2/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0448 - accuracy: 0.9198
Epoch 3/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0323 - accuracy: 0.9436
Epoch 4/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0259 - accuracy: 0.9553
Epoch 5/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0201 - accuracy: 0.9648
Epoch 6/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0171 - accuracy: 0.9708
Epoch 7/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0141 - accuracy: 0.9756
Epoch 8/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0120 - accuracy: 0.9790
Epoch 9/10
36/36 [==============================] - 0s 6ms/step - loss: 0.0102 - accuracy: 0.9822
Epoch 10/10
36/36 [==============================] - 0s 5ms/step - loss: 0.0083 - accuracy: 0.9862
4/4 [==============================] - 0s 3ms/step - loss: 0.0119 - accuracy: 0.9795
Stratified Shuffle Results: 97.63% (0.20%)
total time: 30.383320093154907
GridSearchCV and Cross-Validation#
We perform the
GridSearchCV
to find the optimal optimizersWe determine the best parameter values based on stratified shuffled 5-fold cross validation
def create_model():
model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu')) # An "activation" is just a non-linear function applied to the output
model.add(Dropout(0.2)) # Dropout helps protect the model from memorizing or "overfitting" the training data
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax')) # This special "softmax" a
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
return model
start=time()
## Define cross-validation method
ss = StratifiedShuffleSplit(n_splits=5)
## Define Model
model3 = KerasClassifier(build_fn=create_model)
## Define Hyper-parameter values
optimizers = ['rmsprop', 'adam']
param_grid = dict(optimizer=optimizers, init=init)
## Define GridSearchCV
grid = GridSearchCV(estimator=model, param_grid=param_grid, cv=ss)
grid_result = grid.fit(X_train, Y_train)
print("total time:",time()-start)
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0402 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0275 - accuracy: 0.9605
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0395 - accuracy: 0.9323
188/188 [==============================] - 0s 2ms/step - loss: 0.0300 - accuracy: 0.9485
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9315
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9605
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0400 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0249 - accuracy: 0.9627
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0400 - accuracy: 0.9309
188/188 [==============================] - 0s 2ms/step - loss: 0.0305 - accuracy: 0.9533
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0393 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0237 - accuracy: 0.9608
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0382 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0201 - accuracy: 0.9648
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0387 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0274 - accuracy: 0.9540
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0379 - accuracy: 0.9324
188/188 [==============================] - 0s 2ms/step - loss: 0.0256 - accuracy: 0.9577
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0389 - accuracy: 0.9311
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9633
1688/1688 [==============================] - 6s 4ms/step - loss: 0.0405 - accuracy: 0.9282
188/188 [==============================] - 0s 2ms/step - loss: 0.0236 - accuracy: 0.9647
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0397 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0259 - accuracy: 0.9587
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0405 - accuracy: 0.9285
188/188 [==============================] - 0s 2ms/step - loss: 0.0219 - accuracy: 0.9645
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9305
188/188 [==============================] - 0s 2ms/step - loss: 0.0311 - accuracy: 0.9557
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0398 - accuracy: 0.9308
188/188 [==============================] - 0s 2ms/step - loss: 0.0212 - accuracy: 0.9642
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0392 - accuracy: 0.9298
188/188 [==============================] - 0s 2ms/step - loss: 0.0230 - accuracy: 0.9590
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0388 - accuracy: 0.9310
188/188 [==============================] - 0s 2ms/step - loss: 0.0232 - accuracy: 0.9580
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0390 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0224 - accuracy: 0.9613
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9302
188/188 [==============================] - 0s 2ms/step - loss: 0.0263 - accuracy: 0.9573
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0391 - accuracy: 0.9297
188/188 [==============================] - 0s 2ms/step - loss: 0.0221 - accuracy: 0.9638
1688/1688 [==============================] - 6s 3ms/step - loss: 0.0408 - accuracy: 0.9281
188/188 [==============================] - 0s 2ms/step - loss: 0.0235 - accuracy: 0.9618
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0408 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0246 - accuracy: 0.9605
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0404 - accuracy: 0.9306
188/188 [==============================] - 0s 2ms/step - loss: 0.0271 - accuracy: 0.9593
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0399 - accuracy: 0.9296
188/188 [==============================] - 0s 2ms/step - loss: 0.0231 - accuracy: 0.9638
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0403 - accuracy: 0.9299
188/188 [==============================] - 0s 2ms/step - loss: 0.0213 - accuracy: 0.9653
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0395 - accuracy: 0.9291
188/188 [==============================] - 0s 2ms/step - loss: 0.0239 - accuracy: 0.9612
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0394 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0217 - accuracy: 0.9635
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0401 - accuracy: 0.9289
188/188 [==============================] - 0s 2ms/step - loss: 0.0210 - accuracy: 0.9640
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0397 - accuracy: 0.9286
188/188 [==============================] - 0s 2ms/step - loss: 0.0225 - accuracy: 0.9600
1688/1688 [==============================] - 5s 3ms/step - loss: 0.0405 - accuracy: 0.9280
188/188 [==============================] - 0s 2ms/step - loss: 0.0193 - accuracy: 0.9663
1875/1875 [==============================] - 4s 2ms/step - loss: 0.0384 - accuracy: 0.9313
total time: 203.81905102729797
## The Best Model
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
## Print all experiment results
import pandas as pd
results = pd.DataFrame(grid_result.cv_results_)
results
Best: 0.963000 using {'init': 'uniform', 'optimizer': 'adam'}
mean_fit_time | std_fit_time | mean_score_time | std_score_time | param_init | param_optimizer | params | split0_test_score | split1_test_score | split2_test_score | split3_test_score | split4_test_score | mean_test_score | std_test_score | rank_test_score | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 6.656565 | 0.121780 | 0.524209 | 0.008416 | glorot_uniform | rmsprop | {'init': 'glorot_uniform', 'optimizer': 'rmspr... | 0.960500 | 0.948500 | 0.960500 | 0.962667 | 0.953333 | 0.957100 | 0.005334 | 6 |
1 | 6.022944 | 0.100696 | 0.534721 | 0.013367 | glorot_uniform | adam | {'init': 'glorot_uniform', 'optimizer': 'adam'} | 0.960833 | 0.964833 | 0.954000 | 0.957667 | 0.963333 | 0.960133 | 0.003912 | 4 |
2 | 6.614549 | 0.093743 | 0.522084 | 0.007600 | normal | rmsprop | {'init': 'normal', 'optimizer': 'rmsprop'} | 0.964667 | 0.958667 | 0.964500 | 0.955667 | 0.964167 | 0.961533 | 0.003693 | 3 |
3 | 5.639868 | 0.232617 | 0.496856 | 0.013943 | normal | adam | {'init': 'normal', 'optimizer': 'adam'} | 0.959000 | 0.958000 | 0.961333 | 0.957333 | 0.963833 | 0.959900 | 0.002389 | 5 |
4 | 6.057838 | 0.092201 | 0.491186 | 0.010245 | uniform | rmsprop | {'init': 'uniform', 'optimizer': 'rmsprop'} | 0.961833 | 0.960500 | 0.959333 | 0.963833 | 0.965333 | 0.962167 | 0.002178 | 2 |
5 | 5.619616 | 0.151503 | 0.484414 | 0.004364 | uniform | adam | {'init': 'uniform', 'optimizer': 'adam'} | 0.961167 | 0.963500 | 0.964000 | 0.960000 | 0.966333 | 0.963000 | 0.002224 | 1 |
References#
Requirements#
# %run ./get_modules.py
import pkg_resources
import types
def get_imports():
for name, val in globals().items():
if isinstance(val, types.ModuleType):
# Split ensures you get root package,
# not just imported function
name = val.__name__.split(".")[0]
elif isinstance(val, type):
name = val.__module__.split(".")[0]
# Some packages are weird and have different
# imported names vs. system/pip names. Unfortunately,
# there is no systematic way to get pip names from
# a package's imported name. You'll have to add
# exceptions to this list manually!
poorly_named_packages = {
"PIL": "Pillow",
"sklearn": "scikit-learn"
}
if name in poorly_named_packages.keys():
name = poorly_named_packages[name]
yield name
imports = list(set(get_imports()))
# The only way I found to get the version of the root package
# from only the name of the package is to cross-check the names
# of installed packages vs. imported packages
requirements = []
for m in pkg_resources.working_set:
if m.project_name in imports and m.project_name!="pip":
requirements.append((m.project_name, m.version))
for r in requirements:
print("{}=={}".format(*r))
tensorflow==2.3.0
scikit-learn==0.22.2.post1
pandas==1.1.2
numpy==1.18.5