Use GridSearchCV from sklearn.model_selection. You give it a model, a dictionary of parameter values to try, and a number of cross-validation folds. It fits every combination, scores each one by cross-validation, and keeps the best. Put the model inside a Pipeline first, or your results will be quietly wrong.
Grid search is the first tuning method most people learn, and it is also the one most often used incorrectly. The mistakes are not in the grid. They are in what surrounds it: preprocessing that sees the validation fold, a scoring metric chosen by habit, and a best score reported as though it were a prediction of future performance.
All code here was run on scikit-learn 1.9.1. Every number is copied from the actual run.
What is a hyperparameter, and why does it need tuning?
A hyperparameter is a setting you choose before training, as opposed to a parameter the model learns from the data during training. The weights in a logistic regression are parameters. The C that controls how strongly it is regularised is a hyperparameter.
The model cannot learn its own hyperparameters, because they govern how the learning happens. So you try several values and keep whichever performs best, measured honestly. That is all tuning is.
The examples below use the breast cancer dataset built into scikit-learn: 569 rows, 30 features, split into 455 training rows and 114 held-out test rows.
How do you run a grid search in scikit-learn?
Build a pipeline, define a dictionary of values to try, pass both to GridSearchCV, and call fit.
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipe = Pipeline([("scale", StandardScaler()), ("clf", SVC(random_state=42))])
grid = {
"clf__C": [0.1, 1, 10],
"clf__gamma": ["scale", 0.01],
"clf__kernel": ["rbf"],
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
gs = GridSearchCV(pipe, grid, cv=cv, scoring="accuracy", n_jobs=-1)
gs.fit(X_train, y_train)best_params_: {'clf__C': 10, 'clf__gamma': 0.01, 'clf__kernel': 'rbf'}
best_score_ : 0.9758Three details in that code are doing real work.
The double underscore. clf__C means "the C parameter of the step named clf". That is how you reach inside a pipeline. One underscore will not work, and the error message is not obvious.
StratifiedKFold. For classification, each fold should hold roughly the same class balance as the whole dataset. Passing an integer to cv gives you stratified folds for a classifier anyway, but writing it out means you control the shuffle and the random seed, so the run is reproducible.
n_jobs=-1. Use every available core. Grid search is embarrassingly parallel, and this is the cheapest speed-up available.
How do you see what actually happened?
Read cv_results_ into a DataFrame. The best parameters alone hide whether the winner won by a mile or by noise.
import pandas as pd
res = pd.DataFrame(gs.cv_results_)
res[["param_clf__C", "param_clf__gamma", "mean_test_score",
"std_test_score", "rank_test_score"]].sort_values("rank_test_score") param_clf__C param_clf__gamma mean_test_score std_test_score rank_test_score
10.0 0.01 0.975824 0.016150 1
1.0 0.01 0.971429 0.014906 2
10.0 scale 0.969231 0.012815 3
1.0 scale 0.969231 0.014579 3
0.1 0.01 0.947253 0.012815 5
0.1 scale 0.942857 0.016150 6Look at the top two rows. The winner scores 0.9758 and the runner-up 0.9714, a gap of 0.0044. Now look at std_test_score: about 0.016, roughly four times the gap. The difference between first and second place is smaller than the fold-to-fold noise. Treating that winner as meaningfully better than the runner-up is reading signal into a rounding error.
This is why cv_results_ matters. It is the difference between "C=10 is the best value" and "anything from 1 to 10 works, and 0.1 is genuinely worse".
Why must the grid search go inside a Pipeline?
Because any preprocessing step fitted outside cross-validation has already seen the validation fold, which inflates your score by an amount you cannot measure. This is data leakage, and it is the single most expensive mistake in this article.
Here is the demonstration. Take 200 rows of pure random noise with 2,000 features and random labels. There is no signal whatsoever, so an honest classifier should score about 0.50.
# WRONG: pick the best features using all the data, then cross-validate
sel = SelectKBest(f_classif, k=20).fit(X_noise, y_noise)
cross_val_score(LogisticRegression(), sel.transform(X_noise), y_noise, cv=5).mean()
# RIGHT: selection happens inside each fold
pipe = Pipeline([("sel", SelectKBest(f_classif, k=20)),
("clf", LogisticRegression())])
cross_val_score(pipe, X_noise, y_noise, cv=5).mean()selection BEFORE cv (leaky) : 0.755 selection INSIDE a pipeline : 0.555 inflation from leakage : 0.200
0.755 on data that contains no information at all. The feature selector looked at all 200 rows, found the 20 features that happened to correlate with the labels by chance, and handed them to a model that was then tested on those same rows. The pipeline version gets 0.555, which is close to the honest 0.50 for a sample this small.
Scaling leaks less dramatically than feature selection, but it leaks. So does imputation, encoding and resampling. The rule is simple and has no exceptions worth learning: every step that learns anything from the data goes inside the pipeline.
How many models does a grid search actually fit?
The number of combinations multiplied by the number of folds, plus one final refit. The example above fits 6 × 5 = 30 models, then refits the winner on all the training data, giving 31.
That is fine. The problem is how fast it grows, because the combinations multiply rather than add.
| Parameters | Values each | Folds | Combinations | Model fits |
|---|---|---|---|---|
| 2 | 3 | 5 | 9 | 45 |
| 3 | 4 | 5 | 64 | 320 |
| 4 | 5 | 5 | 625 | 3,125 |
| 5 | 5 | 10 | 3,125 | 31,250 |
Adding one more parameter with five values multiplies your runtime by five. A grid that feels reasonable while you are typing it can take overnight. Work out the fits before you run it:
from sklearn.model_selection import ParameterGrid print(len(ParameterGrid(grid)) * cv.get_n_splits())
Is best_score_ the number you should report?
No. best_score_ is the score of the configuration you selected because it scored highest, so it is biased upward by the selection itself. Report a score from data the search never touched.
print(gs.best_score_) # 0.9758 cross-validation, used for selection print(gs.score(X_test, y_test)) # 0.9825 held-out test set
Here the held-out score came out slightly higher, at 0.9825. That is worth dwelling on, because the usual advice is that best_score_ always overstates performance, and this run shows otherwise.
Both statements can be true. The selection bias is real and pushes best_score_ up in expectation. But on any single run, that bias is competing with ordinary sampling noise, and with only 114 test rows, one extra correct prediction moves the test score by almost 0.009. The two numbers here differ by 0.0066, well inside the noise.
The honest reading is not "the test score was better". It is: these two numbers are not precise enough to be compared, and neither one is a promise. If the decision matters, use nested cross-validation, or hold out a test set large enough that its score means something.
Which scoring metric should you use?
Whichever one matches the decision the model will be used for, chosen deliberately, because the metric decides which model wins. On balanced data this often makes no difference. On imbalanced data it changes the answer completely.
Here is the same grid, on the same data, with only the scoring string changed. The dataset is 4,000 rows with a 4% positive rate, like a fraud or default problem.
scoring | Winning C | Winning class_weight | Best score |
|---|---|---|---|
accuracy | 1 | None | 0.9642 |
roc_auc | 10 | None | 0.7281 |
average_precision | 0.01 | None | 0.3468 |
recall | 0.01 | balanced | 0.6150 |
Four metrics, four different winners. C ranges across three orders of magnitude and class_weight flips, purely from which string you passed to scoring. The accuracy figure of 0.9642 is the least useful number on that table: with a 4% positive rate, predicting "negative" every time scores about 0.96.
If you are tuning on rare-event data, decide the metric before you decide the grid.
Is random search better than grid search?
Usually, for the same time budget. Random search samples combinations instead of trying all of them, which means it spends its budget exploring more distinct values of whichever parameter actually matters.
Bergstra and Bengio (2012) made this argument formally: in most problems only a few hyperparameters matter much, and a grid wastes most of its fits re-testing the ones that do not.
Here is a 72-combination grid on a random forest, against a random search allowed only 12 of them.
grid : 72 combinations, best score 0.9626, 33.4s random : 12 combinations, best score 0.9626, 5.4s
Identical score, 84% less time. Swapping GridSearchCV for RandomizedSearchCV is a one-word change plus an n_iter:
from sklearn.model_selection import RandomizedSearchCV rs = RandomizedSearchCV(pipe, grid, n_iter=12, cv=3, n_jobs=-1, random_state=42)
Grid search still earns its place when the grid is genuinely small, when you need to be able to say you tried every combination, or when the parameters are categorical with few levels.
What does this look like end to end?
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42) # hold out FIRST
pipe = Pipeline([("scale", StandardScaler()), # inside the pipeline
("clf", SVC(random_state=42))])
grid = {"clf__C": [0.1, 1, 10], "clf__gamma": ["scale", 0.01]}
cv = StratifiedKFold(5, shuffle=True, random_state=42)
gs = GridSearchCV(pipe, grid, cv=cv, scoring="roc_auc", n_jobs=-1)
gs.fit(X_tr, y_tr)
print(gs.best_params_)
print(pd.DataFrame(gs.cv_results_)[["params", "mean_test_score", "std_test_score"]])
print("held-out:", gs.score(X_te, y_te)) # the number you reportThe order of those steps is the article in miniature. Hold out the test set first, put every learned step inside the pipeline, inspect the spread rather than just the winner, and report the held-out score. gs itself is already refitted on all the training data, so gs.predict(X_new) works directly.
When should you not tune at all?
Four situations where tuning is the wrong use of your time, or will mislead you.
1. Your features or labels are the real problem. Tuning moves a score by fractions of a percent. Fixing a mislabelled target, a leaking feature or a missing data source moves it by much more. Tune last, not first.
2. The gap between configurations is inside the noise. If first and second place differ by less than std_test_score, you have not found a better model. You have found a better random seed. Widen the range of values rather than refining the grid around a winner that is not really winning.
3. You cannot afford the fits. A grid that takes nine hours and improves cross-validated accuracy from 0.9758 to 0.9769 has cost more than it returned. Decide the budget first, then size the grid to fit it.
4. You assume the newest method is faster. Successive halving, available as HalvingGridSearchCV, trains many candidates on small subsets and promotes the survivors. It is genuinely useful on large datasets with expensive models. On the small problem used here it was slower and worse: 45.0 seconds against the full grid's 33.4, for a best score of 0.9512 against 0.9626. With 455 training rows, the early rounds are too noisy to pick good candidates and the overhead is not repaid. Measure it on your own data before adopting it.
Finally, a tuned model is still only as good as its evaluation. For choosing the model family in the first place, which usually matters more than tuning it, see decision tree vs random forest and the difference between random forest and XGBoost.
What should you do next?
- Take one model you have already built and wrap it in a
Pipeline, with every preprocessing step inside. Re-run your cross-validation and compare the score to what you had before. If it drops, you have just found a leak. - Print
len(ParameterGrid(grid)) * n_foldsbefore every search. If the number surprises you, cut the grid or switch toRandomizedSearchCV. - Report the held-out score, not
best_score_, and state the fold-to-fold standard deviation next to it so the reader knows how precise it is.
Frequently asked questions
What is GridSearchCV in scikit-learn?
It is a class that performs an exhaustive search over the parameter values you specify, scoring every combination by cross-validation. After fitting, it exposes best_params_, best_score_ and a best_estimator_ that has been refitted on all the training data.
What is the difference between grid search and random search?
Grid search tries every combination in the grid, while random search samples a fixed number of combinations. For the same time budget, random search usually explores more distinct values of the parameters that matter, which is why it often matches grid search in a fraction of the time.
Why do I need a Pipeline with GridSearchCV?
Because preprocessing fitted outside cross-validation has already seen the validation fold, which inflates the score. In the noise example in this article, doing feature selection outside the pipeline produced 0.755 on data with no signal in it at all.
What does the double underscore mean in a parameter grid?
It separates the pipeline step name from the parameter name, so clf__C means the C parameter of the step named clf. A single underscore will not be recognised.
Is best_score_ the accuracy I should report?
No. It is the cross-validated score of the configuration chosen because it scored highest, so the selection itself biases it upward. Report a score computed on data the search never saw, and state how noisy that estimate is.
How many combinations is too many for a grid search?
It depends on how long one fit takes, so calculate it rather than guessing: len(ParameterGrid(grid)) * n_folds gives the number of model fits. Multiply by the time for a single fit, and if the answer does not fit your budget, use RandomizedSearchCV.
Sources
- scikit-learn documentation. "GridSearchCV." https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
- scikit-learn user guide. "Tuning the hyper-parameters of an estimator." https://scikit-learn.org/stable/modules/grid_search.html
- scikit-learn user guide. "Common pitfalls and recommended practices," on data leakage. https://scikit-learn.org/stable/common_pitfalls.html
- scikit-learn user guide. "Successive halving for hyperparameter tuning." https://scikit-learn.org/stable/modules/grid_search.html#successive-halving-user-guide
- Bergstra, J. and Bengio, Y. "Random Search for Hyper-Parameter Optimization." Journal of Machine Learning Research, 13, 281–305, 2012. https://jmlr.org/papers/v13/bergstra12a.html

By