AI Help Center › Machine Learning › Hyperparameter Tuning Using Grid Search in Python
MACHINE LEARNING · BEGINNER GUIDE

Hyperparameter Tuning Using Grid Search in Python

GridSearchCV end to end, and the one-line pipeline rule that stops a score being quietly wrong.

10 min readPublished 17 Nov 2026Verified on scikit-learn 1.9.1
Quick insight

Leakage beats tuning, every time. On 200 rows of pure noise, selecting features outside the pipeline scored 0.755. Inside a pipeline it scored 0.555, near the honest 0.50. No amount of tuning recovers from that.

Use GridSearchCV from sklearn.model_selection. You give it a model, a dictionary of parameter values to try, and a number of cross-validation folds. It fits every combination, scores each one by cross-validation, and keeps the best. Put the model inside a Pipeline first, or your results will be quietly wrong.

Grid search is the first tuning method most people learn, and it is also the one most often used incorrectly. The mistakes are not in the grid. They are in what surrounds it: preprocessing that sees the validation fold, a scoring metric chosen by habit, and a best score reported as though it were a prediction of future performance.

All code here was run on scikit-learn 1.9.1. Every number is copied from the actual run.

Machine Learning

What is a hyperparameter, and why does it need tuning?

A hyperparameter is a setting you choose before training, as opposed to a parameter the model learns from the data during training. The weights in a logistic regression are parameters. The C that controls how strongly it is regularised is a hyperparameter.

The model cannot learn its own hyperparameters, because they govern how the learning happens. So you try several values and keep whichever performs best, measured honestly. That is all tuning is.

The examples below use the breast cancer dataset built into scikit-learn: 569 rows, 30 features, split into 455 training rows and 114 held-out test rows.

Machine Learning

How do you run a grid search in scikit-learn?

Build a pipeline, define a dictionary of values to try, pass both to GridSearchCV, and call fit.

python
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

pipe = Pipeline([("scale", StandardScaler()), ("clf", SVC(random_state=42))])

grid = {
    "clf__C":      [0.1, 1, 10],
    "clf__gamma":  ["scale", 0.01],
    "clf__kernel": ["rbf"],
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
gs = GridSearchCV(pipe, grid, cv=cv, scoring="accuracy", n_jobs=-1)
gs.fit(X_train, y_train)
output
best_params_: {'clf__C': 10, 'clf__gamma': 0.01, 'clf__kernel': 'rbf'}
best_score_ : 0.9758

Three details in that code are doing real work.

The double underscore. clf__C means "the C parameter of the step named clf". That is how you reach inside a pipeline. One underscore will not work, and the error message is not obvious.

StratifiedKFold. For classification, each fold should hold roughly the same class balance as the whole dataset. Passing an integer to cv gives you stratified folds for a classifier anyway, but writing it out means you control the shuffle and the random seed, so the run is reproducible.

n_jobs=-1. Use every available core. Grid search is embarrassingly parallel, and this is the cheapest speed-up available.

Machine Learning

How do you see what actually happened?

Read cv_results_ into a DataFrame. The best parameters alone hide whether the winner won by a mile or by noise.

python
import pandas as pd
res = pd.DataFrame(gs.cv_results_)
res[["param_clf__C", "param_clf__gamma", "mean_test_score",
     "std_test_score", "rank_test_score"]].sort_values("rank_test_score")
output
 param_clf__C param_clf__gamma  mean_test_score  std_test_score  rank_test_score
         10.0             0.01         0.975824        0.016150                1
          1.0             0.01         0.971429        0.014906                2
         10.0            scale         0.969231        0.012815                3
          1.0            scale         0.969231        0.014579                3
          0.1             0.01         0.947253        0.012815                5
          0.1            scale         0.942857        0.016150                6

Look at the top two rows. The winner scores 0.9758 and the runner-up 0.9714, a gap of 0.0044. Now look at std_test_score: about 0.016, roughly four times the gap. The difference between first and second place is smaller than the fold-to-fold noise. Treating that winner as meaningfully better than the runner-up is reading signal into a rounding error.

This is why cv_results_ matters. It is the difference between "C=10 is the best value" and "anything from 1 to 10 works, and 0.1 is genuinely worse".

Common error

Why must the grid search go inside a Pipeline?

Because any preprocessing step fitted outside cross-validation has already seen the validation fold, which inflates your score by an amount you cannot measure. This is data leakage, and it is the single most expensive mistake in this article.

Here is the demonstration. Take 200 rows of pure random noise with 2,000 features and random labels. There is no signal whatsoever, so an honest classifier should score about 0.50.

python
# WRONG: pick the best features using all the data, then cross-validate
sel = SelectKBest(f_classif, k=20).fit(X_noise, y_noise)
cross_val_score(LogisticRegression(), sel.transform(X_noise), y_noise, cv=5).mean()

# RIGHT: selection happens inside each fold
pipe = Pipeline([("sel", SelectKBest(f_classif, k=20)),
                 ("clf", LogisticRegression())])
cross_val_score(pipe, X_noise, y_noise, cv=5).mean()
output
selection BEFORE cv (leaky) : 0.755
selection INSIDE a pipeline : 0.555
inflation from leakage      : 0.200

0.755 on data that contains no information at all. The feature selector looked at all 200 rows, found the 20 features that happened to correlate with the labels by chance, and handed them to a model that was then tested on those same rows. The pipeline version gets 0.555, which is close to the honest 0.50 for a sample this small.

Scaling leaks less dramatically than feature selection, but it leaks. So does imputation, encoding and resampling. The rule is simple and has no exceptions worth learning: every step that learns anything from the data goes inside the pipeline.

Machine Learning

How many models does a grid search actually fit?

The number of combinations multiplied by the number of folds, plus one final refit. The example above fits 6 × 5 = 30 models, then refits the winner on all the training data, giving 31.

That is fine. The problem is how fast it grows, because the combinations multiply rather than add.

ParametersValues eachFoldsCombinationsModel fits
235945
34564320
4556253,125
55103,12531,250

Adding one more parameter with five values multiplies your runtime by five. A grid that feels reasonable while you are typing it can take overnight. Work out the fits before you run it:

python
from sklearn.model_selection import ParameterGrid
print(len(ParameterGrid(grid)) * cv.get_n_splits())
Common error

Is best_score_ the number you should report?

No. best_score_ is the score of the configuration you selected because it scored highest, so it is biased upward by the selection itself. Report a score from data the search never touched.

python
print(gs.best_score_)            # 0.9758  cross-validation, used for selection
print(gs.score(X_test, y_test))  # 0.9825  held-out test set

Here the held-out score came out slightly higher, at 0.9825. That is worth dwelling on, because the usual advice is that best_score_ always overstates performance, and this run shows otherwise.

Both statements can be true. The selection bias is real and pushes best_score_ up in expectation. But on any single run, that bias is competing with ordinary sampling noise, and with only 114 test rows, one extra correct prediction moves the test score by almost 0.009. The two numbers here differ by 0.0066, well inside the noise.

The honest reading is not "the test score was better". It is: these two numbers are not precise enough to be compared, and neither one is a promise. If the decision matters, use nested cross-validation, or hold out a test set large enough that its score means something.

Machine Learning

Which scoring metric should you use?

Whichever one matches the decision the model will be used for, chosen deliberately, because the metric decides which model wins. On balanced data this often makes no difference. On imbalanced data it changes the answer completely.

Here is the same grid, on the same data, with only the scoring string changed. The dataset is 4,000 rows with a 4% positive rate, like a fraud or default problem.

scoringWinning CWinning class_weightBest score
accuracy1None0.9642
roc_auc10None0.7281
average_precision0.01None0.3468
recall0.01balanced0.6150

Four metrics, four different winners. C ranges across three orders of magnitude and class_weight flips, purely from which string you passed to scoring. The accuracy figure of 0.9642 is the least useful number on that table: with a 4% positive rate, predicting "negative" every time scores about 0.96.

If you are tuning on rare-event data, decide the metric before you decide the grid.

Worked example

What does this look like end to end?

python
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)     # hold out FIRST

pipe = Pipeline([("scale", StandardScaler()),             # inside the pipeline
                 ("clf", SVC(random_state=42))])

grid = {"clf__C": [0.1, 1, 10], "clf__gamma": ["scale", 0.01]}
cv = StratifiedKFold(5, shuffle=True, random_state=42)

gs = GridSearchCV(pipe, grid, cv=cv, scoring="roc_auc", n_jobs=-1)
gs.fit(X_tr, y_tr)

print(gs.best_params_)
print(pd.DataFrame(gs.cv_results_)[["params", "mean_test_score", "std_test_score"]])
print("held-out:", gs.score(X_te, y_te))                  # the number you report

The order of those steps is the article in miniature. Hold out the test set first, put every learned step inside the pipeline, inspect the spread rather than just the winner, and report the held-out score. gs itself is already refitted on all the training data, so gs.predict(X_new) works directly.

Honest limits

When should you not tune at all?

Four situations where tuning is the wrong use of your time, or will mislead you.

1. Your features or labels are the real problem. Tuning moves a score by fractions of a percent. Fixing a mislabelled target, a leaking feature or a missing data source moves it by much more. Tune last, not first.

2. The gap between configurations is inside the noise. If first and second place differ by less than std_test_score, you have not found a better model. You have found a better random seed. Widen the range of values rather than refining the grid around a winner that is not really winning.

3. You cannot afford the fits. A grid that takes nine hours and improves cross-validated accuracy from 0.9758 to 0.9769 has cost more than it returned. Decide the budget first, then size the grid to fit it.

4. You assume the newest method is faster. Successive halving, available as HalvingGridSearchCV, trains many candidates on small subsets and promotes the survivors. It is genuinely useful on large datasets with expensive models. On the small problem used here it was slower and worse: 45.0 seconds against the full grid's 33.4, for a best score of 0.9512 against 0.9626. With 455 training rows, the early rounds are too noisy to pick good candidates and the overhead is not repaid. Measure it on your own data before adopting it.

Finally, a tuned model is still only as good as its evaluation. For choosing the model family in the first place, which usually matters more than tuning it, see decision tree vs random forest and the difference between random forest and XGBoost.

Next steps

What should you do next?

  • Take one model you have already built and wrap it in a Pipeline, with every preprocessing step inside. Re-run your cross-validation and compare the score to what you had before. If it drops, you have just found a leak.
  • Print len(ParameterGrid(grid)) * n_folds before every search. If the number surprises you, cut the grid or switch to RandomizedSearchCV.
  • Report the held-out score, not best_score_, and state the fold-to-fold standard deviation next to it so the reader knows how precise it is.
FAQ

Frequently asked questions

What is GridSearchCV in scikit-learn?

It is a class that performs an exhaustive search over the parameter values you specify, scoring every combination by cross-validation. After fitting, it exposes best_params_, best_score_ and a best_estimator_ that has been refitted on all the training data.

What is the difference between grid search and random search?

Grid search tries every combination in the grid, while random search samples a fixed number of combinations. For the same time budget, random search usually explores more distinct values of the parameters that matter, which is why it often matches grid search in a fraction of the time.

Why do I need a Pipeline with GridSearchCV?

Because preprocessing fitted outside cross-validation has already seen the validation fold, which inflates the score. In the noise example in this article, doing feature selection outside the pipeline produced 0.755 on data with no signal in it at all.

What does the double underscore mean in a parameter grid?

It separates the pipeline step name from the parameter name, so clf__C means the C parameter of the step named clf. A single underscore will not be recognised.

Is best_score_ the accuracy I should report?

No. It is the cross-validated score of the configuration chosen because it scored highest, so the selection itself biases it upward. Report a score computed on data the search never saw, and state how noisy that estimate is.

How many combinations is too many for a grid search?

It depends on how long one fit takes, so calculate it rather than guessing: len(ParameterGrid(grid)) * n_folds gives the number of model fits. Multiply by the time for a single fit, and if the answer does not fit your budget, use RandomizedSearchCV.

Sources

Sources