-- Practice Midterm 2 Solutions
2) Charles MacKay & Nick Mauthes
Cross validation
A technique used to evaluate the error of the trained model when our dataset is not large. There might be a case your model performs badly on one test set, but good on another test set. We can divide the dataset into subsets, and aggregate the results of each, by taking an average. This will be a better predictor of overall accuracy of the model. We can make a claim the model works well if we do cross validation and we still have a high average accuracy.
Exhaustive, we consider all possible ways to divide the dataset, if we partition a p subset.
Example:
leave-p-out , cycle over all possible p subsets, train on the data excluding p-subset, test on the p-subset. This is a combination of all possible subsets like n choose p
inexhaustive – doesn’t consider all possible subsets
examples:
k-fold – divide the dataset into k equal subsets. For S1, …, Sk, for I = 1 to k, train on all but Si, and test on Si. Aggregate the results
Repeated random sub sampling – randomly repeat m times, choose a subset of size p as the test set, train on all other data, test on subset p.
2) Charles MacKay & Nick Mauthes
Cross validation
A technique used to evaluate the error of the trained model when our dataset is not large. There might be a case your model performs badly on one test set, but good on another test set. We can divide the dataset into subsets, and aggregate the results of each, by taking an average. This will be a better predictor of overall accuracy of the model. We can make a claim the model works well if we do cross validation and we still have a high average accuracy.
Exhaustive, we consider all possible ways to divide the dataset, if we partition a p subset.
Example:
leave-p-out , cycle over all possible p subsets, train on the data excluding p-subset, test on the p-subset. This is a combination of all possible subsets like n choose p
inexhaustive – doesn’t consider all possible subsets
examples:
k-fold – divide the dataset into k equal subsets. For S1, …, Sk, for I = 1 to k, train on all but Si, and test on Si. Aggregate the results
Repeated random sub sampling – randomly repeat m times, choose a subset of size p as the test set, train on all other data, test on subset p.