Create a Hyperparameter Tuning Plan for Machine Learning Models

You are a machine learning optimization specialist. I need a structured hyperparameter tuning plan for my model. **Model and Data Context:** - Algorithm: [ALGORITHM e.g., XGBoost, Random Forest, Neural Network, SVM] - Dataset size: [DATASET_SIZE] - Number of features: [NUM_FEATURES] - Problem type: [PROBLEM_TYPE e.g., binary classification, multi-class, regression] - Primary optimization metric: [METRIC e.g., AUC-ROC, RMSE, F1-macro] - Computational resources: [RESOURCES e.g., single CPU, multi-core, GPU cluster] - Time budget for tuning: [TIME_BUDGET e.g., 2 hours, overnight, 1 week] **Please provide:** 1. **Hyperparameter Inventory:** List all tunable hyperparameters for my chosen algorithm. For each parameter, provide: - Parameter name and description - Recommended search range (min, max) - Scale type (linear, logarithmic, categorical) - Priority level (high/medium/low impact on performance) - Interaction effects with other parameters 2. **Search Strategy Recommendation:** Compare Grid Search, Random Search, Bayesian Optimization (e.g., Optuna, Hyperopt), and Successive Halving/Hyperband for my scenario. Recommend the optimal strategy given my time and compute budget with a clear rationale. 3. **Phased Tuning Plan:** Design a multi-phase tuning approach: - Phase 1: Coarse search over high-priority parameters - Phase 2: Fine-grained search around promising regions - Phase 3: Final refinement and interaction tuning Specify the number of iterations, parameter ranges, and expected duration for each phase. 4. **Early Stopping and Efficiency:** Describe techniques to reduce wasted computation, including early stopping criteria, warm starting, and pruning unpromising trials. 5. **Implementation Code:** Provide a complete Python implementation using [PREFERRED_FRAMEWORK e.g., Optuna, scikit-learn GridSearchCV, Ray Tune] with logging, best parameter tracking, and visualization of the optimization history. 6. **Validation and Overfitting Prevention:** Explain how to detect if hyperparameter tuning is overfitting to the validation set and what safeguards to implement (e.g., holdout test set, nested CV). 7. **Results Documentation Template:** Provide a template for recording tuning experiments including parameters tested, scores achieved, runtime, and final selected configuration.