GTU 3170724 Machine Learning (ML) Summer 2026 Solved Paper & Solutions

Well-Posed Learning Problem

According to Tom Mitchell (1997), a computer program is said to learn from experience if it satisfies three formal criteria:

"A computer program is said to learn from Experience $E$ with respect to some class of Tasks $T$ and Performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$."

The Three Fundamental Components:

1. Task ($T$): The specific real-world problem to be executed (e.g., Playing Chess).
2. Performance Measure ($P$): The quantitative metric used to assess the system's success (e.g., Percentage of games won against human grandmasters).
3. Training Experience ($E$): The historical observation data or practice trials the model learns from (e.g., Playing $100,000$ practice games against itself).

Classification vs Regression

End-to-End Machine Learning Workflow

Developing and deploying a machine learning solution for any real-world problem follows a structured lifecycle:

Key Steps and Activities:

1. Problem Definition & Framing:

• Translate business objectives into a concrete ML problem: Supervised vs Unsupervised vs Reinforcement learning.
• Define key metrics for success (e.g., F1-score $\ge 0.92$, inference latency $< 50\text{ ms}$).

2. Data Collection & Ingestion:

• Aggregate raw data from databases, REST APIs, web scrapers, IoT sensors, or public repositories.
• Ensure data quality, volume, and balance across target labels.

3. Exploratory Data Analysis (EDA) & Data Preprocessing:

• Handling Missing Values: Imputation via mean/median/mode or removal.
• Outlier Treatment: Detection via Z-score / IQR and winsorization.
• Data Normalization: Min-Max scaling ($[0, 1]$) or Standard Z-Score scaling ($\mu=0, \sigma=1$).

4. Feature Engineering & Selection:

• One-hot encoding for nominal categories; target encoding for high-cardinality features.
• Creating domain-specific interaction features.
• Selecting optimal feature subsets via correlation matrices or PCA to prevent curse of dimensionality.

5. Model Selection, Training & Cross-Validation:

• Partition data into Train ($70\%$), Validation ($15\%$), and Test ($15\%$) sets.
• Train candidate baseline models (Linear/Logistic, Tree ensembles like XGBoost, Neural Nets).
• Perform $k$-fold cross-validation to assess generalization.

6. Hyperparameter Tuning & Evaluation:

• Tune parameters using Grid Search, Random Search, or Bayesian Optimization.
• Evaluate against held-out test data using unseen edge cases.

7. Production Deployment & Monitoring:

• Containerize model using Docker and expose via high-performance REST APIs (FastAPI/Flask).
• Monitor for Data Drift (input distributions changing over time) and Concept Drift (statistical relationships shifting), triggering automated retraining pipelines.

Outliers in Machine Learning

An outlier is an extreme data observation that deviates markedly from the overall pattern of distribution of the rest of the sample data, typically caused by measurement errors, sensor faults, or rare natural events.

How to Handle Outliers:

1. Detection:

• Z-Score Method: Data points with $|Z| > 3$ ($|\frac{x - \mu}{\sigma}| > 3$) in normal distributions.
• IQR (Interquartile Range) Rule: Values below $Q_1 - 1.5 \times \text{IQR}$ or above $Q_3 + 1.5 \times \text{IQR}$.

2. Handling Strategies:

• Trimming / Removal: Drop outlier records if verified as corrupted sensor data.
• Winsorization (Capping): Cap extreme values to the $1^{\text{st}}$ and $99^{\text{th}}$ percentiles.
• Mathematical Transformation: Apply log or square-root transforms to compress long-tail distributions.
• Use Robust Models: Choose tree-based models (Random Forest, XGBoost) that are naturally resilient to outliers.

Central Tendency vs Data Dispersion

Example Comparison:

Consider two student batches with exam scores out of 100:

• Batch A: $\{48, 50, 52\} \implies \text{Mean} = 50$, Range $= 4$, Std Dev $\sigma = 1.63$ (Very tight, low dispersion).
• Batch B: $\{10, 50, 90\} \implies \text{Mean} = 50$, Range $= 80$, Std Dev $\sigma = 32.66$ (Wide spread, high dispersion).
• Insight: Both batches have the identical central tendency ($ext{Mean} = 50$), but Batch B has vastly higher data dispersion!

Types of Data in Machine Learning

Data is categorized into distinct hierarchical types that dictate the choice of preprocessing, visualization, and ML algorithms:

1. Qualitative (Categorical) Data

Represents characteristics, labels, or groups without direct mathematical magnitude:

• Nominal Data:
• Categories with no intrinsic order or ranking.
• Mathematical operations ($+, -$) are meaningless; compared only by equality ($=$ or $\ne$).
• Examples: Gender (Male, Female), Marital Status (Single, Married), Colors (Red, Blue).
• Preprocessing: One-Hot Encoding.
• Ordinal Data:
• Categories that follow a clear, meaningful rank or natural order, but the exact difference between ranks is not quantified.
• Examples: Education (High School < Bachelor's < Master's < PhD), Customer Satisfaction (Poor, Average, Good, Excellent).
• Preprocessing: Ordinal / Integer Encoding.

2. Quantitative (Numerical) Data

Represents measurable numeric values where arithmetic operations ($+, -, \times, /$) are mathematically valid:

• Discrete Data:
• Countable whole numbers (integers); cannot be subdivided meaningfully.
• Examples: Number of children in a family ($0, 1, 2$), Number of customer service calls received per hour.
• Continuous Data:
• Measurable quantities along an unbroken real interval; can take any fractional value within a range.
• Examples: Temperature ($36.6^\circ\text{C}$), Body weight ($68.45\text{ kg}$), Car speed ($85.4\text{ km/h}$).
• Sub-types:
• Interval Scale: Meaningful differences, but arbitrary zero point (e.g., Celsius temperature; $0^\circ\text{C} \ne$ no heat).
• Ratio Scale: Has an absolute, true zero point (e.g., Height, Weight, Salary; $$0$ means absence of money).

3. Structural Classifications:

• Structured Data: Tabular rows and columns (SQL databases, CSV files).
• Unstructured Data: Free-form text, images, video, audio recordings (processed via Deep Learning).
• Semi-Structured Data: Key-value pairs, JSON files, XML trees.

Central Limit Theorem (CLT)

The Central Limit Theorem states that:

Given an arbitrary population with a finite mean $\mu$ and finite standard deviation $\sigma$, the distribution of the sample mean ($\bar{X}$) of $n$ independent and identically distributed (i.i.d.) random variables approaches a Normal (Gaussian) Distribution as the sample size $n$ becomes sufficiently large ($n \ge 30$), regardless of the underlying distribution of the original population.

Mathematical Formulation:

Standardized form:

• Significance in ML: Justifies the use of Gaussian assumptions in linear regression, hypothesis testing, confidence intervals, and parametric estimation.

Feature Construction vs Feature Extraction

Parametric Discrete Probability Distributions

A discrete probability distribution describes random variables that take on countable values ($x \in \mathbb{Z}$), governed by fixed parameters:

1. Bernoulli Distribution

Models a single trial that has only two mutually exclusive outcomes: Success ($x = 1$) or Failure ($x = 0$).

• Parameter: $p$ (probability of success, $0 \le p \le 1$).
• Probability Mass Function (PMF):

• Mean & Variance:

• ML Application: Foundation of Logistic Regression and Binary Cross-Entropy loss.

2. Binomial Distribution

Models the number of successes ($k$) in a fixed sequence of $n$ independent and identical Bernoulli trials.

• Parameters: $n$ (number of trials), $p$ (success probability).
• PMF:

• Mean & Variance:

• ML Application: A/B testing conversions, evaluation of classification accuracy over sample test sets.

3. Poisson Distribution

Models the count of independent, rare events occurring within a fixed interval of time or space, where events occur at a known constant average rate $\lambda$.

• Parameter: $\lambda > 0$ (average rate of occurrence).
• PMF:

• Mean & Variance:

• ML Application: Poisson Regression, web server request modeling, call center queue forecasting.

Techniques to Prevent Overfitting in Decision Trees:

1. Pre-Pruning (Early Stopping):

• Halt tree growth before it perfectly classifies training noise:
• Set maximum tree depth (max_depth).
• Require a minimum number of samples to split an internal node (min_samples_split).
• Require a minimum number of samples at leaf nodes (min_samples_leaf).

2. Post-Pruning (Cost-Complexity Pruning):

• Allow the tree to grow fully, then prune subtrees that add little predictive value using cost-complexity parameter $\alpha$:

3. Ensemble Methods:

• Transition from a single high-variance decision tree to Random Forests (bagging) or Gradient Boosted Trees (XGBoost/LightGBM) that average out individual tree errors.

Methods to Improve Linear Regression Accuracy

1. Feature Engineering & Interaction Terms:

• Introduce non-linear relationships using Polynomial Features ($x^2, x^3$) and multiplicative interaction features ($x_1 \cdot x_2$).

2. Outlier Detection & Removal:

• Extreme leverage points pull the Ordinary Least Squares (OLS) regression line away from true trends. Remove or winsorize high-residual outliers.

3. Log / Power Transforms:

• If target or feature distributions are heavily right-skewed, apply $\log(y)$ or Box-Cox transformations to linearize variance.

4. Regularization (Ridge / Lasso):

• Add penalty terms to shrink regression coefficients, eliminating multicollinearity and reducing overfitting:
• Lasso (L1): Performs automated feature selection by zeroing out coefficients.
• Ridge (L2): Handles collinear features effectively.

5. Feature Scaling:

• Standardize all features via Z-score scaling to ensure stable gradient updates.

Naïve Bayes Classifier

The Naïve Bayes classifier is a probabilistic supervised learning algorithm based on Bayes’ Theorem, with the "naïve" assumption that all features $X = (x_1, x_2, \dots, x_n)$ are mutually conditionally independent given the class label $y$:

Since the denominator is identical for all candidate classes, the classification rule picks the maximum a posteriori (MAP) class:

Step-by-Step Example (Play Tennis?):

Given historical weather records:

• Prior Probabilities: $P(\text{Play}=\text{Yes}) = \frac{9}{14} \approx 0.64$, $P(\text{Play}=\text{No}) = \frac{5}{14} \approx 0.36$.

Predict for Query: $X = (\text{Outlook}=\text{Sunny}, \; \text{Wind}=\text{Strong})$

Step 1: Compute Likelihoods:

• For Class Yes:
• $P(\text{Sunny} \mid \text{Yes}) = \frac{2}{9}$
• $P(\text{Strong} \mid \text{Yes}) = \frac{3}{9} = \frac{1}{3}$
• Score($\text{Yes}$) $= P(\text{Yes}) \times P(\text{Sunny} \mid \text{Yes}) \times P(\text{Strong} \mid \text{Yes})$:

• For Class No:
• $P(\text{Sunny} \mid \text{No}) = \frac{3}{5}$
• $P(\text{Strong} \mid \text{No}) = \frac{3}{5}$
• Score($\text{No}$) $= P(\text{No}) \times P(\text{Sunny} \mid \text{No}) \times P(\text{Strong} \mid \text{No})$:

Step 2: Final Prediction:

• Since $\text{Score}(\text{No}) = 0.1286 > \text{Score}(\text{Yes}) = 0.0476$, the model predicts:

• Laplace Smoothing: If a feature-class combination never occurs in training ($P(x_i \mid y) = 0$), add pseudocount $\alpha = 1$ to prevent zeroing out the entire product: $P(x_i \mid y) = \frac{\text{count} + 1}{N + K}$.

Statement: "Association rule mining is a probabilistic method of unsupervised learning."

Justification:

1. Deterministic, Not Probabilistic:

• Association Rule Mining (e.g., Apriori Algorithm, FP-Growth) is a deterministic, frequency-based pattern-mining technique, not a probabilistic model.
• It relies on counting exact empirical co-occurrences of itemsets in transaction databases using set-theoretic threshold metrics:
• Support: $\text{Supp}(A \to B) = \frac{\text{Frequency}(A \cup B)}{\text{Total Transactions}}$
• Confidence: $\text{Conf}(A \to B) = \frac{\text{Frequency}(A \cup B)}{\text{Frequency}(A)}$
• Lift: $\text{Lift}(A \to B) = \frac{\text{Conf}(A \to B)}{\text{Supp}(B)}$

2. It does not estimate underlying parametric probability distributions or use probabilistic graphical models (unlike Bayesian networks or Gaussian Mixture Models).

Domains Where Deep Learning Outperforms Traditional ML

Traditional ML relies heavily on manual hand-crafted feature engineering that fails on raw, high-dimensional unstructured data. Deep Learning excels in:

1. Computer Vision & Image Processing:

• Convolutional Neural Networks (CNNs) automatically learn hierarchical spatial features (edges $\to$ textures $\to$ object parts).
• Applications: Autonomous vehicle perception, Medical imaging (MRI tumor detection), Facial recognition.

2. Natural Language Processing (NLP) & Large Language Models:

• Transformer architectures with self-attention capture deep contextual semantics across long sequences.
• Applications: Machine translation, Automated summarization, ChatGPT, Speech-to-text.

3. Generative Modeling & Creative AI:

• Diffusion models and GANs synthesize photorealistic images and audio.

4. Speech & Audio Recognition:

• Recurrent and Conformer networks process raw acoustic waveforms with near-human transcription accuracy.

Comparison of Neural Network Activation Functions

Activation functions introduce non-linearity into neural networks, enabling them to learn arbitrary complex continuous functions (Universal Approximation Theorem).

Comparative Analysis: