GTU 3150713 Python for Data Science Winter 2025 Solved Paper & Solutions

Advantages of Python for Data Science

1. Rich Scientific Ecosystem: Comprehensive specialized libraries for manipulation (Pandas, NumPy), visualization (Matplotlib, Seaborn), and modeling (Scikit-Learn, PyTorch, TensorFlow).
2. Simple Syntax & High Readability: Clean, intuitive syntax allows data scientists and analysts to focus on algorithmic logic and statistical modeling rather than boilerplate code.
3. Seamless Integration: Effortlessly interfaces with C/C++ backends for high-performance computation and connects with big data frameworks (Apache Spark, Hadoop, SQL databases).
4. Active Open-Source Community: Massive community support providing pre-built packages, active documentation, and state-of-the-art tools.

Python Program: Longest Word and Its Length

Output:

Longest word: machinelearning
Length: 15

Explanation: The max() built-in function evaluates the list using key=len, traversing elements in $O(n)$ time complexity to identify the longest string.

Covariance vs. Correlation in Exploratory Data Analysis (EDA)

Both metrics quantify the directional linear association between two continuous random variables $X$ and $Y$, but they provide distinct mathematical and practical interpretations:

Why It Is Essential to Analyze Both in EDA

1. Scale Sensitivity vs. Standardization:

• Covariance depends directly on the units of measurement. For instance, measuring height in meters versus millimeters scales covariance by $1,000$, making it impossible to determine the true strength of relationship.
• Correlation ($r$) normalizes covariance by the product of standard deviations ($sigma_X sigma_Y$), bounding values strictly between $-1$ and $+1$, allowing direct comparison between features with different units (e.g., age in years vs. salary in dollars).

2. Mathematical Utility in Linear Algebra & Modeling:

• Covariance is required to construct the Covariance Matrix ($Sigma$), which is fundamental for Principal Component Analysis (PCA) to calculate eigenvalues and preserve absolute variance.
• Correlation provides the Correlation Matrix ($R$), crucial during EDA feature selection to detect multicollinearity (pairs where $|r| > 0.85$) to prevent unstable regression coefficients.

3. Complementary Diagnostic Value:

• While covariance indicates whether variables move together in raw physical magnitude, correlation reveals the relative tightness and predictive reliability of the linear relationship.

Python Function: Character Frequency Count

Output:

{'d': 1, 'a': 2, 't': 1, ' ': 1, 's': 1, 'c': 2, 'i': 1, 'e': 2, 'n': 1}

Explanation: The function iterates through each character in $O(n)$ time. The dictionary .get(char, 0) idiom safely handles new characters without raising a KeyError.

The Data Science Pipeline

Detailed Steps

1. Problem Definition: Translate a business question into an analytical or machine learning problem (e.g., classification, regression, or clustering) and define evaluation metrics (RMSE, F1-score).
2. Data Acquisition: Gather relevant datasets from heterogeneous sources including relational databases (SQL), RESTful APIs, web scraping, and distributed logs.
3. Data Cleaning & Preprocessing:

• Handle missing values (imputation or removal).
• Address outliers using IQR or z-scores.
• Clean anomalous data types and encode categorical variables.

4. Exploratory Data Analysis (EDA) & Feature Engineering:

• Visualize distributions, correlations, and feature interactions using Matplotlib/Seaborn.
• Construct new predictive domain features, apply scaling (MinMaxScaler, StandardScaler), and perform dimensionality reduction.

5. Model Building & Validation:

• Train candidate algorithms (Linear Models, Random Forests, XGBoost).
• Evaluate using cross-validation and optimize hyperparameters using GridSearchCV or Optuna.

6. Deployment & Monitoring:

• Deploy model endpoints via REST APIs (FastAPI/Flask) or containerized microservices (Docker).
• Monitor model drift and latency in production.

Advantages of Jupyter Notebook

1. Interactive Literate Computing: Combines live executable code, rich text explanations (Markdown), LaTeX mathematical formulas, and inline plots within a single reproducible document.
2. Cell-by-Cell Execution: Allows data scientists to run code incrementally, inspecting intermediate data structures in memory without re-running long-running computations.
3. Instant Visual Feedback: Displays dataframes, charts, and interactive widgets directly below the calling code cell.
4. Easy Collaboration & Export: Can be shared across teams in .ipynb, HTML, or PDF formats and integrates natively with GitHub.

One-Hot Encoding for Text Representation

Given text snippet: "The quick brown fox"

Step 1: Tokenization and Vocabulary Construction

Extract unique tokens in alphabetical or appearance order:

Vocabulary size $|V| = 4$.

Step 2: Binary Vector Mapping

Each unique word is represented as a 4-dimensional binary vector containing a single $1$ at its unique index and $0$ elsewhere:

Representation of the Snippet:

• "The" $\to [1, 0, 0, 0]$
• "quick" $\to [0, 0, 0, 1]$
• "brown" $\to [0, 1, 0, 0]$
• "fox" $\to [0, 0, 1, 0]$

HTML Parsing with Beautiful Soup

Beautiful Soup is a Python library for pulling structured data out of HTML and XML files by navigating and searching the parse tree.

Extraction Code Example

Output:

Page Title: UnFold Prep Academic Hub
H1 Header: Computer Science Engineering
H2 Headers: ['Semester 5 Subject List']

Extracted Hyperlinks:
- Python for Data Science -> https://unfoldprep.com/pyds
- Computer Networks -> https://unfoldprep.com/cn
- Algorithms Design -> https://unfoldprep.com/ada

Importance of Pie Chart and Box Plot in Data Visualization

1. Pie Chart:

• Role: Visualizes proportional part-to-whole compositions of categorical variables with 3–6 distinct classes.
• Value: Instantly communicates relative percentage market share, budget splits, or survey proportions.

2. Box Plot (Box-and-Whisker):

• Role: Displays the 5-number summary: Minimum, First Quartile ($Q_1$), Median ($Q_2$), Third Quartile ($Q_3$), and Maximum.
• Value in EDA:
• Shows distributional skewness and variance without assuming normality.
• Automatically identifies statistical outliers (points beyond $1.5 \times \text{IQR}$).

Magic Functions in Jupyter / IPython

Magic functions are specialized command shortcuts built into the IPython kernel that extend standard Python capabilities, prefixed with % (line magics) or %% (cell magics).

Two Widely Used Magic Functions

1. %timeit:

• Measures the precise execution time of a Python statement by running it thousands of times in a loop to produce mean and standard deviation.
• Example: %timeit [x**2 for x in range(1000)]

2. %matplotlib inline:

• Directs the IPython environment to render Matplotlib figures directly below the code cell within the notebook document rather than opening separate window dialogs.

Creating an Undirected Graph with NetworkX and Matplotlib

Explanation of Code

1. nx.Graph(): Instantiates a container for undirected edges where an edge $(u, v)$ is identical to $(v, u)$.
2. add_nodes_from() & add_edges_from(): Batch-inserts nodes and connecting graph edges.
3. nx.spring_layout(): Positions nodes using the Fruchterman-Reingold force-directed algorithm.
4. nx.draw(): Renders the network using Matplotlib with customized node colors, edge styling, and bold node labels.

Role of the Chi-Square ($\chi^2$) Test in Data Analysis

The Chi-Square Test is a non-parametric hypothesis test that analyzes categorical variables:

1. Test of Independence:

• Evaluates whether two categorical variables are statistically independent or associated.
• Example: Testing whether customer subscription renewal (Yes/No) is independent of user device type (Mobile/Desktop).

2. Goodness-of-Fit Test:

• Determines whether observed sample frequencies match expected theoretical distributions.

3. Feature Selection for ML:

• In classification tasks, $\chi^2$ feature selection ranks categorical features based on dependency with the target label, eliminating irrelevant features.

Skewness and Kurtosis

1. Skewness (Asymmetry):

• Measures the asymmetry of a distribution around its mean.
• Zero Skew: Symmetric distribution (Normal distribution).
• Positive (Right) Skew: Long right tail; Mean $>$ Median $>$ Mode (e.g., household incomes).
• Negative (Left) Skew: Long left tail; Mean $<$ Median $<$ Mode (e.g., age at natural death).

2. Kurtosis (Tailedness / Peakedness):

• Measures the weight of the tails relative to the center (presence of outliers).
• Mesokurtic (Kurtosis = 3): Standard normal distribution.
• Leptokurtic (Kurtosis > 3): Sharp central peak with heavy, fat tails (higher probability of extreme outliers, common in financial asset returns).
• Platykurtic (Kurtosis < 3): Flatter peak with thin tails.

Methods for Slicing Rows and Columns in Pandas

1. Label-Based Slicing using .loc[]

Slices rows and columns using their index and column labels (both start and end labels are inclusive):

# Slicing rows 'r2' through 'r4' and columns 'Name' through 'Department'
subset1 = df.loc['r2':'r4', 'Name':'Department']

2. Position-Based Slicing using .iloc[]

Slices using integer zero-based index positions (end index is exclusive, standard Python slice syntax):

# Slicing first 3 rows (0, 1, 2) and first 2 columns (0, 1)
subset2 = df.iloc[0:3, 0:2]

3. Boolean Conditional Masking

Selects rows meeting specific logical criteria:

# Select all employees in IT with Salary > 60000
subset3 = df[(df['Department'] == 'IT') & (df['Salary'] >= 60000)]

4. Direct Column Selection and Slicing

# Select specific columns by list
cols = df[['Name', 'Salary']]
# Slice rows using index range
first_two_rows = df[:2]