Module-1: EDA / Data Analytics using Python [14 Sessions] [using Pandas, Numpy, Matplotlib & Seaborn]

Day 1

Section-1: Data Analysis using Pandas

1. Structural Perspective of Datatypes

2. What is Data Analysis
3. What is Pandas
4. What is a DataFrame
5. What is Jupyter Notebook
6. How to use Anaconda
7. Program: Reading tabular data file (data with rows and columns) into pandas – demo on read_table()
8. Program: To display 1st 7 rows only
9. Program: Explicitly specifying the delimiter symbol and solving header row problem
10. Program: Explicitly specifying the column names
11. Program: demo on read_csv()

Day 2

1. Program: Demo on type() and selecting a pandas series from a DataFrame
2. Program: Creating a new series/column in a DataFrame
3. Program: Filtering rows of a pandas DataFrame by column value
4. Program: using the loc indexer – conditional lookup
5. Program: Applying multiple filter criteria to a pandas DataFrame
6. Program: Reading from a selection of columns
7. Program: Demo on dropping a column or a row from display, also covers usage of “axis” parameter
8. Program: Demo on mean() method

Day 3

1. Program: Demo on “groupby” in pandas
2. Program: “groupby” Ex2
3. Creating a new DataFrame from a collection of Series
4. Program: Creating DataFrame from a Python Dictionary
5. Program: Demo on using multiple aggregate functions at one go
6. Program: Series Ex1
7. Program: Retrieving alternate rows and retrieving rows in reverse order
8. Program: Checking for null values in a DataFrame
9. Program: To figure out whether there are any null values in a DataFrame
10. Program: To figure out how many null values are present in a DataFrame
11. Program: Dealing with null values/Missing data
12. Program: Concatenation of DataFrames
13. Program: Setting a column as Index
14. loc() vs iloc()

Day 4

1. Program: Merging of DataFrames based on single key
2. Program: Joining of DataFrames
3. Program: Getting unique elements of a column in a DataFrame
4. Program: Getting count of unique elements of a column in a DataFrame
5. Program: Getting count of each unique element of a column in a DataFrame
6. Program: Demo on apply() on a DataFrame
7. Program: Demo on apply() on a lambda expression
8. Program: Knowing column names of a DataFrame
9. Program: Sorting a DataFrame

Day 5

1. Program: Writing to a CSV file

2. Program: Reading from an Excel Spreadsheet file
3. Program: Writing to an Excel Spreadsheet file scrapping
4. Program: Reading from an HTML table (Web Scrapping in Pandas)
5. Program: Pandas Data Analysis – Analysis on San Francisco Employees Salaries Data
6. Program: Pandas Data Analysis - Analysis on E-Commerce Purchase Data

Day 6

1. Change the column names of a DataFrame

2. Changing name of a Particular Column in a DataFrame
3. Setting Column names while creating a DataFrame
4. Changing Row Index
5. Renaming Row Index
6. Add a new Column with single value to a Dataframe
7. Add a new Row with same value to a Dataframe
8. Using iloc indexer to update a value
9. Sorting by index column
10. Sorting in descending order based on a column
11. Sorting using two columns
12. Grouping by multiple columns in a DataFrame
13. Pandas Data Pre-processing for Optimal Model Execution
14. Demo on Replacing the Missing Data – with the Mode
15. Demo on fillna()’s parameter: method = ‘ffill’
16. Demo1 on Processing Duplicate Data
17. Demo2 - Find the duplicate value in the specific column data of the DataFrame
18. Demo3 - Remove duplicate rows based on one or more columns

Day 7

1. Data Feature Engineering – Encoding Categorical Variables
2. Demo on Integer Encoding
3. Demo on One-Hot Encoding
4. Demo on Dummy Variable Encoding
5. Mean Target Encoding
6. Demo on Mean Target Encoding
7. Data Cleaning – Handling Outliers – method 1
8. Data Cleaning – Handling Outliers – method 2
9. Data Cleaning – Handling Inconsistent data
10. Exploratory Data Analysis – Scores and Rankings

Day 8

Section-2: Numpy

1. Why use NumPy Arrays
2. NumPy Datatypes
3. Program: NumPy Demo1
4. Program: NumPy Demo2 – transforming the 1D array into 2D array
5. Program: NumPy avoids copies wherever possible – demo
6. Program: If we need a true copy of an array – demo
7. Program: To demonstrate that in NumPy arrays, the operations are propagated to the individual elements.
8. Program: Indexing Demo1
9. Program: Array Indexing Demo3 – Generating arrays using arange()
10. Program: Generating arrays using arange() – Ex2
11. Program: To generate zeros
12. Program: To generate ones
13. Program: Demo on linspace()
14. Program: To generate random numbers between 0 to 1
15. Program: Generating random integers
16. Program: Finding index of max value and min value in an array
17. Program: Knowing datatype of an array
18. Program: Slicing / Indexing 2-D arrays
19. NumPy Operations
20. Program: Performing arithmetic operations between two 1-D arrays
21. Program: Applying a scalar value to a 1-D array
22. Program: Using Universal array Functions on 1-D arrays
23. Program: To retrieve non-zero elements from an array

Day 9

1. Program: Aliasing the arrays()
2. Program: Viewing arrays (shallow copying)
3. Program: Copying arrays (deep copying)
4. Program: Attributes of an Array – ndim
5. Program: Attributes of an array – shape
6. Program: Attributes of an array – size, itemsize, dtype, nbytes
7. Program: Demo on reshape()
8. Program: Demo on flatten()
9. Program: Matrix object in numpy – demo1
10. Program: Getting diagonal elements of a Matrix
11. Program: Finding Max and Min elements in a Matrix
12. Program: Finding Sum and Average of Matrix elements
13. Program: Product of Matrix elements
14. Program: Sorting of Matrix elements
15. Program: Transpose of a Matrix
16. Program: Addition of two matrices
17. Program: Multiplication of two matrices
18. Creating a 3-D array
19. Demo on Converting datatype
20. Demo on np.random.randn()
21. Comparing Python Numpy Arrays Vs Python Lists: Processing Speed
22. Demo on repeat() and tile()
23. Demo on using -1 in reshape()
24. Demo on np.random.seed()
25. Demo on np.unique()

Day 10

Section-3: Data Visualizations using Matplotlib, Seaborn Chapter-1 (Matplotlib)

1. What is Data Visualization
2. Why should we go for Data Visualization
3. Variables (Column names in a table)
4. Types of Variables
5. Qualitative variables
6. Quantitative variables
7. Continuous variable
8. Discrete variable
9. Basic Visualization Types
10. What is Matplotlib & How to Install of Matplotlib
11. Program: A Simple plot using Matplotlib
12. Program: To Demonstrate creating multiple plots on the same canvas
13. Program: Settings colors, line widths, line types and markers
14. Program: Demo on Bar Chart
15. Program: Demo on Histogram Demo
16. Program: Demo on Scattered Plot
17. Program: Demo on Pie Chart
18. Demo on Bar Chart with Multiple colors
19. Demo on BoxPlot

Day 11

Chapter-2 (Seaborn)

1. What is Seaborn, Seaborn Vs Matplotlib & How to install Seaborn
2. What does the Seaborn Library Provides
3. Program: Demo on loading a seaborn built-in dataset – “tips”
4. Program: Demo on distplot
5. Program: Demo on jointplot
6. Program: Demo on pairplot
7. Program: Pairplot with a “hue” argument
8. Program: pairplot with palette argument
9. Program: Demo on barplot
10. Program: Demo on using the estimator argument for the barplot
11. Program: Demo on countplot
12. Program: Demo on Boxplot
13. Program: Demo on heatmap
14. Program: Demo on clustermap

Day 12

1. Program: Demo on PairGrid
2. Program: Demo on facetGrid
3. Program: Demo on controlling size and color of seaborn plots
4. Program: Demo on line plot
5. Program: Seaborn Data Visualizations on the titanic dataset

Day 13

Data Analytics Project-1: Big Basket Ecommerce Data Analysis

Day 14

Data Analytics Project-2: IMDB Movie Dataset Analysis

Day 15

Section-1: Mathematics Chapter-1: Algebra

1. What is Algebra
2. What is Linear Algebra
3. Vectors & Matrices
4. Solving simple Equations with One Arithmetic Operation
5. Solving simple Equations with Two Arithmetic Operations
6. Polynomials
7. Functions in Algebra
8. Calculating Slope of any line and Calculating the Distance between any two points on a line
9. Slope-Intercept Form (y = mx + b)
10. Sequences

Chapter-2: Concept of Functions in Maths

1. Introduction
2. Rules for Functions
3. Graph of a Function
4. Types of Functions
5. Arrow & Graph representation of a Function
6. Composite Functions
7. Trigonometric Function

Day 16

Chapter-3: Exponential & Logarithm Functions

1. Exponents
2. Logarithms
3. Logarithmic Function
4. Exponential Plot: Draw the Graph of the Function y = 3x-1 + 2
5. Exponential Plot: Draw the Graph of the Function y = 2-x-2
6. Logarithm Plot: Draw the Graph of the Following Logarithmic Function
7. Euler’s constant
8. Natural Logarithm

Chapter-4 Calculus (Integrations and Differentiations)

1. What is Calculus and why should we study it
2. Applications of Calculus
3. Branches of Calculus
4. Limits
5. Derivatives (Differential Calculus)
6. Partial Derivatives
7. Integrals (Integration)

Section-2: Statistics & Probability Chapter-1: Introduction to Statistics

1. What is Statistics
2. Why Statistics is needed
3. Statistics Vs Mathematics
4. Types of Statistics
5. What is Descriptive Statistics
6. Descriptive Vs Inferential Statistics
7. Variables
8. Qualitative variables
9. Quantitative variables
10. Continuous variables
11. Discrete variables
12. Scales/Levels of Measurement
13. Sample space
14. Univariate Vs Bivariate Data
15. Correlation

Chapter-2: Central Limit Theorem

1. Measures of Central Tendency Set-1: Arithmetic Mean, Weighted Arithmetic Mean, Harmonic Mean and Geometric Mean
2. Measures of Central Tendency Set-2: Median and Mode
3. Measures of Dispersion: Range, Interquartile Range, Variance & Standard Deviation

Day 18

Chapter-3

1. Population and Sample
2. Sampling Techniques
3. Sampling Bias
4. Normal Distribution
5. Sampling and Estimation
6. Correlation in Statistics
7. How is the strength of the correlation calculated?
8. Correlation Vs Causality
9. Standard Scores (Z-Score)
10. Patterns in Data with Charts and Graphs
11. Tabular displays
12. Segmented Bar Charts
13. Probability Density Function
14. What is Hypothesis

Chapter-4: Probability

1. Statistical experiments
2. Sample Space
3. Types of Events
4. Combinations, Permutations and Counting Events
5. Probability
6. Sum of Probabilities
7. Complementary Event
8. How to compute probability: law of large numbers
9. Conditional Probability
10. Bayes Theorem (aka Bayes Rule)
11. Random Variables
12. Theorems of Probability

Day 19

Chapter-5: Probability Distribution and its types Round-1: Types of Probability Distributions

1. What is Probability Distribution
2. Normal Distribution
3. Discrete Probability Distribution
4. List of Continuous Probability Distribution
5. List of Discrete Probability Distribution

Round-2

1. Discrete Distributions Vs Continuous Distribution
2. Standard Normal Distribution
3. Normal Distribution Vs Standard Normal Distribution

Round-3: Continuous Probability Distribution

1. Continuous Uniform Distribution
2. Exponential Distribution
3. Chi-Squared Distribution
4. Student’s T Distribution / T Distribution

Round-4: Discrete Probability Distribution

1. The Poisson Distribution
2. Bernoulli Distribution
3. Discrete Uniform Distribution
4. Binomial Distribution
5. Negative Binomial Distribution
6. Geometric Distribution

Day 20

Chapter-6: Hypothesis Testing

1. What is Hypothesis Testing
2. Principles of Hypothesis Testing
3. When is Hypothesis Testing required
4. Null Hypothesis and Alternative Hypothesis
5. Null Hypothesis Vs Alternative Hypothesis
6. p-value (Probability Value)
7. Chi-square test
8. Chi-square goodness of fit test (test for One-way table)
9. Chi-square test of independence / Chi-square test of association (test for Two–way table)
10. t-test
11. When to use a t-test
12. Types of t-tests
13. One-Sample t-test
14. Independent Two-Sample t-test
15. Paired Two-Sample t-test
16. Equal Variance t-Test
17. Un-Equal Variance t-Test

Day 21

Chapter-7: ANOVA Test / Hypothesis Test of the Frequencies

1. Introduction to ANOVA
2. Terminologies related to ANOVA
3. Grand Mean
4. Hypothesis
5. Between Group Variability
6. Within Group Variability / error group / error variance
7. One-Way ANOVA
8. Two-Way ANOVA

Chapter-8: Estimation Theory

1. Point estimation Vs Interval estimation
2. Unbiased Estimator
3. Efficiency Estimator
4. Consistency Estimator

Day 22

Chapter-1: Introduction to Machine Learning

1. What is Machine Learning
2. What is Machine Learning – algorithms point of view
3. Statistics Terminology
4. Application Areas of Machine Learning
5. Popular Machine Learning Algorithms
6. Connection among Machine Learning and different kinds of study
7. Types of Machine Learning Algorithms
8. Supervised Learning
9. UnSupervised Learning
10. Reinforcement Learning
11. Classification Supervised learning
12. Regression Supervised learning
13. Clustering
14. Recommendation Systems / Association
15. Machine Learning Workflow Process
16. What is scikit-learn
17. Features of scikit-learn
18. Important functions and classes of scikit-learn library

Day 23

Chapter-2: Supervised ML - Linear Regression

1. Statistical Modeling
2. Regression Analysis
3. Linear Regression Analysis (for Regression problem)
4. Finding best fit line in Linear Regression
5. Installation of Scikit-Learn package
6. Program: Linear Regression Analysis with Python – to predict house pricing
7. Normal Equation Vs SGD
8. Different types of SGD
9. Program: Predicting House Prices using SGDRegressor
10. Program: Linear Regression Project - to predict yearly amount spent via Mobile App and Website by Ecommerce customers

Day 24

Chapter-3: Supervised ML - Logistic Regression

1. Logistic Regression Analysis (for classification problems) and confusion matrix
2. Program: Logistic Regression with Python - to predict whether someone has survived or not (yes or no) in the Titanic ship mishap. Also covers data Cleansing like: Changing the datatypes of values, dealing with missing data like dropping, imputing with other values & Converting Textual data to numerical data

Day 25

1. Understand “Accuracy”, “Precision”, “Recall” & F1-Score (Classification Metrics)
2. Program: Logistic Regression Project – predict whether or not an internet user will click on an Advertisement
3. Maximum Likelihood Estimation (MLE) in the context of logistic regression
4. Formula for Sigmoid Function
5. Maximum Likelihood Estimation (MLE) in the context of logistic regression

Day 26

Chapter-4: Feature scaling in ML

1. What is Feature Scaling in ML
2. Why Feature Scaling is required
3. Algorithms requiring Feature Scaling
4. Types of Feature Scaling techniques
5. Program: Demo1 on Feature Scaling - MinMaxScalar
6. Program: Demo2 on Feature Scaling – StandardScalar
7. Project: Feature Scaling with Logistic Regression – for multiclass classification – wine classification
8. Softmax function formula

Day 27

Chapter-5: Supervised ML - KNN

1. K Nearest Neighbors (KNN) (Classification algorithm)
2. Program: KNN with Python Demo1 - working with anonymous data
3. Program: KNN Project – working with UCI dataset – classify type of breast cancer for a set of patients

Day 28

Chapter-6: Supervised ML - Decision Trees and Random Forests

1. Decision Trees and Random Forests Introduction
2. Gini Index
3. Program: Decision Trees and Random Forests using Python – classify whether a kyphosis (a type deformation) condition was present or absent after the operation

Day 29

1. Program: Decision Trees - Classify whether an employee would get salary > 100K $ or not based on the features provided (also covers “Entropy”)

Day 30

Chapter-7: Machine Learning related some fundamental theory concepts

1. What is Regression
2. Linear Regression Mathematics
3. Cost function
4. Logistic Regression Mathematics / Sigmoid Function
5. Gradient Descent – Step by Step
6. Bias Vs Variance
7. Bias Variance Trade-Off
8. Overfitting & Underfitting in ML
9. Ensemble Learning
10. Cross Validation
11. Imbalanced Datasets
12. Parameters Vs Hyperparameters

Day 31

Chapter-8: Principal Component Analysis (PCA

1. Principal Component Analysis
2. Program: Principal Component Analysis – Demo

Day 32

Chapter-9: Supervised ML – SVM (Support Vector Machines)

1. SVM – Support Vector Machines (for classification analysis)
2. Program: SVM Demo using Python – to predict whether the tumor of breast cancer would be either malignant or benign
3. Program: SVM Project – to classify iris flowers dataset

Day 33

Chapter-10: Supervised ML – Naïve Bayes

Round-1: Converting Text to Features

1. Word Counts with CountVectorizer
2. Program: Demo on CountVectorizer
3. Word Frequencies with TF-IDF Vectorizer
4. Program: Demo on TF-IDF Vectorizer

Round-2: Document/Text Classification using Naïve Bayes

1. Principle of Naïve Bayes Classification
2. Types of Naïve Bayes Classification
3. Program: Classify Yelp Reviews into 1-star or 5-star categories based on the text content in the reviews

1. Program: Document Classification using Naïve Bayes Algorithm
2. Program: Building a Pipeline

Chapter-11: Supervised ML – Time Series Analysis & Forecasting

1. What is Time-Series Data
2. Why should we analyze Time-Series data?
3. Applications of Time-Series Data Analysis and Forecasting
4. Program: Time-Series data analysis Demo1
5. How to use Google Colab

Day 36

Chapter-12: UnSupervised ML – Clustering – K-Means

1. What is Cluster Analysis
2. What is K-Means Clustering
3. K-Means Clustering Algorithm Steps
4. K-Means Clustering Algorithm explanation with an example
5. Program: Demo on K-Means Cluster Analysis on IRIS Dataset
6. Program: K-Means Clustering Project – to cluster Universities into two groups – Private and Public

Day 37

Chapter-13: UnSupervised ML – Hierarchical Clustering

1. What is Hierarchical Clustering
2. What is a Dendrogram
3. Drawing of a Dendrogram – step by step
4. Applications of Hierarchical Clustering
5. Approaches of Hierarchical Clustering
6. Agglomerative clustering linkage algorithm (Cluster Distance Measure)
7. Choosing number of clusters using dendrogram
8. Where to cut a dendrogram
9. Program: To cluster customers based on their spending scores using Hierarchical Clustering

Day 38

Chapter-14: UnSupervised ML – Clustering – DBSCAN

1. Why do we need another clustering algorithm?
2. What is DBSCAN Clustering
3. How does DBSCAN Clustering work
4. Advantages of DBSCAN
5. Disadvantages of DBSCAN
6. Program: Un-Supervised Clustering using DBSCAN Algorithm

Day 39

1. Program: Un-Supervised Clustering using DBSCAN Algorithm - For Multi-Clustering Analysis – (choosing number of centers/clusters)
2. Program: Un-Supervised Clustering using DBSCAN Algorithm - for Circles Classification Problems

Day 40

Chapter-15: UnSupervised ML – Market Basket Analysis

1. What is Market Basket Analysis / Association Rule Mining?
2. What is Apriori Algorithm
3. Common ways to measure association – Support, Confidence & lift
4. Program: Market Basket Analysis Implementation in Python using the Apriori Algorithm (Association Rule Mining) – Demo1
5. Program: Market Basket Analysis Demo2 – also covers finding top 5 items sold

Day 41

Chapter-16: Recommendation Engines

1. Introduction
2. Collaborative Filtering and Content-based Filtering
3. Cold Start
4. Project: Recommendation Systems using Python Pandas – Recommending Similar Movies for the new users

Chapter-17: Machine Learning advanced Miscellaneous Topics

1. Demo on Linear Regression – using galton’s parent and child heights – dataset - without train test split
2. Demo on Linear Regression – using Boston’s house prices dataset – including both mathematical and Visual Diagnostics – also covers in-sample and out-sample prediction analysis
3. Demo on Logistic Regression – with Cutoff Value and Sensitivity & Specificity . Also covers ROC Curve and AUC Score
4. Decision Trees – Some important Hyperparameters
5. Demo on Decision Trees – for Classification – with Hyperparameters
6. Demo on Decision Trees &ndas