程序代写代做代考 algorithm data science Introduction to information system

Introduction to information system

Logistic Regression

Deema Abdal Hafeth

Bowei Chen

CMP3036M/CMP9063M Data Science

2016 – 2017

Objectives

 Limitation of Linear Regression

 Basic Concepts

 Logit Function

 Odds Ratio

 Logistic Function

 Logistic Regressions

 Simple Logistic Regression

 Multiple Logistic Regression

References

• James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An introduction

to statistical learning. Springer. (Chapter 4)

• Abu-Mostafa, Yaser S., Malik Magdon-Ismail, and Hsuan-Tien Lin. Learning

from data. AMLBook, 2012. (Chapter 3)

• 𝑦 – Response variable

• 𝑥 – Predictor variable(s)

• 𝛽 – Intercept and slop

• 𝜀 – Error term
Linear

Regression

Simple

𝑦 = 𝑓 𝑥 = 𝛽0 + 𝛽1𝑥 + 𝜀

Multiple

𝑦 = 𝛽0 + 𝛽1𝑥𝑖,1 + 𝛽2𝑥𝑖,2 + …+ 𝛽𝑝𝑥𝑖,𝑝 + 𝜀

Price House size

1 420 5850

2 385 4000

3 495 3060

4 605 6650

5 610 6360

6 660 4160

7 660 3880

8 690 4160

9 838 4800

10 885 5500

… … …

Dataset

For this new house with size 4050 (sq ft), can we predict what is it the rent price?

Simple Linear Regression

Price House size

1 420 5850

2 385 4000

3 495 3060

4 605 6650

5 610 6360

6 660 4160

7 660 3880

8 690 4160

9 838 4800

10 885 5500

…

Response variable

Independent variable (x): Ppredictors
variable, feature or explanatory variable

Predictor

4050

𝑦 ≔ 𝑓 𝑥 = 𝛽0 + 𝛽1𝑥 + ε

Slop Intercept

𝑥

𝑦

𝑦

For this new house with size 4050 (sq ft), 4 bedrooms and 2 bathrooms,

can we predict what is it the rent price?

There Are Other Features of Houses

Price House size Bedrooms Bathrms Stories Driveway Recroom Fullbase

1 420 5850 3 1 2 1 0 1

2 385 4000 2 1 1 1 0 0

3 495 3060 3 1 1 1 0 0

4 605 6650 3 1 2 1 1 0

5 610 6360 2 1 1 1 0 0

6 660 4160 3 1 1 1 1 1

7 660 3880 3 2 2 1 0 1

8 690 4160 3 1 3 1 0 0

9 838 4800 3 1 1 1 1 1

10 885 5500 3 2 4 1 1 0

… … … … … … … … …

Multiple Linear Regression

Simple expression:

𝑦𝑖 = 𝛽0 + 𝛽1𝑥𝑖,1 + 𝛽2𝑥𝑖,2 + ⋯+ 𝛽𝑝𝑥𝑖,𝑝 + 𝜀𝑖,

Matrix expression:

𝒚 = 𝒙𝜷 + 𝜺,

where

𝐲 =

𝑦1
⋮
𝑦𝑛

, 𝜷 =

𝛽0
𝛽1
⋮
𝛽𝑝

, 𝒙 =

1 𝑥1,1 ⋯ 𝑥1,𝑝
⋮ ⋮ ⋱ ⋮
1 𝑥𝑛,1 ⋯ 𝑥𝑛,𝑝

, 𝜺 =

𝜀1
⋮
𝜀𝑛

Limitation of Linear Regression

Price Fullbase

1 420 1

2 385 0

3 495 0

4 605 0

5 610 0

6 660 1

7 660 1

8 690 0

9 838 1

10 885 0

… … …

Housing dataset

Response

variable
Predictor

Logit Function and Odds Ratio

The logit function of 𝑝, where 𝑝 is between 0 and 1, can be expressed as

logit 𝑝 = log
𝑝

1 − 𝑝
= log 𝑝 − log 1 − p

𝑝

1−𝑝
is called odds ratio

If 𝑝 = 0, logit 𝑝 → −∞

If 𝑝 = 1, logit 𝑝 → ∞

Logistic Function

The logit function is the inverse of logistic function. If we let 𝛼 = logit 𝑝 , then

logistic 𝛼 = logit−1 𝑝 =
1

e−𝛼 + 1
=

𝑒𝛼

1 + 𝑒𝛼

𝑒𝛼

1 + 𝑒𝛼

log
𝑝

1 − 𝑝

Simple Logistic Regression

The logit of the underlying probability 𝑝𝑖 is a linear function of the predictors

logit 𝑝𝑖 = 𝛽0 + 𝛽1𝑥𝑖 ,

then

𝑝𝑖 =
1

1 + 𝑒−(𝛽0+𝛽1𝑥𝑖)
=

𝑒𝛽0+𝛽1𝑥𝑖

1 + 𝑒𝛽0+𝛽1𝑥𝑖
.

We should predict yi = 1 when 𝑝𝑖 ≥ 0.5 and y𝑖 = 0 when 𝑝𝑖 < 0.5. This means guessing 1 whenever 𝛽0 + 𝛽1𝑥𝑖 is non-negative, and 0 otherwise. So logistic regression gives us a linear classifier. The decision boundary separating the two predicted classes is the solution of 𝛽0 + 𝛽1𝑥𝑖 = 0. Simple Logistic Regression Parameters Estimation We use maximum likelihood estimation (MLE) method to estimate parameters’ values. Simple, the likelihood function can be written as ℒ 𝛽0 , 𝛽1 = ℙ(𝒚 ∣ 𝒙, 𝛽0, 𝛽1) = 𝑝𝑖 𝑦𝑖 𝑛 𝑖=1 (1 − 𝑝𝑖) 1−𝑦𝑖 Then 𝛽0 , 𝛽1 = argmax log{ℒ 𝛽0 , 𝛽1 } We can quickly obtain 𝛽0, 𝛽1 in R for logistic regression. The algorithmic solution of parameters estimation is not required but I hope you could search it after class. 𝑛 observations 𝑦𝑖 = 0 or 1 𝑒𝛽0+𝛽1𝑥𝑖 1 + 𝑒𝛽0+𝛽1𝑥𝑖 bodysize survive 1 1.747326 0 2 1.752239 0 3 1.806183 0 4 1.812122 0 5 1.816628 0 6 1.847038 1 7 1.860676 0 8 1.897639 1 9 1.902124 0 10 1.904108 0 … … … Multiple Logistic Regression The logit of the underlying probability 𝑝𝑖 is a linear function of the predictors logit 𝑝𝑖 = 𝒙𝒊 𝑻𝜷, where 𝜷 = 𝛽0 𝛽1 ⋮ 𝛽𝑝 , 𝒙 = 1 𝑥1,1 ⋯ 𝑥1,𝑝 ⋮ ⋮ ⋱ ⋮ 1 𝑥𝑛,1 ⋯ 𝑥𝑛,𝑝 . Then 𝑝𝑖 = 𝑒𝒙𝒊 𝑻𝜷 1 + 𝑒𝒙𝒊 𝑻𝜷 . 𝑥𝑛 bodysize survive age 1 1.773196 0 34 2 1.775586 0 64 3 1.826180 0 30 4 1.832622 0 11 5 1.838073 0 22 6 1.850233 1 68 7 1.850588 0 13 8 1.865037 1 48 9 1.883509 0 22 10 1.886488 0 20 … … … … Summary  Limitation of Linear Regression  Basic Concepts  Logit Function  Odds Ratio  Logistic Function  Logistic Regressions  Simple Logistic Regression  Multiple Logistic Regression Thank You bchen@Lincoln.ac.uk mailto:bchen@Lincoln.ac.uk