CS代考程序代写 deep learning python algorithm CSCC11 Introduction to Machine Learning, Winter 2021 Assignment 2, Due Thursday, February 25, 10am

CSCC11 Introduction to Machine Learning, Winter 2021 Assignment 2, Due Thursday, February 25, 10am
This assignment makes use of material from week 3 to week 5 (specifically Chapter 9.6). To begin the program- ming component, download a2.tgz from the course website and untar it. A directory A2 will be created; please don’t change its structure, nor the headers of the python methods therein. Please read the assignment handout carefully. Submission instructions are at the end.
We prefer you ask questions on Piazza. Otherwise, for your first point of contact, please reach out to your TA, Mustafa Ammous, through email: mustafa.ammous@mail.utoronto.ca.
Written Component
1) Understanding Binary Class Logistic Regression
In this question, we investigate the assumptions and limitations of the binary class logistic regression model. Suppose we have two classes with labels c1 and c2, we can express the posterior probability of class c1 given input x as
P (c1|x) = 1 = g(wT x), (1) 1 + e−wT x
where w is the weight vector with bias included and g is the Sigmoid function. Written Problems.
1. First, let’s investigate the decision boundary of a binary class logistic regression model. Recall that the decision boundary is the set of points where P(c1|x) = 0.5 (i.e. when the decision function α(x) = 0). Show that when P(c1|x) = 0.5, the decision boundary α(x) = 0 is a linear function.
2. Knowing that logistic regression has a linear decision boundary, what datasets would this model perform poorly on? In this case, performing poorly means the model is unable to classify all the training data correct. Let’s look at a toy example – the XOR (Exclusive OR) dataset. XOR is a binary input Boolean function that outputs TRUE when both inputs are the same and outputs FALSE otherwise. The dataset is represented as the table below. Our goal is to predict the output y ∈ {0, 1} given the input vector x=[x1,x2]T,wherex1,x2 ∈{0,1}.
x
x1 x2 y 000 011 101 110
(a) Assume we are using binary class logistic regression on the XOR dataset. What is the maximum classification accuracy we can obtain? Explain.
(b) As in basis function regression, we can apply basis functions to create a more sophisticated model. Consider the following feature map (basis functions) on the inputs:
ψ1(x)  x1 
ψ(x) = ψ2(x) =  x2 . (2)
ψ3 (x) x1 x2

Then, we can express the posterior probability of class c1 given input x as P (c1|x) = g(wT ψ(x)), where w = [w1, w2, w3]T is the weight vector of the model. Note that we exclude the bias term in this question. Specify the conditions and provide an example for the weight vector w such that this model perfectly classifies the XOR dataset.
2) Multiclass Logistic Regression
In this question we will consider multi-class logistic regression. We will begin by extending the formulation of 2-class logistic regression to the multi-class case. For background, see Chapter 9.6 of the online lecture notes.
We will denote input feature vectors by x, and let’s assume the first element of x is 1 to allow for a bias term; i.e., x = (1, x1, …, xd)T . For now, suppose there are 2 classes, with class labels 1 and 2. And let c denote the class variable. In these terms, the conditional probability of class 1, given the data, is simply
p(c=1|x) = p(c=1)p(x|c=1) (3) p(c=1)p(x|c=1) + p(c=2)p(x|c=2)
For the purposes of logistic regression, we define a mapping from an input feature vector to what we might call unnormalized probabilities as follows:
ewkTx = αp(c=k)p(x|c=k).
for some unknown constant α, where k is either 1 or 2 in our 2-class case. Such unnormalized probabilities are just an exponentiated linear functions of the inputs (with the addition of the bias term which is the first element of the weight vector). We call them unnormalized because we don’t know the value of α, and hence the exponentiated inner products ewkT x don’t sum to one like a probability distribution should.
For two classes the model is specified by two weight vectors, w1 and w2. And according to the model, Eqn. (3) can be rewritten as follows
e w 1T x 1 p(c=1|x,w1:2) = ew1Tx + ew2Tx = 1+e(w2−w1)Tx .
This is immediately recognizable as the sigmoidal function g(·) from Eqn. (39) in Chapter 9.6 of the online lecture notes, but here it is applied to (w1 − w2)T x instead of wT x. In other words, in the online notes on 2-class logistic regression, the weights we learned were equal to the difference of the weights used here, i.e., w1 − w2, which defines the normal vector to the decision boundary (w1 − w2)T x = 0.
Now we’re ready to look at the more general case with K classes. Assume again that inputs are x = (1, x1, …, xd)T . And for each of K classes, let there be a weight vector, wk = (bk, wk,1, …, wk,d)T . Assuming c denotes the class variable, we can write the probability for class c = k, where 1 ≤ k ≤ K, as
p(c = k) p(x | c = k)
p(c=k|x) = 􏰅Kl=1p(c=l)p(x|c=l) (4)
As above, the logistic regression model defines a parametric mapping from inputs to unnormalized probabilities. Given the model parameters, w1:K , the model specifies that
e w kT x
p(c=k|x,w1:K) = 􏰅Kl=1ewlTx (5)
1
= 1 + 􏰅l∈{1,…K}k e(wl−wk)T x . (6)
The sum in the denominator in Eqn. (6), l takes on all values between 1 and K, except k. Note that while Eqn. (6) more closely resembles the sigmoidal function in 2-class logistic regression, it will be more convenient to use

Eqn. (5) in what follows. The function in Eqn. (5), which maps K score values (i.e., the inner products wkT x) to probabilities, is used often in deep learning; in neural network jargon it is called a softmax function.
Now let’s consider the objective for learning. As with 2-class logistic regression we are going to use a log loss. Given training data {(xi, yi)}Ni=1, assumed to be IID, we form the loss under the model as
N
E(w1:K) = −log􏰇p(yi|xi,w1:K).
i=1
In the lectures and notes on the 2-class case we manipulated the form of the log probability of the model in terms of the sigmoid function with the class labels being either 0 or 1. We can do the same thing here but it’s just a little trickier. Rather than assume that yi takes on a value from 1 to K, instead we let yi be a so-called one-hot binary vector. That is, we define yi to be a vector of length K whose elements are 0 or 1. When the correct class for the ith data point is k, then all elements of yi are 0 except for the kth; i.e.,
􏰂 1 when k is the correct class yi,k = 0 otherwise
With this one-hot label encoding we can express the negative log likelihood of the training data as follows:
NK
− log 􏰇 􏰇 p(yi,k | xi, w1:K )yi,k
E(w1:K ) =
= − 􏰆􏰆 yi,k logp(yi,k |xi,w1:K)
i=1 k=1 NK
i=1 k=1
Next we need to derive the form of the gradients of E so we can figure out the necessary conditions for the optimal values of the parameters w1:K . To this end, just as we defined the sigmoid function for 2-class logistic regression, here we use a similar notation to define the softmax function: i.e.,
ezk
σ(z1:K,k)=􏰅K ez . (7)
l
l=1
With this notation, and defining zi,k = wkT xi, for k = 1…K, we can express the loss as
NK
E(w1:K) = − 􏰆􏰆 yi,k logσ(zi,1:K, k) . (8)
i=1 k=1
We will solve the learning problem by minimizing E using gradient descent to find the weight vectors. But we
still need to find the gradient first.
Written Problems.
1. Findthegradientofσ(z1:K,k)inEqn.(7)withrespecttozj:
∂σ(z1:K, k). (9)
∂zj
Hint: Consider the cases when k = j and k ̸= j. You may find it helpful to look at the structure of the
gradient in the 2-class case in Eqn. (46) in Chapter 9.6 of the online lecture notes. 2. Find the gradient of the log likelihood for a single point (x, y) with respect to wj :
∂ 􏰆K
∂w yk log σ(z1:K , k) (10)
j k=1

3. Find the gradient of the loss with respect to wj :
∂E(w1:K). (11)
∂wj
Hint: Use the results above. And the gradient should have a form similar to Eqn. (48) in Chapter 9.6 of
the online lecture notes.
4. Now, suppose we have D dimensional inputs and K classes. For each of K classes, let there be a weight vector, wk = (bk, wk,1, …, wk,d)T . If we include regularization, with a (diagonal) Gaussian prior, the negative log-posterior becomes, up to an additive constant,
􏰃N􏰄
Eˆ(w1:K) = −log p(w1:K)􏰇p(y=yi|xi,w1:K) , (12)
i=1
where p(w1:K ) is the joint distribution over K indepdendent Gaussian densities with the same diagonal
covariance. That is,
p(w1:K) =
􏰇K 􏰀 1 􏰀 w kT C − 1 w k 􏰁 􏰁
((2π)(D+1)βαD)1/2 exp − 2
. (13)
k=1
where the covariance matrix is given by C = diag(β, α, α, . . . , α) ∈ R(D+1)×(D+1). Here, α denotes the
variance of the prior on each of the weights, and β is the prior variance on the bias term. Usually we don’t
want a strong prior on the bias term so β ≫ α. Derive ∂Eˆ for this regularized objective function. ∂wk
Hint: Your negative log-posterior should have form Eˆ (w1:K ) = E (w1:K ) + E2 (w1:K ), where E2 (w1:K ) is the regularization term.
Programming Component
You are asked to implement a logistic regression classification algorithm (we provide gradient descent code). The starter code is in directory A2. You will then apply this algorithm on four datasets, each comprises a training set and a test set. You should train your algorithm on the training data, then evaluate performance on the test set. The test performance should be measured as the average 0-1 test loss; i.e., once you’ve fit a model to the training data, evaluate it by counting the number of correctly labelled test points and dividing the count by the size of the test set.
Datasets: Inthestarterfilethereisadatasetsdirectory.Inthatdirectorythereareseveralpicklefilesforthe different datasets, as described below.
• Generic datasets: There are three generic datasets, generic_.pkl where = {1, 2, 3}, all of which have the same format. Given 2D, real-valued measurements (features), we want to classify whether a test point is class 0 or class 1 for the first two datasets, and class 0, class 1, or class 2 for the third dataset. Each pickle file contains four arrays, train_X, train_y, test_X, and test_y. The arrays train_X ∈ R100×2 and train_y ∈ R100×1 form 100 input/output pairs of the training set. The arrays test_X ∈ R50×2 and test_y ∈ R50×1 form 50 input/output pairs of the test set.
• Wine dataset: The wine dataset, wine.pkl, has 13 attributes: Alcohol, Malic acid, Ash, Alcalinity of ash, Magnesium, Total phenols, Flavanoids, Nonflavanoid phenols, Proanthocyanins, Color intensity, Hue, OD280/OD315 of diluted wines and Proline (i.e. 13D measurements/features). Given the 13D measure- ments, we want to classify whether the wine class is wine 0, wine 1, or wine 2 (i.e. 3 classes). The file contains four arrays, train_X, train_y, test_X, and test_y. The arrays train_X ∈ R148×13 and train_y ∈ R148×1 form 148 input/output pairs of the training set. The arrays test_X ∈ R30×13 and test_y ∈ R30×1 form 30 input/output pairs of the test set.