Introduction to Machine Learning (CS 5710)¶
Assignment 2¶
Due by 26th September (Thursday) 11:59pm¶
In this assignment, you need to complete the following four sectoins:
- KNN
- Linear regression
- Logistic regression
- Regularization
Submission guideline¶
- Open this notebook with jupyter notebook and start writing codes
- After finishing writing your codes, click the Save button at the top of the Jupyter Notebook.
- Please make sure to have entered your UCM ID below.
- Select Cell -> All Output -> Clear. This will clear all the outputs from all cells (but will keep the content of all cells).
- Select Cell -> Run All. This will run all the cells in order.
- Once you’ve rerun everything, select File -> Download as -> HTML or PDF via LaTeX
- Look at the HTML/PDF file and make sure all your solutions are there, displayed correctly.
- Zip BOTH the HTML/PDF file and this .ipynb notebook (updated with your codes). Rem
- Submit your zipped file.
# Please Write Your UCM ID Here:
Section 1. KNN [30 pts]¶
The following KNN assignment is modified from Stanford CS231n. Please complete and hand in this completed worksheet.
In [1]:
# Run some setup code for this notebook.
from __future__ import print_function
import random
import numpy as np
from data_utils import load_CIFAR10
import matplotlib.pyplot as plt
# This is a bit of magic to make matplotlib figures appear inline in the notebook
# rather than in a new window.
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
# Some more magic so that the notebook will reload external python modules;
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2
Download data:¶
Once you have the starter code (regardless of which method you choose above), you will need to download the CIFAR-10 dataset. Run the following from the assignment1 directory:
cd data ./get_datasets.sh
In [2]:
# Load the raw CIFAR-10 data.
cifar10_dir = 'data/cifar-10-batches-py'
# Cleaning up variables to prevent loading data multiple times (which may cause memory issue)
try:
del X_train, y_train
del X_test, y_test
print('Clear previously loaded data.')
except:
pass
X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)
# As a sanity check, we print out the size of the training and test data.
print('Training data shape: ', X_train.shape)
print('Training labels shape: ', y_train.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)
Training data shape: (50000, 32, 32, 3)
Training labels shape: (50000,)
Test data shape: (10000, 32, 32, 3)
Test labels shape: (10000,)
In [3]:
# Visualize some examples from the dataset.
# We show a few examples of training images from each class.
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
num_classes = len(classes)
samples_per_class = 7
for y, cls in enumerate(classes):
idxs = np.flatnonzero(y_train == y)
idxs = np.random.choice(idxs, samples_per_class, replace=False)
for i, idx in enumerate(idxs):
plt_idx = i * num_classes + y + 1
plt.subplot(samples_per_class, num_classes, plt_idx)
plt.imshow(X_train[idx].astype('uint8'))
plt.axis('off')
if i == 0:
plt.title(cls)
plt.show()
In [4]:
# Subsample the data for more efficient code execution in this exercise
num_training = 5000
mask = list(range(num_training))
X_train = X_train[mask]
y_train = y_train[mask]
num_test = 500
mask = list(range(num_test))
X_test = X_test[mask]
y_test = y_test[mask]
In [5]:
# Reshape the image data into rows
X_train = np.reshape(X_train, (X_train.shape[0], -1))
X_test = np.reshape(X_test, (X_test.shape[0], -1))
print(X_train.shape, X_test.shape)
(5000, 3072) (500, 3072)
To make things much structural, we now put everything together into the KNearestNeighbor class. You don’t need to implement any fucntion in this class now. Later you will need to come back here and implement the asked function, per the instruction.
In [6]:
import numpy as np
class KNearestNeighbor(object):
""" a kNN classifier with L2 distance """
def __init__(self):
pass
def train(self, X, y):
"""
Train the classifier. For k-nearest neighbors this is just
memorizing the training data.
Inputs:
- X: A numpy array of shape (num_train, D) containing the training data
consisting of num_train samples each of dimension D.
- y: A numpy array of shape (N,) containing the training labels, where
y[i] is the label for X[i].
"""
self.X_train = X
self.y_train = y
def predict(self, X, k=1, num_loops=0):
"""
Predict labels for test data using this classifier.
Inputs:
- X: A numpy array of shape (num_test, D) containing test data consisting
of num_test samples each of dimension D.
- k: The number of nearest neighbors that vote for the predicted labels.
- num_loops: Determines which implementation to use to compute distances
between training points and testing points.
Returns:
- y: A numpy array of shape (num_test,) containing predicted labels for the
test data, where y[i] is the predicted label for the test point X[i].
"""
if num_loops == 0:
dists = self.compute_distances_no_loops(X)
elif num_loops == 1:
dists = self.compute_distances_one_loop(X)
elif num_loops == 2:
dists = self.compute_distances_two_loops(X)
else:
raise ValueError('Invalid value %d for num_loops' % num_loops)
return self.predict_labels(dists, k=k)
def compute_distances_two_loops(self, X):
"""
Compute the distance between each test point in X and each training point
in self.X_train using a nested loop over both the training data and the
test data.
Inputs:
- X: A numpy array of shape (num_test, D) containing test data.
Returns:
- dists: A numpy array of shape (num_test, num_train) where dists[i, j]
is the Euclidean distance between the ith test point and the jth training
point.
"""
num_test = X.shape[0]
num_train = self.X_train.shape[0]
dists = np.zeros((num_test, num_train))
for i in range(num_test):
for j in range(num_train):
#####################################################################
# TODO: #
# Compute the l2 distance between the ith test point and the jth #
# training point, and store the result in dists[i, j]. You should #
# not use a loop over dimension. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return dists
def compute_distances_one_loop(self, X):
"""
Compute the distance between each test point in X and each training point
in self.X_train using a single loop over the test data.
Input / Output: Same as compute_distances_two_loops
"""
num_test = X.shape[0]
num_train = self.X_train.shape[0]
dists = np.zeros((num_test, num_train))
for i in range(num_test):
#######################################################################
# TODO: #
# Compute the l2 distance between the ith test point and all training #
# points, and store the result in dists[i, :]. #
#######################################################################
#######################################################################
#######################################################################
return dists
def compute_distances_no_loops(self, X):
"""
Compute the distance between each test point in X and each training point
in self.X_train using no explicit loops.
Input / Output: Same as compute_distances_two_loops
"""
num_test = X.shape[0]
num_train = self.X_train.shape[0]
dists = np.zeros((num_test, num_train))
#########################################################################
# TODO: #
# Compute the l2 distance between all test points and all training #
# points without using any explicit loops, and store the result in #
# dists. #
# #
# You should implement this function using only basic array operations; #
# in particular you should not use functions from scipy. #
# #
# HINT: Try to formulate the l2 distance using matrix multiplication #
# and two broadcast sums. #
#########################################################################
#########################################################################
# END OF YOUR CODE #
#########################################################################
return dists
def predict_labels(self, dists, k=1):
"""
Given a matrix of distances between test points and training points,
predict a label for each test point.
Inputs:
- dists: A numpy array of shape (num_test, num_train) where dists[i, j]
gives the distance betwen the ith test point and the jth training point.
Returns:
- y: A numpy array of shape (num_test,) containing predicted labels for the
test data, where y[i] is the predicted label for the test point X[i].
"""
num_test = dists.shape[0] #num_test has same no of elements as testing data
y_pred = np.zeros(num_test)
for i in range(num_test):
# A list of length k storing the labels of the k nearest neighbors to
# the ith test point.
#########################################################################
# TODO: #
# Use the distance matrix to find the k nearest neighbors of the ith #
# testing point, and use self.y_train to find the labels of these #
# neighbors. Store these labels in closest_y. #
# Hint: Look up the function numpy.argsort. #
#########################################################################
#########################################################################
# TODO: #
# Now that you have found the labels of the k nearest neighbors, you #
# need to find the most common label in the list closest_y of labels. #
# Store this label in y_pred[i]. Break ties by choosing the smaller #
# label. #
#########################################################################
#########################################################################
# END OF YOUR CODE #
#########################################################################
return y_pred
In [7]:
# Create a kNN classifier instance.
# Remember that training a kNN classifier is a noop:
# the Classifier simply remembers the data and does no further processing
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)
We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:
- First we must compute the distances between all test examples and all train examples.
- Given these distances, for each test example we find the k nearest examples and have them vote for the label
Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.
First, open k_nearest_neighbor.py and implement the function compute_distances_two_loops that uses a (very inefficient) double loop over all pairs of (test, train) examples and computes the distance matrix one element at a time.
In [8]:
# Implement compute_distances_two_loops.
# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print(dists.shape)
(500, 5000)
In [9]:
# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()
Second, open k_nearest_neighbor.py and implement the function predict_labels that predicts a label for each test point.
In [10]:
# Now implement the function predict_labels and run the code below:
# We use k = 1 (which is Nearest Neighbor).
y_test_pred = classifier.predict_labels(dists, k=1)
# Compute and print the fraction of correctly predicted examples
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))
Got 137 / 500 correct => accuracy: 0.274000
You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:
In [11]:
y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))
Got 139 / 500 correct => accuracy: 0.278000
In [12]:
# Now lets speed up distance matrix computation by using partial vectorization
# with one loop. Implement the function compute_distances_one_loop and run the
# code below:
dists_one = classifier.compute_distances_one_loop(X_test)
# To ensure that our vectorized implementation is correct, we make sure that it
# agrees with the naive implementation. There are many ways to decide whether
# two matrices are similar; one of the simplest is the Frobenius norm. In case
# you haven't seen it before, the Frobenius norm of two matrices is the square
# root of the squared sum of differences of all elements; in other words, reshape
# the matrices into vectors and compute the Euclidean distance between them.
difference = np.linalg.norm(dists - dists_one, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
print('Good! The distance matrices are the same')
else:
print('Uh-oh! The distance matrices are different')
Difference was: 0.000000
Good! The distance matrices are the same
In [13]:
# Now implement the fully vectorized version inside compute_distances_no_loops
# and run the code
dists_two = classifier.compute_distances_no_loops(X_test)
# check that the distance matrix agrees with the one we computed before:
difference = np.linalg.norm(dists - dists_two, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
print('Good! The distance matrices are the same')
else:
print('Uh-oh! The distance matrices are different')
Difference was: 0.000005
Good! The distance matrices are the same
In [14]:
# Let's compare how fast the implementations are
def time_function(f, *args):
"""
Call a function f with args and return the time (in seconds) that it took to execute.
"""
import time
tic = time.time()
f(*args)
toc = time.time()
return toc - tic
two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print('Two loop version took %f seconds' % two_loop_time)
one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print('One loop version took %f seconds' % one_loop_time)
no_loop_time = time_function(classifier.compute_distances_no_loops, X_test)
print('No loop version took %f seconds' % no_loop_time)
# you should see significantly faster performance with the fully vectorized implementation
Two loop version took 65.921167 seconds
One loop version took 149.449537 seconds
No loop version took 0.807989 seconds
Cross-validation¶
We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.
In [15]:
num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]
X_train_folds = []
y_train_folds = []
################################################################################
# TODO: #
# Split up the training data into folds. After splitting, X_train_folds and #
# y_train_folds should each be lists of length num_folds, where #
# y_train_folds[i] is the label vector for the points in X_train_folds[i]. #
# Hint: Look up the numpy array_split function. #
################################################################################
################################################################################
# END OF YOUR CODE #
################################################################################
# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
################################################################################
# TODO: #
# Perform k-fold cross validation to find the best value of k. For each #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times, #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all #
# values of k in the k_to_accuracies dictionary. #
################################################################################
################################################################################
# END OF YOUR CODE #
################################################################################
# Print out the computed accuracies
for k in sorted(k_to_accuracies):
for accuracy in k_to_accuracies[k]:
print('k = %d, accuracy = %f' % (k, accuracy))
k = 1, accuracy = 0.263000
k = 1, accuracy = 0.257000
k = 1, accuracy = 0.264000
k = 1, accuracy = 0.278000
k = 1, accuracy = 0.266000
k = 3, accuracy = 0.239000
k = 3, accuracy = 0.249000
k = 3, accuracy = 0.240000
k = 3, accuracy = 0.266000
k = 3, accuracy = 0.254000
k = 5, accuracy = 0.248000
k = 5, accuracy = 0.266000
k = 5, accuracy = 0.280000
k = 5, accuracy = 0.292000
k = 5, accuracy = 0.280000
k = 8, accuracy = 0.262000
k = 8, accuracy = 0.282000
k = 8, accuracy = 0.273000
k = 8, accuracy = 0.290000
k = 8, accuracy = 0.273000
k = 10, accuracy = 0.265000
k = 10, accuracy = 0.296000
k = 10, accuracy = 0.276000
k = 10, accuracy = 0.284000
k = 10, accuracy = 0.280000
k = 12, accuracy = 0.260000
k = 12, accuracy = 0.295000
k = 12, accuracy = 0.279000
k = 12, accuracy = 0.283000
k = 12, accuracy = 0.280000
k = 15, accuracy = 0.252000
k = 15, accuracy = 0.289000
k = 15, accuracy = 0.278000
k = 15, accuracy = 0.282000
k = 15, accuracy = 0.274000
k = 20, accuracy = 0.270000
k = 20, accuracy = 0.279000
k = 20, accuracy = 0.279000
k = 20, accuracy = 0.282000
k = 20, accuracy = 0.285000
k = 50, accuracy = 0.271000
k = 50, accuracy = 0.288000
k = 50, accuracy = 0.278000
k = 50, accuracy = 0.269000
k = 50, accuracy = 0.266000
k = 100, accuracy = 0.256000
k = 100, accuracy = 0.270000
k = 100, accuracy = 0.263000
k = 100, accuracy = 0.256000
k = 100, accuracy = 0.263000
In [16]:
# plot the raw observations
for k in k_choices:
accuracies = k_to_accuracies[k]
plt.scatter([k] * len(accuracies), accuracies)
# plot the trend line with error bars that correspond to standard deviation
accuracies_mean = np.array([np.mean(v) for k,v in sorted(k_to_accuracies.items())])
accuracies_std = np.array([np.std(v) for k,v in sorted(k_to_accuracies.items())])
plt.errorbar(k_choices, accuracies_mean, yerr=accuracies_std)
plt.title('Cross-validation on k')
plt.xlabel('k')
plt.ylabel('Cross-validation accuracy')
plt.show()
In [17]:
# Based on the cross-validation results above, choose the best value for k,
# retrain the classifier using all the training data, and test it on the test
# data. You should be able to get above 28% accuracy on the test data.
best_k = 10
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)
y_test_pred = classifier.predict(X_test, k=best_k)
# Compute and display the accuracy
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))
Got 141 / 500 correct => accuracy: 0.282000
Section 2. Linear Regression [25 pts]¶
The following linear regression assignment is modified from Stanford CS229. Please complete and hand in this completed worksheet.
Linear regression with one variable¶
Before starting on any task, it is often useful to understand the data by visualizing it. For this dataset, you can use a scatter plot to visualize the data, since it has only two properties to plot (profit and population). (Many other problems that you will encounter in real life are multi-dimensional and can’t be plotted on a 2-d plot.)
The dataset is loaded from the data file into the variables X and y:
In [18]:
data = np.loadtxt('data/ex1data1.txt', delimiter=",") # read comma separated data
m = data.shape[0] # number of training example
X = data[:,0].reshape(m,1)
y = data[:,1].reshape(m,1)
print (X.shape)
print (y.shape)
(97, 1)
(97, 1)
In [19]:
plt.plot(X,y, 'rx') # Plot the data
plt.xlabel('Population of City in 10,000s')
plt.ylabel('Profit in $10,000s')
plt.show()
In this part, you will fit the linear regression parameters $\theta$ to our dataset using gradient descent.
The objective of linear regression is to minimize the cost function \begin{equation*} J(\theta) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^2 \end{equation*}
where the hypothesis $h_\theta(x)$ is given by the linear mode \begin{equation*} h_{\theta}(x^{(i)}) = \theta^Tx = \theta_0 + \theta_1 x_1 \end{equation*}
Recall that the parameters of your model are the $\theta_j$ values. These are the values you will adjust to minimize cost $J(\theta)$. One way to do this is to use the batch gradient descent algorithm. In batch gradient descent, each iteration performs the update \begin{equation*} \theta_j := \theta_j – \alpha \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)}) x_j^{(i)} \end{equation*}
With each step of gradient descent, your parameters $\theta_j$ come closer to the optimal values that will achieve the lowest cost $J(\theta)$.
As you perform gradient descent to learn minimize the cost function J(θ), it is helpful to monitor the convergence by computing the cost. In this section, you will implement a function to calculate J(θ) so you can check the convergence of your gradient descent implementation.
Your next task is to complete the compute_cost function, which is a function that computes J(θ). As you are doing this, remember that the variables X and y are not scalar values, but matrices whose rows represent the examples from the training set.
In [20]:
def compute_cost(X, y, theta):
m = len(y)
# You need to return the following variables correctly
J = 0 #####################################################################
# Compute the cost of a particular choice of theta #
# You should set J to the cost. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
####################################################################
return J
In [21]:
X = np.concatenate((np.ones((m, 1)), data[:,0].reshape(m,1)), axis=1)
theta = np.zeros((2, 1))
compute_cost(X, y, theta)
Out[21]:
32.072733877455676
You should expect to see a cost of 32.07.
Next, you will implement gradient descent function. The loop structure has been written for you, and you only need to supply the updates to θ within each iteration.
As you program, make sure you understand what you are trying to optimize and what is being updated. Keep in mind that the cost J(θ) is parameterized by the vector θ, not X and y. That is, we minimize the value of J(θ) by changing the values of the vector θ, not by changing X or y.
A good way to verify that gradient descent is working correctly is to look at the value of J(θ) and check that it is decreasing with each step. The starter code calls compute_cost on every iteration and prints the cost. Assuming you have implemented gradient descent and compute_cost correctly, your value of J(θ) should never increase, and should converge to a steady value by the end of the algorithm.
In [22]:
def gradient_descent(X, y, theta, alpha, num_iters):
# GRADIENTDESCENT Performs gradient descent to learn theta
# theta = GRADIENTDESCENT(X, y, theta, alpha, num_iters) updates theta by
# taking num_iters gradient steps with learning rate alpha
# Initialize some useful values
m = len(y)
J_history = []
for iter in range(num_iters):
#####################################################################
# Instructions: Perform a single gradient step on the parameter #
# vector theta. #
# #
# Hint: While debugging, it can be useful to print out the values #
# of the cost function (compute_cost) and gradient here. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
# Save the cost J in every iteration
J = compute_cost(X, y, theta)
J_history.append(J)
return theta, J_history
Now let’s find the parameter θ and plot the linear fit.
In [23]:
print('Running Gradient Descent ...\n')
X = np.concatenate((np.ones((m, 1)), data[:,0].reshape(m,1)), axis=1) # Add a column of ones to x
theta = np.zeros((2, 1)) # initialize fitting parameters
# Some gradient descent settings
iterations = 1500
alpha = 0.01
# gradient descent
theta, J_history = gradient_descent(X, y, theta, alpha, iterations)
print('Theta found by gradient descent: ')
print(theta[0], theta[1])
plt.plot(X[:,1], y, 'rx') # Plot the data
plt.xlabel('Population of City in 10,000s')
plt.ylabel('Profit in $10,000s')
plt.plot(X[:,1], np.dot(X, theta), '-')
plt.show()
Running Gradient Descent ...
Theta found by gradient descent:
[-3.63029144] [1.16636235]
Linear regression with multiple variable¶
In this part, you will implement linear regression with multiple variables to predict the prices of houses. Suppose you are selling your house and you want to know what a good market price would be. One way to do this is to first collect information on recent houses sold and make a model of housing prices.
The file ex1data2.txt contains a training set of housing prices in Portland, Oregon. The first column is the size of the house (in square feet), the second column is the number of bedrooms, and the third column is the price of the house.
In [24]:
data = np.loadtxt('data/ex1data2.txt', delimiter=",") # read comma separated data
m = data.shape[0] # number of training example
X = data[:,0:2].reshape(m,2)
y = data[:,2].reshape(m,1)
By looking at the values, note that house sizes are about 1000 times the number of bedrooms. When features differ by orders of magnitude, first performing feature scaling can make gradient descent converge much more quickly.
In [25]:
def feature_normalize(X):
# FEATURENORMALIZE Normalizes the features in X
# FEATURENORMALIZE(X) returns a normalized version of X where the mean value of each
# feature is 0 and the standard deviation is 1. This is often a good preprocessing
# step to do when working with learning algorithms.
# You need to set these values correctly
X_norm = X
mu = 0
sigma = 0
#####################################################################
# Instructions: First, for each feature dimension, compute the mean #
# of the feature and subtract it from the dataset, #
# storing the mean value in mu. Next, compute the #
# standard deviation of each feature and divide #
# each feature by it's standard deviation, storing #
# the standard deviation in sigma. #
# #
# Note that X is a matrix where each column is a #
# feature and each row is an example. You need #
# to perform the normalization separately for #
# each feature. #
# #
# Hint: You might find the 'mean' and 'std' functions useful. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return X_norm, mu, sigma
Previously, you implemented gradient descent on a univariate regression problem. The only difference now is that there is one more feature in the matrix X. The hypothesis function and the batch gradient descent update rule remain unchanged.
You should complete the function gradientDescentMulti to implement the gradient descent for linear regression with multiple variables.
Make sure your code supports any number of features and is well-vectorized.
In [26]:
X = np.concatenate((np.ones((m, 1)),feature_normalize(data[:,0:2].reshape(m,2))[0]), axis=1)
theta = np.zeros((3, 1))
compute_cost(X, y, theta)
Out[26]:
65591548106.45744
You should expect to see a cost of 65591548106.
Next, you will implement gradient descent function with multiple variable.
In [27]:
def gradient_descent_multi(X, y, theta, alpha, num_iters):
#GRADIENTDESCENTMULTI Performs gradient descent to learn theta
# theta = GRADIENTDESCENTMULTI(x, y, theta, alpha, num_iters) updates theta by
# taking num_iters gradient steps with learning rate alpha
# Initialize some useful values
m = len(y)
J_history = []
for iter in range(num_iters):
#####################################################################
# Instructions: Perform a single gradient step on the parameter #
# vector theta. #
# #
# Hint: While debugging, it can be useful to print out the values #
# of the cost function (compute_cost) and gradient here. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
# Save the cost J in every iteration
J = compute_cost(X, y, theta)
print(J)
J_history.append(J)
return theta, J_history
Now let’s find the parameter θ and plot the linear fit.
In [28]:
alpha = 0.01;
num_iters = 400;
print(np.dot(X,theta).shape)
theta = np.zeros((3, 1))
theta, J_history = gradient_descent_multi(X, y, theta, alpha, num_iters)
(47, 1)
64297776251.62011
63031018305.52132
61790694237.53249
60576236901.991035
59387091739.9886
58222716488.38939
57082580895.8954
55966166445.97885
54872966086.50778
53802483965.89506
52754235175.605446
51727745498.85994
50722551165.380974
49738198612.02588
48774244249.16026
47830254232.6268
46905804241.168976
46000479259.1725
45113873364.59137
44245589521.92844
43395239380.144295
42562443075.371216
41746829038.312386
40948033806.20948
40165701839.264984
39399485341.40871
38649044085.30025
37914045241.46274
37194163211.44539
36489079464.91514
35798482380.58049
35122067090.852936
34459535330.1538
33810595286.77683
33174961458.219242
32552354509.895863
31942501137.15358
31345133930.505222
30759991244.004223
30186817066.683086
29625360896.981297
29075377620.08948
28536627388.13903
28008875503.167988
27491892302.795826
26985453048.54132
26489337816.71971
26003331391.85654
25527223162.557613
25060807019.775593
24603881257.415604
24156248475.223606
23717715483.902462
23288093212.402443
22867196617.33388
22454844594.4512
22050859892.15871
21655069026.99004
21267302201.013813
20887393221.120007
20515179420.14188
20150501579.77011
19793203855.216385
19443133701.585064
19100141801.912357
18764081996.833694
18434811215.84058
18112189410.089695
17796079486.72735
17486347244.693893
17182861311.972996
16885493084.252026
16594116664.960405
16308608806.65347
16028848853.710594
15754718686.316616
15486102665.696718
15222887580.575449
14964962594.831402
14712219196.319695
14464551146.835112
14221854433.189379
13984027219.376747
13750969799.802755
13522584553.551311
13298775899.666433
13079450253.424904
12864515983.577183
12653883370.534124
12447464565.477882
12245173550.375597
12046926098.875242
11852639738.063328
11662233711.064756
11475628940.465513
11292747992.539381
11113515042.260374
10937855839.082855
10765697673.47193
10596969344.166983
10431601126.161716
10269524739.384388
10110673318.062336
9954981380.75535
9802384801.04264
9652820778.848698
9506227812.393553
9362545670.75337
9221715367.017591
9083679132.02917
8948380388.694815
8815763726.852383
8685774878.682936
8558360694.655188
8433469119.990486
8311049171.636583
8191050915.738868
8073425445.597918
7958124860.102508
7845102242.627467
7734311640.386057
7625708044.226663
7519247368.864025
7414886433.535263
7312582943.071296
7212295469.37446
7113983433.293285
7017607086.885628
6923127496.061648
6830506523.598125
6739706812.515992
6650691769.813038
6563425550.543964
6477873042.240136
6393999849.661561
6311772279.873769
6231157327.642514
6152122661.139253
6074636607.950605
5998668141.385206
5924186867.071307
5851163009.838901
5779567400.880069
5709371465.181493
5640547209.223233
5573067208.9379015
5506904597.924616
5442033055.912168
5378426797.46598
5316060560.933568
5254909597.623319
5194949661.2115345
5136156997.3727865
5078508333.628749
5021980869.410768
4966552266.331585
4912200638.661632
4858904544.005542
4806642974.174524
4755395346.250381
4705141493.837055
4655861658.495639
4607536481.3589525
4560146994.921772
4513674615.002972
4468101132.875878
4423408707.563237
4379579858.293242
4336597457.113228
4294444721.657557
4253105208.066543
4212562804.053023
4172801722.1135597
4133806492.8811145
4095561958.6161766
4058053266.8334436
4021265864.061104
3985185489.7299514
3949798170.1895227
3915090212.8486094
3881048200.437472
3847658985.3891406
3814909684.337368
3782787672.7286496
3751280579.545978
3720376282.141932
3690062901.1787825
3660328795.6733546
3631162558.1444583
3602553009.8606644
3574489196.1863756
3546960382.0240526
3519956047.350615
3493465882.846027
3467479785.6120825
3441987854.979558
3416980388.401809
3392447877.4330564
3368381003.789511
3344770635.491656
3321607823.085977
3298883795.944417
3276589958.64
3254717887.3969865
3233259326.613998
3212206185.458606
3191550534.531864
3171284602.6013308
3151400773.401179
3131891582.497912
3112749714.2204375
3093967998.653024
3075539408.689931
3057457057.1503615
3039714193.9525356
3022304203.3455877
3005220601.1981487
2988457032.342386
2972007267.972377
2955865203.095665
2940024854.0369077
2924480355.9925337
2909225960.6353436
2894256033.7680163
2879565053.0245204
2865147605.618414
2850998386.1370893
2837112194.380988
2823483933.2468657
2810108606.654196
2796981317.513811
2784097265.7379103
2771451746.29061
2759040147.2781234
2746857948.0778456
2734900717.505465
2723164112.0193577
2711643873.9614825
2700335829.8340216
2689235888.6110344
2678340040.0844083
2667644353.24338
2657144974.6869745
2646838127.0686274
2636720107.572393
2626787286.4200387
2617036105.408408
2607463076.476443
2598064780.3012333
2588837864.9225197
2579779044.395034
2570885097.4681597
2562152866.292284
2553579255.1513553
2545161229.221069
2536895813.352176
2528780090.878393
2520811202.4484067
2512986344.881492
2505302770.046249
2497757783.761987
2490348744.7222986
2483073063.440365
2475928201.215545
2468911669.120834
2462021027.0107284
2455253882.5491185
2448607890.2567806
2442080750.5780616
2435670208.9663877
2429374054.9881926
2423190121.444897
2417116283.5125785
2411150457.8989606
2405290602.017369
2399534713.177319
2393880827.7913885
2388327020.598047
2382871403.900107
2377512126.818505
2372247374.561065
2367075367.7059803
2361994361.4996643
2357002645.168741
2352098541.2458296
2347280404.908872
2342546623.333738
2337895615.0598025
2333325829.3682785
2328835745.673001
2324423872.9234447
2320088749.0197077
2315828940.2392287
2311643040.674992
2307529671.684991
2303487481.3527303
2299515143.958527
2295611359.4614096
2291774852.991374
2288004374.351835
2284298697.5320034
2280656620.2290435
2277076963.379789
2273558570.701808
2270100308.243673
2266701063.944202
2263359747.200523
2260075288.4447656
2256846638.7292147
2253672769.319745
2250552671.297394
2247485355.1678667
2244469850.4788623
2241505205.445017
2238590486.5803514
2235724778.33804
2232907182.7573757
2230136819.1177626
2227412823.5996304
2224734348.952087
2222100564.1672115
2219510654.1608334
2216963819.4596634
2214459275.894684
2211996254.3006086
2209574000.221364
2207191773.6214075
2204848848.6028104
2202544513.1279545
2200278068.747763
2198048830.3353295
2195856125.8248405
2193699295.9557047
2191577694.021751
2189490685.625428
2187437648.4368777
2185417971.9578023
2183431057.290015
2181476316.9086003
2179553174.4395676
2177661064.4419203
2175799432.194059
2173967733.484414
2172165434.4062343
2170392011.156456
2168646949.838549
2166929746.269276
2165239905.7892857
2163576943.0774593
2161940381.9689326
2160329755.27673
2158744604.616923
2157184480.237262
2155648940.849195
2154137553.4632096
2152649893.227437
2151185543.269453
2149744094.541205
2148325145.667005
2146928302.7945316
2145553179.4487762
2144199396.388881
2142866581.4677956
2141554369.494718
2140262402.10025
2138990327.6042066
2137737800.8860512
2136504483.257877
2135290042.3399014
2134094151.9384081
2132916491.9261124
2131756748.124874
2130614612.1907258
2129489781.5011702
2128381959.0446966
2127290853.3124804
2126216178.1922035
2125157652.8639822
2124115001.6983278
2123087954.1561348
2122076244.6906202
2121079612.6512039
2120097802.1892736
2119130562.1658094
2118177646.0608199
2117238811.884559
2116313822.09049
2115402443.4899635
2114504447.1685586
2113619608.404084
2112747706.5861764
2111888525.137485
2111041851.4363952
2110207476.7412794
2109385196.1162214
2108574808.3582084
2107776115.925742
2106988924.8688555
2106213044.760492
2105448288.6292474
Let’s plot the convergence graph
In [29]:
plt.plot(list(range(0, len(J_history))), J_history, '-b') # Plot the data
plt.xlabel('Number of iterations')
plt.ylabel('Cost J')
plt.show()
Section 3. Logistic Regression [25 pts]¶
The following logistic regression assignment is modified from Stanford CS229. Please complete and hand in this completed worksheet.
Logistic Regression¶
In this section, you need to implement logsitic regression to solve a binary classification problem. Let’s first get our data ready:
In [30]:
# Only use the first 70 samples for training (and validation),
# and treat the rest of them as hold-out testing set.
X = np.loadtxt('data/logistic_x_.txt')
y = np.loadtxt('data/logistic_y_.txt').reshape(-1, 1)
X, mu, std = feature_normalize(X)
# Add a column of ones to X for the bias weight.
m = len(X)
X = np.concatenate((np.ones((m, 1)), X), axis=1)
Here, the input $x^{(i)}\in\mathbb{R^2}$ and $y^{(i)}\in\{-1, 1\}$. Like we have mentioned, it is better to visualize the data first before you start working on it.
In [31]:
# Plot the feature according to their class label.
# Note that we exclude column 0, which is the colunm we padded with one in the previous block.
plt.plot(X[np.where(y==1), 1], X[np.where(y==1), 2], 'rx')
plt.plot(X[np.where(y==-1), 1], X[np.where(y==-1), 2], 'bo')
plt.xlabel('x2')
plt.ylabel('x1')
plt.show()
In the following, you need to implement logistic regression. Recall that when $y^{(i)}\in{-1,1}$, the objective function for binary logistic regression can be expressed as: \begin{equation*} J(\theta) = \frac{1}{m}\sum_{i=1}^{m}\log{\left(1+e^{-y^{(i)\theta^Tx^{(i)}}}\right)}=-\frac{1}{m}\sum_{i=1}^m\log{\left(h_{\theta}(y^{(i)}x^{(i)})\right)} \end{equation*} where the hypothesis is the sigmoid function: \begin{equation*} h_\theta(y^{(i)}x^{(i)})=\frac{1}{1+e^{-y^{(i)}\theta^{T}x^{(i)}}} \end{equation*} which we have seen in class (and assignment 0). Similar to the previous section, we can minimize the objective function $J(\theta)$ using batch gradient descent: \begin{equation*} \theta_j := \theta_j – \alpha \frac{1}{m}\sum_{i=1}^{m}h_\theta(-y^{(i)}x_j^{(i)})(-y^{(i)}x_j^{(i)}) \end{equation*}
Now, your task is to complete the function sigmoid, compute_cost, gradient_descent for logistic regression.
In [32]:
def sigmoid(z):
#####################################################################
# Instructions: Implement sigmoid function g #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return g
def compute_cost(X, y, theta):
# You need to return the following variables correctly
J = 0;
#####################################################################
# Instructions: Implement the objective function J(theta) #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return J
def compute_gradient(X, y, theta):
#####################################################################
# Instructions: Implement gradient function gradient_ #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return gradient_
def gradient_descent_logistic(X, y, theta, alpha, num_iters):
m = len(y)
J_history = []
for iter in range(num_iters):
theta = 0
#####################################################################
# Instructions: Perform a single gradient step on the parameter #
# vector theta using the implemented compute_gradient #
# #
# Hint: While debugging, it can be useful to print out the values #
# of the cost function (compute_cost) and gradient here. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
# Save the cost J in every iteration
J = compute_cost(X, y, theta)
print(J)
J_history.append(J)
return theta, J_history
Now, fit your model, and see if it is learning.
In [33]:
# Train your model.
theta = np.zeros((X.shape[1], 1))
alpha = 0.1;
num_iters = 400;
theta, J_history = gradient_descent_logistic(X, y, theta, alpha, num_iters)
0.6767318709516239
0.6612759836177738
0.6467213868522412
0.6330122173560181
0.6200951054326651
0.6079193220430319
0.5964368590615188
0.5856024539525343
0.5753735694532695
0.565710337891584
0.5565754786361017
0.547934195984304
0.5397540636253004
0.5320049007207323
0.5246586436612436
0.5176892166916711
0.5110724038582442
0.5047857241105058
0.4988083108795615
0.49312079704042866
0.48770520583666754
0.4825448480874091
0.47762422579843516
0.4729289421494971
0.4684456177202448
0.46416181273904
0.4600659550858426
0.4561472737467817
0.4523957373993958
0.448801997800213
0.44535733764742047
0.44205362259850367
0.4388832571341371
0.43583914397384127
0.4329146467649332
0.43010355578324566
0.4274000564013932
0.4247987000975499
0.42229437779447193
0.4198822953346259
0.417557950912616
0.4153171143005741
0.4131558077157223
0.41107028819193164
0.4090570313288082
0.40711271630263857
0.40523421203348253
0.4034185644118437
0.401662984496735
0.39996483760461843
0.3983216332157216
0.3967310156306258
0.3951907553158623
0.3936987408825787
0.3922529716471801
0.39085155072727046
0.389492678630239
0.3881746472954943
0.386895834554683
0.38565469897726545
0.3844497750715729
0.38327966881399933
0.3821430534812575
0.38103866576272566
0.3799653021318076
0.378921815456965
0.37790711183465914
0.3769201476278858
0.3759599266953023
0.3750254977971449
0.37411595216524024
0.37323042122541344
0.3723680744615167
0.3715281174111411
0.3707097897838466
0.3699123636934485
0.3691351419965436
0.36837745673005506
0.3676386676411173
0.36691816080311873
0.36621534731218414
0.3655296620587962
0.3648605625696431
0.3642075279151391
0.36357005767838924
0.36294767098167463
0.3623399055668146
0.36174631692601383
0.36116647748004677
0.3605999758008458
0.36004641587576497
0.35950541641097494
0.35897661017162175
0.35845964335653885
0.3579541750054525
0.3574598764367555
0.3569764307140532
0.3565035321398041
0.35604088577448395
0.3555882069798088
0.35514522098464013
0.35471166247229
0.3542872751880217
0.3538718115656143
0.3534650323719397
0.3530667063685542
0.3526766099893788
0.3522945270335911
0.3519202483729106
0.35155357167250495
0.3511943011247944
0.3508422471954714
0.3504972263810946
0.3501590609776562
0.34982757885955074
0.3495026132684144
0.34918400261132765
0.348871590267909
0.348565224405848
0.34826475780445815
0.3479700476858499
0.34768095555334516
0.3473973470367804
0.34711909174436023
0.3468460631207451
0.3465781383110707
0.34631519803061844
0.34605712643986486
0.34580381102465674
0.34555514248127206
0.345311014606137
0.3450713241899841
0.34483597091624574
0.3446048572634892
0.3443778884117099
0.34415497215230506
0.3439360188015665
0.34372094111753104
0.34350965422004076
0.3433020755138719
0.34309812461479494
0.3428977232784407
0.3427007953318474
0.34250726660757574
0.34231706488027847
0.3421301198056233
0.3419463628614649
0.3417657272911741
0.341588148049033
0.3414135617476077
0.34124190660702
0.34107312240603516
0.3409071504348955
0.3407439334498255
0.34058341562914135
0.34042554253089924
0.34027026105202324
0.34011751938884927
0.33996726699903296
0.33981945456476426
0.3396740339572391
0.3395309582023386
0.33939018144746774
0.3392516589295106
0.33911534694385576
0.33898120281445493
0.33884918486487225
0.33871925239028705
0.33859136563041586
0.33846548574331686
0.3383415747800449
0.3382195956601256
0.3380995121478175
0.33798128882913353
0.3378648910895934
0.33775028509268207
0.3376374377589863
0.337526316745985
0.3374168904284732
0.3373091278795917
0.33720299885244454
0.33709847376228236
0.33699552366923224
0.3368941202615529
0.3367942358393999
0.336695843299079
0.33659891611777665
0.33650342833874436
0.33640935455692644
0.3363166699050138
0.3362253500399096
0.33613537112959346
0.3360467098403689
0.33595934332448446
0.335873249208112
0.33578840557967354
0.3357047909785037
0.3356223843838368
0.3355411652041074
0.33546111326655587
0.33538220880712655
0.3353044324606509
0.33522776525130665
0.33515218858334067
0.3350776842320533
0.3350042343350292
0.33493182138361116
0.33486042821460876
0.33479003800223184
0.33472063425024473
0.3346522007843335
0.3345847217446783
0.3345181815787264
0.33445256503415804
0.3343878571520404
0.3343240432601633
0.33426110896655176
0.3341990401531485
0.33413782296966316
0.33407744382758214
0.33401788939433474
0.3339591465876101
0.3339012025698215
0.33384404474271323
0.3337876607421052
0.3337320384327727
0.3336771659034561
0.3336230314619962
0.33356962363059456
0.33351693114119074
0.3334649429309573
0.33341364813790614
0.3333630360966051
0.3333130963339994
0.3332638185653382
0.3332151926901997
0.33316720878861517
0.33311985711728553
0.3330731281058928
0.3330270123534994
0.33298150062503384
0.332936583847863
0.3328922531084448
0.33284849964906227
0.33280531486463416
0.33276269029960326
0.3327206176448954
0.332679088734953
0.33263809554483686
0.33259763018739597
0.33255768491050325
0.3325182520943564
0.33247932424884014
0.3324408940109503
0.33240295414227644
0.33236549752654304
0.3323285171672061
0.33229200618510496
0.33225595781616735
0.33222036540916666
0.33218522242352894
0.3321505224271905
0.3321162590945021
0.3320824262041812
0.332049017637309
0.33201602737537256
0.3319834494983492
0.3319512781828345
0.33191950770021006
0.33188813241485315
0.33185714678238315
0.33182654534794825
0.33179632274454823
0.3317664736913936
0.3317369929923008
0.3317078755341208
0.33167911628520264
0.33165071029388815
0.3316226526870411
0.33159493866860495
0.3315675635181929
0.331540522589707
0.3315138113099868
0.33148742517748564
0.3314613597609752
0.33143561069827676
0.33141017369501896
0.3313850445234213
0.33136021902110263
0.33133569308991445
0.33131146269479794
0.33128752386266447
0.331263872681299
0.3312405052982859
0.33121741791995624
0.33119460681035684
0.3311720682902398
0.3311497987360723
0.3311277945790667
0.33110605230422985
0.3310845684494312
0.3310633396044891
0.33104236241027657
0.33102163355784303
0.33100114978755546
0.33098090788825313
0.33096090469642286
0.3309411370953882
0.330921602014514
0.33090229642842806
0.33088321735625664
0.3308643618608752
0.33084572704817367
0.3308273100663361
0.3308091081051331
0.330791118395229
0.3307733382075021
0.33075576485237684
0.3307383956791697
0.3307212280754471
0.3307042594663958
0.33068748731420383
0.3306709091174556
0.3306545224105354
0.3306383247630441
0.33062231377922613
0.33060648709740686
0.33059084238944064
0.3305753773601688
0.33056008974688883
0.33054497731883087
0.3305300378766469
0.3305152692519064
0.3305006693066033
0.33048623593267107
0.33047196705150633
0.33045786061350146
0.3304439145975868
0.3304301270107778
0.33041649588773486
0.33040301929032695
0.33038969530720547
0.33037652205338547
0.330363497669833
0.3303506203230612
0.33033788820473337
0.3303252995312721
0.3303128525434763
0.3303005455061445
0.3302883767077052
0.3302763444598531
0.33026444709719155
0.3302526829768828
0.3302410504783019
0.3302295480026991
0.3302181739728658
0.330206926832808
0.33019580504742474
0.3301848071021919
0.33017393150285146
0.33016317677510665
0.33015254146432144
0.3301420241352258
0.330131623371626
0.33012133777611874
0.3301111659698119
0.33010110659204867
0.3300911583001365
0.3300813197690814
0.33007158969132516
0.3300619667764893
0.33005244975112075
0.33004303735844304
0.33003372835811245
0.33002452152597644
0.3300154156538369
0.33000640954921795
0.32999750203513634
0.3299886919498766
0.32997997814676927
0.32997135949397327
0.3299628348742615
0.32995440318480934
0.32994606333698856
0.3299378142561623
0.3299296548814842
0.32992158416570166
0.32991360107496076
0.3299057045886162
0.32989789369904143
0.3298901674114456
0.3298825247436895
0.3298749647261081
0.3298674864013327
0.32986008882411827
0.32985277106117206
0.3298455321909866
0.3298383713036725
0.32983128750079727
0.3298242798952241
Again, plot and check to see if the model is converging.
In [34]:
plt.plot(list(range(0, len(J_history))), J_history, '-b')
plt.xlabel('Number of iterations')
plt.ylabel('Cost J')
plt.show()
print (theta)
[[-0.03801384]
[ 1.38454246]
[ 1.91783357]]
Decision Boundary¶
In addition to checking convergence graph and accuracy, we can also plot out the decision boundary to see what does the model actually learn.
In [35]:
# Plot the feature according to their class label.
# Note that we exclude column 0, which is the colunm we padded with one in the previous block.
plt.plot(X[np.where(y==1), 1], X[np.where(y==1), 2], 'rx')
plt.plot(X[np.where(y==-1), 1], X[np.where(y==-1), 2], 'bo')
#####################################################################
# Instructions: Plot out the decision boundary. #
# Hint: To plot the boundary, which is a straight line in our case, #
# you need to find the two ends of the line, and plot it with #
# plt.plot(). Note that the decision boundary is the line that#
# y = 0. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
plt.xlabel('x1')
plt.ylabel('x2')
plt.show()
Section 4. Regularization [30 pts]¶
In this section, you need to incorporate L2 regularization into your logistic regression.
L2 Regularization¶
Overfitting is a notorious problem in the world of machine learning. One simple way to counter this issue is to put constraints on your model weights $\theta$, as we have discussed in class. In this section, you need to modify the the objective function to impose L2 regularization on the logistic regression: \begin{equation*} J(\theta) = -\frac{1}{m}\sum_{i=1}^m\log{\left(h_{\theta}(y^{(i)}x^{(i)})\right)} + \lambda\vert\vert\theta\vert\vert_2^2 \end{equation*} Derive the gradient for this new objective to incorporate it into your logistic regression model.
To make things much structural, we now put everything together into a class. Please use the class template below to implement your logistic regression. Note that you can add your own class methods if needed.
In [36]:
class LogisticRegression(object):
def __init__(self, alpha=0.1, lamb=0.1, regularization=None):
# setting the class attribute.
self.alpha = alpha # Set up your learning rate alpha.
self.lamb = lamb # Strength of regularization.
self.regularization = regularization
assert regularization == 'l2' or regularization == None # we only consider these two cases
def _compute_cost(self, X, y):
#####################################################################
# Instructions: Compute the cost function here. #
# You need to handle both the cases with, and without #
# regularization here. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return J
def _compute_gradient(self, X, y):
#####################################################################
# Instructions: Compute the gradient here. #
# You need to handle both the cases with, and without #
# regularization here. #
#####################################################################
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return gradient
def fit(self, X, y, num_iter=5):
self.theta = np.zeros((X.shape[1], 1))
m = len(y)
J_history = []
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return J_history
def predict(self, X):
#####################################################################
# Instructions: Use your hypothese to make predictions. #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
return y_hat
Load the wine datasets, in which $x_j\in\mathbb{R}^{12}$ is different attribute for alcohol, and $y\in\{-1,1\}$ is that class label (red or white wine).
In [37]:
# Load dataset
X_train = np.loadtxt('data/wine_train_X.txt')
y_train = np.loadtxt('data/wine_train_y.txt').reshape(-1, 1)
X_test = np.loadtxt('data/wine_test_X.txt')
y_test = np.loadtxt('data/wine_test_y.txt').reshape(-1, 1)
X_train, mu, std = feature_normalize(X_train)
X_test, mu, std = feature_normalize(X_test)
X_train = np.concatenate((np.ones((X_train.shape[0], 1)), X_train), axis=1)
X_test = np.concatenate((np.ones((X_test.shape[0], 1)), X_test), axis=1)
Now, let’s train two different logistic regression models: one with, and one without regularization.
In [38]:
log_reg = LogisticRegression(alpha=0.1) # Without regularization
log_reg_l2 = LogisticRegression(alpha=0.1, lamb=1.0, regularization='l2') # Without regularization
J_history = log_reg.fit(X_train, y_train, num_iter=500)
J_history_l2 = log_reg_l2.fit(X_train, y_train, num_iter=500)
Next, we evaluate the accuracy for each method:
In [39]:
def evaluate_accuracy(X, y, model):
y_pred = model.predict(X)
y_pred[y_pred > 0.5] = 1
y_pred[y_pred <= 0.5] = -1
return np.mean(y_pred == y)
print("Accuracy on training set: ", evaluate_accuracy(X_train, y_train, log_reg))
print("Accuracy on testing set: ", evaluate_accuracy(X_test, y_test, log_reg))
print("Accuracy w/ L2 training set: ", evaluate_accuracy(X_train, y_train, log_reg_l2))
print("Accuracy w/ L2 testing set: ", evaluate_accuracy(X_test, y_test, log_reg_l2))
Accuracy on training set: 0.9925
Accuracy on testing set: 0.9925
Accuracy w/ L2 training set: 0.9925
Accuracy w/ L2 testing set: 0.9925
To see the effect of regularization on $\theta$, we can plot out each $\theta_j$ under different $\lambda$.
In [40]:
def plot_theta(theta, lamb):
"""
Helper function for plotting out the value of theta with respect to different lambda.
theta (list): list of theta under different lambda.
lambda (list): list of lambda values you tried.
"""
plt.hlines(y=0, xmin=0, xmax=np.max(lamb), color='red', linewidth = 2, linestyle = '--')
for i in range(theta.shape[1]):
plt.plot(lamb, theta[:,i])
plt.ylabel('theta')
plt.xlabel('lambda')
plt.xscale('log')
plt.show()
In [41]:
lamb = [0.1, 1, 10, 100, 1000]
theta = []
#####################################################################
# Instructions: For each value in lamb, try a model for it, and #
# append the trained weights into the theta #
#####################################################################
#####################################################################
# END OF YOUR CODE #
#####################################################################
plot_theta(np.array(theta), lamb)
In [ ]:
Do you need a similar assignment done for you from scratch? Order now!
Use Discount Code "Newclient" for a 15% Discount!
