Implement Naive Bayes Classifier with Matlab

Estimating joint density directly from the joint density table requires intensive computation, especially with large dataset. Naive Bayes classifier was developed to simplify the problem by assuming the conditional independence of of the features given the class. To implement a Naive Bayes classifier in Matlab, two separate functions could be developed: nb_train and nb_test. The nb_train() function takes in training dataset x and y, with each row of x represents the feature vector of one training instance and the corresponding row in y contains the class label for that instance. Inside nb_train() function, data points for each class are grouped. For each class, the conditional probability of each feature is calculated. With Matlab, computation time could be greatly reduced if using matrix arithmetics rather than looping through every element. Another thing worth attention is that, smoothing is a widely employed technique in Naive Bayes algorithm implementation when calculating conditional probability. One common approach is additive smoothing, as illustrated below. This example calculates conditional probabilities of all n features for a binary class problem. catXOnes and catXZeros are the concatenated matrices for the two classes individually. The computed conditional probabilities matrices as well as class priors should be put together inside a model struct as the return value. Inside nb_test function, class label prediction is performed according to the following formula: Information retrieved from the model struct is applied on the testing data points to calculate the probability of belonging to each class. The class label is determined as the one with the highest probability.

Implement Logistic Regression Classifier with Matlab

Logistic Regression classifier is a linear classifier, roughly of the form P(y|x,w) = sigmoid(x.w) (as compared to sign(x.w) in linear classifier). The cost function is defined as follows: where S() - the sigmoid function,theta - current logistic regression parameter,lambda - the regularization term. The code implementation for this representation is straightforward.  One common approach to minimize the cost function is gradient descent.  Based on these helper functions implemented, the training phase could be simplified as: 1. randomly initialize theta 2. call minimize() on the cost function with the initial theta. 3. update theta the finally returned value from minimize() The prediction function could hence be written as:

Implement Neural Network Classifier with Matlab

Neural Network classifier is a multilayer network of logistic units, with each unit takes some inputs and produces one output using a logistic classifier and output of one unit can be the input of another. The following schematic represents a neural network with one hidden layer.  Similar to Linear Regression classifier, squared error loss will be used. Below is the forward propagation algorithm to compute the activation “act", where W1/W2 represent weights of each edge, b1/b2 represent biases.  According to the back propagation algorithm, the cost function and gradients could be calculated as follows.

Implement K-Means Clustering with Matlab

K-Means is a popular clustering algorithm with fast running speed and high scalability. Moreover, this algorithm could be easily implemented using the Matlab built in function “pdist2" ( for details please refer to http://www.mathworks.com/help/stats/pdist2.html) [clusterCenters, clusterBelonging] = k_means(data, k, startingPoints) data: points to be clustered K: # of clusters startPoints:the starting centroids of the k clusters. If not given explicitly, one common approach is to randomly select k points from input dataset clusterCenter: the centers of the clusters after running K-Means clusterBelonging: the cluster each data point belongs to [clusterCenters, clusterBelonging] = k_means(data, k, startingPoints) data: points to be clustered K: # of clusters  startPoints:the starting centroids of the k clusters. If not given explicitly, one common approach is to randomly select k points from input dataset clusterCenter: the centers of the clusters after running K-Means clusterBelonging: the cluster each data point belongs to