Logistic Regression classifier

Logistic Regression

ML implements logistic regression, which is a probabilistic classification technique. Logistic Regression is a binary classification algorithm which is closely related to Support Vector Machines (SVM). Like SVM, Logistic Regression can be extended to work on multi-class classification problems like digit recognition (i.e. recognizing digitis like 0,1 2, 3,... from the given images). This version of Logistic Regression supports both binary and multi-class classifications (for multi-class it creates a multiple 2-class classifiers). In order to train the logistic regression classifier, Batch Gradient Descent and Mini-Batch Gradient Descent algorithms are used (see [BatchDesWiki]). Logistic Regression is a discriminative classifier (see [LogRegTomMitch] for more details). Logistic Regression is implemented as a C++ class in cv.LogisticRegression.

In Logistic Regression, we try to optimize the training paramater theta such that the hypothesis 0 <= h_theta(x) <= 1 is acheived. We have h_theta(x) = g(h_theta(x)) and g(z)=1/(1+e^(-z)) as the logistic or sigmoid function. The term "Logistic" in Logistic Regression refers to this function. For given data of a binary classification problem of classes 0 and 1, one can determine that the given data instance belongs to class 1 if h_theta(x) >= 0.5 or class 0 if h_theta(x) < 0.5.

In Logistic Regression, choosing the right parameters is of utmost importance for reducing the training error and ensuring high training accuracy:

• The learning rate can be set with LearningRate property. It determines how fast we approach the solution. It is a positive real number.
• Optimization algorithms like Batch Gradient Descent and Mini-Batch Gradient Descent are supported in cv.LogisticRegression. It is important that we mention the number of iterations these optimization algorithms have to run. The number of iterations can be set with Iterations property. This parameter can be thought as number of steps taken and learning rate specifies if it is a long step or a short step. This and previous parameter define how fast we arrive at a possible solution.
• In order to compensate for overfitting regularization is performed, which can be enabled with Regularization property. One can specify what kind of regularization has to be performed by passing one of regularization kinds to this property.
• Logistic regression implementation provides a choice of 2 training methods with Batch Gradient Descent or the MiniBatch Gradient Descent. To specify this, set TrainMethod property as either Batch or MiniBatch. If training method is set to MiniBatch, the size of the mini batch has to be to a postive integer set with MiniBatchSize property.

Example

A sample set of training parameters for the Logistic Regression classifier can be initialized as follows:

lr = cv.LogisticRegression();
lr.LearningRate = 0.05;
lr.Iterations = 1000;
lr.Regularization = 'L2';
lr.TrainMethod = 'MiniBatch';
lr.MiniBatchSize = 10;

References

[LogRegWiki]:

http://en.wikipedia.org/wiki/Logistic_regression

[BatchDesWiki]:

http://en.wikipedia.org/wiki/Gradient_descent_optimization

[LogRegTomMitch]:

"Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression" in Machine Learning, Tom Mitchell. http://www.cs.cmu.edu/~tom/NewChapters.html

[RenMalik2003]:

"Learning a Classification Model for Segmentation". Proc. CVPR, Nice, France (2003).