W = VL_SVMPEGASOS(DATA, LAMBDA) learns a linear SVM W given training struct DATA, and the regularization parameter LAMBDA using the PEGASOS [1] solver. The algorithm finds a minimizer W of the objective function
LAMBDA/2 |W|^2 + 1/N SUM_i LOSS(W, X(:,i), Y(i))
where LOSS(W,X,Y) = MAX(0, 1 - Y W'X) is the hinge loss and N is the number of training vectors in X.
The training struct DATA is created using the function VL_MAKETRAININGSET.
[W B INFO] = VL_SVMPEGASOS(DATA, LAMBDA) learns a linear SVM W and a bias B given training struct DATA, and the regularization parameter LAMBDA using the PEGASOS [1] solver. INFO is a struct containing the input parameters plus diagnostic informations:
SVM energy value.
Number of iterations performed.
Elapsed time since the start of the SVM learning.
Value of the SVM regulizer term.
Value of loss function only for data points labeled positives.
Value of loss function onlt for data points labeled negatives.
Number of mislabeled positive points.
Number of mislabeled negative points.
ALGORITHM. PEGASOS is an implementation of stochastic subgradient descent. At each iteration a data point is selected at random, the subgradient of the cost function relative to that data point is computed, and a step is taken in that direction. The step size is inversely proportional to the iteration number. See [1] for details.
VL_SVMPEGASOS() accepts the following options:
Specify the SVM stopping criterion threshold. If not specified VL_SVMPEGASOS will finish when the maximum number of iterations is reached. The stopping criterion is tested after each ENERGYFREQ iteration.
Sets the maximum number of iterations.
Appends to the data X the specified scalar value B. This approximates the training of a linear SVM with bias.
Specify the initial value for the weight vector W.
Specify the iteration number to start from. The only effect is to change the step size, as this is inversely proportional to the iteration number.
Specify the inital bias value.
Specify the frequency of the bias learning. The default setting updates the bias at each iteration.
Specify a permutation PERM to be used to sample the data (this disables random sampling). Specifically, at the T-th iteration the algorithm takes a step w.r.t. the PERM[T']-th data point, where T' is T modulo the number of data samples (i.e. MOD(T'-1,NUMSAMPLES)+1). PERM needs not to be bijective. This allows specifying certain data points more or less frequently, implicitly increasing their relative weight in the error term. A common application is to balance an unbalanced dataset.
Specify a function handle to be called every ENERGYFREQ iterations. The function must be of the form:
function o = diagnostics(svm,x)
Where in the first iteration x is the variable passed as "DiagnosticCallRef", and the consecutive ones x is the variable o returned in the previous iteration.
Specify a paramater to be passed to the DIAGNOSTICFUNCTION handle.
Specify how often the SVM energy is computed.
Specify a validation dataset. The validation dataset must be created using VL_MAKETRAININGSET. If specified, the energy value and all the diagnostic informations are computed on the validation dataset, otherwise the training dataset is used.
Be verbose.
The options StartingModel and StartingIteration can be used to continue training. I.e., the command
vl_twister('state',0) ; dataset = vl_maketrainingset(x,y) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',1000) ;
produces the same result as the sequence
vl_twister('state',0) ; dataset = vl_maketrainingset(x,y) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',500) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',500, ... 'StartingIteration', 501, ... 'StartingModel', w) ;
[1] S. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter. Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. MBP, 2010.
See also: VL_MAKETRAININGSET(), VL_HOMKERMAP(), VL_HELP().