W = VL_SVMPEGASOS(DATA, LAMBDA) learns a linear SVM W given training struct DATA, and the regularization parameter LAMBDA using the PEGASOS [1] solver. The algorithm finds a minimizer W of the objective function
LAMBDA/2 |W|^2 + 1/N SUM_i LOSS(W, X(:,i), Y(i))
where LOSS(W,X,Y) = MAX(0, 1 - Y W'X) is the hinge loss and N is the number of training vectors in X.
The training struct DATA is created using the function VL_MAKETRAININGSET.
[W B INFO] = VL_SVMPEGASOS(DATA, LAMBDA) learns a linear SVM W and a bias B given training struct DATA, and the regularization parameter LAMBDA using the PEGASOS [1] solver. INFO is a struct containing the input parameters plus diagnostic informations:
- energy
SVM energy value.
- iterations
Number of iterations performed.
- elapseTime
Elapsed time since the start of the SVM learning.
- regulizerTerm
Value of the SVM regulizer term.
- lossPos
Value of loss function only for data points labeled positives.
- lossNeg
Value of loss function onlt for data points labeled negatives.
- hardLossPos
Number of mislabeled positive points.
- hardLossNeg
Number of mislabeled negative points.
ALGORITHM. PEGASOS is an implementation of stochastic subgradient descent. At each iteration a data point is selected at random, the subgradient of the cost function relative to that data point is computed, and a step is taken in that direction. The step size is inversely proportional to the iteration number. See [1] for details.
VL_SVMPEGASOS() accepts the following options:
- Epsilon [empty]
Specify the SVM stopping criterion threshold. If not specified VL_SVMPEGASOS will finish when the maximum number of iterations is reached. The stopping criterion is tested after each ENERGYFREQ iteration.
- MaxIterations [10 / LAMBDA]
Sets the maximum number of iterations.
- BiasMultiplier [0]
Appends to the data X the specified scalar value B. This approximates the training of a linear SVM with bias.
- StartingModel [null vector]
Specify the initial value for the weight vector W.
- StartingIteration [1]
Specify the iteration number to start from. The only effect is to change the step size, as this is inversely proportional to the iteration number.
- StartingBias [0]
Specify the inital bias value.
- BiasLearningRate [1]
Specify the frequency of the bias learning. The default setting updates the bias at each iteration.
- Permutation [empty]
Specify a permutation PERM to be used to sample the data (this disables random sampling). Specifically, at the T-th iteration the algorithm takes a step w.r.t. the PERM[T']-th data point, where T' is T modulo the number of data samples (i.e. MOD(T'-1,NUMSAMPLES)+1). PERM needs not to be bijective. This allows specifying certain data points more or less frequently, implicitly increasing their relative weight in the error term. A common application is to balance an unbalanced dataset.
- DiagnosticFunction [empty]
Specify a function handle to be called every ENERGYFREQ iterations. The function must be of the form:
function o = diagnostics(svm,x)
Where in the first iteration x is the variable passed as "DiagnosticCallRef", and the consecutive ones x is the variable o returned in the previous iteration.
- DiagnosticCallRef [empty]
Specify a paramater to be passed to the DIAGNOSTICFUNCTION handle.
- EnergyFreq [100]
Specify how often the SVM energy is computed.
- ValidationData
Specify a validation dataset. The validation dataset must be created using VL_MAKETRAININGSET. If specified, the energy value and all the diagnostic informations are computed on the validation dataset, otherwise the training dataset is used.
- Verbose
Be verbose.
- Example
The options StartingModel and StartingIteration can be used to continue training. I.e., the command
vl_twister('state',0) ; dataset = vl_maketrainingset(x,y) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',1000) ;
produces the same result as the sequence
vl_twister('state',0) ; dataset = vl_maketrainingset(x,y) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',500) ; w = vl_svmpegasos(dataset,lambda,'NumIterations',500, ... 'StartingIteration', 501, ... 'StartingModel', w) ;
- REFERENCES
[1] S. Shalev-Shwartz, Y. Singer, N. Srebro, and A. Cotter. Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. MBP, 2010.
See also: VL_MAKETRAININGSET(), VL_HOMKERMAP(), VL_HELP().