aib.h File Reference
Detailed Description
This provides an implementation of Agglomerative Information Bottleneck (AIB) as first described in:[Slonim] N. Slonim and N. Tishby. Agglomerative information bottleneck. In Proc. NIPS, 1999
AIB takes a discrete valued feature and a label
and gradually compresses
by iteratively merging values which minimize the loss in mutual information
.
While the algorithm is equivalent to the one described in [Slonim], it has some speedups that enable handling much larger datasets. Let N be the number of feature values and C the number of labels. The algorithm of [Slonim] is in space and
in time. This algorithm is
space and
time in common cases (
in the worst case).
Overview
Given a discrete feature


![$[x]_{ij}$](form_10.png)



AIB iterates this procedure until the desired level of compression is achieved.
Algorithm details
Computing

Thus in a basic implementation of AIB, finding the optimal pair of feature values requires
operations in total. In order to join all the
values, we repeat this procedure
times, yielding
time and
space complexity (this does not account for the space need to store the input).
The complexity can be improved by reusing computations. For instance, we can store the matrix (which requires
space). Then, after joining
, all of the matrix D except the rows and columns (the matrix is symmetric) of indexes i and j is unchanged. These two rows and columns are deleted and a new row and column, whose computation requires
operations, are added for the merged value
. Finding the minimal element of the matrix still requires
operations, so the complexity of this algorithm is
time and
space.
We can obtain a much better expected complexity as follows. First, instead of storing the whole matrix D, we store the smallest element (index and value) of each row as (notice that this is also the best element of each column since D is symmetric). This requires
space and finding the minimal element of the matrix requires
operations. After joining
, we have to efficiently update this representation. This is done as follows:
- The entries
and
are deleted.
- A new entry
for the joint value
is added. This requires
operations.
- We test which other entries
need to be updated. Recall that
means that, before the merge, the value closest to
was
at a distance
. Then
- If
,
and
, then
is still the closest element and we do not do anything.
- If
,
and
, then the closest element is
and we update the entry in constant time.
- If
or
, then we need to re-compute the closest element in
operations.
- If
This algorithm requires only space and
time, where
is the expected number of times we fall in the last case. In common cases one has
, so the time saving is significant.
Definition in file aib.h.
#include "generic.h"
#include "mathop.h"
Go to the source code of this file.
Data Structures | |
struct | _VlAIB |
AIB algorithm data. More... | |
Functions | |
Create and destroy | |
VL_EXPORT VlAIB * | vl_aib_new (double *Pcx, vl_uint nvalues, vl_uint nlabels) |
Allocates and initializes the internal data structure. | |
VL_EXPORT void | vl_aib_delete (VlAIB *aib) |
Deletes AIB data structure. | |
Process data | |
VL_EXPORT void | vl_aib_process (VlAIB *aib) |
Runs AIB on Pcx. | |
Retrieve results | |
VL_INLINE vl_uint * | vl_aib_get_parents (VlAIB const *aib) |
Get resulting list of parents. | |
VL_INLINE double * | vl_aib_get_costs (VlAIB const *aib) |
Get a list of merge costs. |
Function Documentation
VL_EXPORT void vl_aib_delete | ( | VlAIB * | aib | ) |
VL_INLINE double * vl_aib_get_costs | ( | VlAIB const * | aib | ) |
- Parameters:
-
aib AIB filter.
- Returns:
- An array of costs
Definition at line 109 of file aib.h.
References _VlAIB::costs.
- Parameters:
-
aib AIB filter.
- Returns:
- An array of parents
Definition at line 97 of file aib.h.
References _VlAIB::parents.
- Parameters:
-
Pcx A pointer to a 2D array of probabilities nvalues The number of rows in the array nlabels The number of columns in the array
Allocates memory for the following:
- Px (nvalues*sizeof(double))
- Pc (nlabels*sizeof(double))
- nodelist (nvalues*sizeof(vl_uint))
- which (nvalues*sizeof(vl_uint))
- beta (nvalues*sizeof(double))
- bidx (nvalues*sizeof(vl_uint))
- parents ((2*nvalues-1)*sizeof(vl_uint))
- costs (nvalues*sizeof(double))
Since it simply copies to pointer to Pcx, the total additional memory requirement is:
(3*nvalues+nlabels)*sizeof(double) + 4*nvalues*sizeof(vl_uint)
- Returns:
- An allocated and initialized VlAIB pointer
Definition at line 484 of file aib.c.
References _VlAIB::beta, _VlAIB::bidx, _VlAIB::costs, _VlAIB::nentries, _VlAIB::nlabels, _VlAIB::nodes, _VlAIB::nvalues, _VlAIB::nwhich, _VlAIB::parents, _VlAIB::Pc, _VlAIB::Pcx, _VlAIB::Px, vl_aib_new_nodelist(), vl_aib_new_Pc(), vl_aib_new_Px(), vl_aib_normalize_P(), vl_malloc(), and _VlAIB::which.
VL_EXPORT void vl_aib_process | ( | VlAIB * | aib | ) |
- Parameters:
-
aib AIB object to process
x
that causes the smallest decrease in mutual information between the random variables x
and c
.
Merge operations are arranged in a binary tree. The nodes of the tree correspond to the original feature values and any other value obtained as a result of a merge operation. The nodes are indexed in breadth-first order, starting from the leaves. The first index is zero. In this way, the leaves correspond directly to the original feature values. In total there are 2*nvalues-1
nodes.
The results may be accessed through vl_aib_get_parents which returns an array with one element per tree node. Each element is the index the parent node. The root parent is equal to zero. The array has 2*nvalues-1
elements.
Feature values with null probability are ignored by the algorithm and their nodes have parents indexing a non-existent tree node (a value bigger than 2*nvalues-1
).
Then the function will also compute the information level after each merge. vl_get_costs will return a vector with the information level after each merge. cost has nvalues
entries: The first is the value of the cost functional before any merge, and the others are the cost after the nvalues-1
merges.
Definition at line 582 of file aib.c.
References _VlAIB::costs, _VlAIB::nodes, _VlAIB::nvalues, _VlAIB::nwhich, _VlAIB::parents, vl_aib_calculate_information(), vl_aib_merge_nodes(), vl_aib_min_beta(), vl_aib_update_beta(), VL_NAN_D, and VL_PRINTF.