[Work Log] FIRE - Refactoring, k-folds cross validation, baseline evaluation
Refactored code base to better support model evaluation and k-folds cross validation.
Run: Cross validation (baseline)
Description: Run 10-folds cross validation on the non-sampling (analytical) model.
Results:
Error values are mean per-dimension negative log-likelihood.
3 kmeans iterations:
------------
training error: 3.94522
test error: 4.9166
5 kmeans iterations:
---------------
training error: 3.97775
test error: 4.9501
10 kmeans iterations:
------------------
training error: 4.01519
test error: 5.0074
20 kmeans iterations:
-----------
training error: 4.01538
test error: 5.0068
Discussion:
Increasing iterations also increases training error, which seems counterintuitive, especially because the algorithm debugging messages show the total log likelihood increasing.
But the log likelihood uses known membership, so values will naturally be higher. Model evaluation marginalizes over clusters, and since we have fixed the cluster weights to be equal, the dominating cluster is under-promoted. I would guess setting cluster weights from training membership proportions will reverse this trend, so more iterations will improve training error.
Run: Re-run with uneven cluster wieghts
Description: Set cluster weights proportional to the training membership values.
Results:
5 itns
--------
training error: 3.62931
test error: 4.61624
10 itns
---------
training error: 3.61348
test error: 4.60147
15 itns
-------
training error: 3.60094
test error: 4.58815
20 itns
--------
training error: 3.60094
test error: 4.58815
Discussion: Success! Error dropped as kmeans converged further
Posted by Kyle Simek