[Work Log] TULIPS - Debugging training
Issue: after training, still getting a very large perturb_smoothing_variance: 1e-4 instead of more reasonable 1e-6.
Need to update tracks to reflect new noise parameter.
Cleanup: building data pipeline
Most of the processing pipeline is currently implemented in wacv_2014/run_wacv_4.m. Moving the track-processing part into process_tracks.m. Using the '7-stage' pipeline [as documented here].
Retraining
updated tracks, retrained. New trained parameters look reasonable, but the reconstruction is terrible:
Maybe index optimization is failing, because the model #5 gradient has a bug? Or hitting the iteration limit? Running reconstruction with model #3 (whose gradient is proven).
Nope, it seems to be the new parameters:
It's weird, because the noise variance dropped signficantly, but we're seeing greater drift away from the data.
Lets try relaxing the iteration limit anyway...
More iterations makes it more crazy.
Sanity check time. Roll back to old parameters and reconstruct.
Check. Old parameters give a sensible reconstruction.
Differences: (1) much lower noise variance, and (2) much lower perturb smoothing variance.
The noise variance seems to be the issue here -- raising it to pre-training levels returns us to sensible reconstructions.
Running out of steam -- time to take a break. Next steps -
- manual test noise variance; hypothesis: large variance gives higher likelihood.
- ensure we aren't suffering from near-singular matrix during reconstruction. (piece-wise reconstruction?)