[Work Log] Work Log
Finished cleanup and committing of code for GMM EM with missing data.
Reviewing where we left off after WACV, hopefully looking at this indexing issue with fresh eyes should help.
As it stands, we never put a nail in the coffin of the drifting index. The marginal likelihood should discourage dramatic stretching of endpoint indices like we see in the image below.
Recalling the problem history. Was getting weird loops in reconstruction; bad indices were causing the curve to bind and twist to fit the spacing. Point-wise correspondence matching (the DTW part of the pipeline) assumes (naively) no motion in the scene. In some geometric configurations, this causes drifting offsets in the matching as described here. The hacks to improve indices post-hoc (described here) also assumes no motion, so doesn't help. Needed an optimization scheme that accounts for motion, and the natural choice is to maximize the marginal likelihood.
After adding likelihood maximization, started getting a new problem: the final point's likelihood is extremely far from the previous point's, resulting in results described on this page, example below.
Tried several heuristic energy functions to the optimization here (and several entried following it), including (a) keeping the index spacing equal to the distance between points, (b) keeping the index spacing between points smooth. The latter seemed to help, but reflecting on the code, it seems there may be a bug in the gradient computation, and I question the validity of the results. Needs more investigation.
Also issue of mean curve drifing far from maximum likelihood curve:
According to this entry, increasing the temporal correlation between views improved it. I also recall decreasing the perturb_position_variance helps too. It's disappointing that the trained values for these parameters are causing these problems. It's possible our model is them problem: camera perturbations are independent, while scene perturbations are correlated, but we're lumping both sources of noise into the scene perturbation model.
Separate perturbation process into two processes: scene perturbation (with perturb_smoothing_variance) and camera noise (with perturb_position_variance and perturb_rate_variance). This should allow the scene perturbation to have longer temporal correlations, because the i.i.d. camera noise is separated out from it.
It is interesting that the perturb_position_variance is so low (0.64) but we get such variation. Is this because the likelihood is so much stronger? Or a side-effect of the marginal likelihood preferring data points to bunch together, so pulling toward the camera is common.
Training didn't use good indices. need to optimize indices jointly with parameters
Re-running training
Files in the train/ subdirectory is really old and use out-of-date data structures. Working on getting it running again.
Done. Added new model "#5" that separates camera perturbation into a separate process.
Retrained on new model. Changes:
- perturbation scale doubled
- perturb smoothing variance increased by two orders of magnitude
Offsetting isn't resolved, and reconstructed scene moves around like crazy!
Next Steps
- Separate out camera perturbation process
- We're due for a retraining Fix training to use better index optimization start with linear index spacing, learn, re-fit indices using ML, learn again ** use new model w/ camera perturbation process
Future steps
- adapt camera parameters per-dataset?