[Work Log] Work Log
Correspondence Misalignment
WACV ground truth reconstruction errors are due to correpondence misalignment that occurs when different views don't line up (esp because of drift in the plant position over time).
Ultimately, this causes index set estimates to be suboptimal. To correct this, we can either focus on fixing the point-correspondence stage, or fixing the index set directly, after the correspondence stage.
Two possible approaches come to mind.
The first approach attempts to fix the point correspondences. Do some keypoint matching and use this to constrain certain parts of the correspondence-matching (i.e. force the dynamic programming to pass through certain entries in the cost matrix). Currently unclear how to deal with incorrect keypoint matches, or many-to-many correspondenes.
The second approach is to do some optimization on the index points to improve the marginal likelihood.
Optimizing index points
The following assumes that the problem is the index set obtained by greedy point correspondence is sub-optimal under the marginal likelihood, and the indices at the true optimium would be immune to misalignment, because our kernel is robust to trnalsation and scaling errors.
Doing 350-dimensional optimization seems like a terrible pain, but we might be able to take some shortcuts.
First, we observe that the analytical derivative of the ML w.r.t. any change in the index set will require a computation on the same order as evaluating the marginal likelihood (nxn matrix inversion, multiplication of several nxn matrices).
However, if we cache some values, we can avoid matrix inversion when computing subsequent elements of the gradient. This still leaves nxn matrix multilication, but this becomes sparse, since most elements of K don't change when a single index changes.
Alternatively, we can choose a few good directions and optimize in those directions only. For example, when (a) all values are shifted together, (b) one endpoint is unchanged, and each subsequent index is changed more and (c) the complement of b.
Regardless of the gradient approach, we still have to do a matrix inverse at every iteration. We might be able to do this blockwise and save computation for indices that don't change. Are there other approximations that we can do? This doesn't need to be exact, but good enough to improve the indices
Lets keep in mind, we're dealing with relatively small matrices, here.
Other thoughts
It's possible a long, curvy curve (like the one's we get when misalignment occurs) are actually the result best supported under the marginal likelihood. In that case, we need to think more deeply about our overall approach.
Camera calibration
There is likely some room for improvement in the camera calibration. Maybe this is the best approach overall?
Tasks
- Improve camera calibration
- Write function to get derivatives of covariance matrix.
- Try index optimization.