[Work Log] refactoring; dependencies

TODO

• investigate difference between ML implementations
• handle nonzero start-index in branching. re-run confirmation
• ground truth - trace datasets 7 through 10
• retrain using attachment

Invstigating consistent ML implementations

Discovered issue: was calling curve_tree_ml instead of curve_tree_ml_2. Working now

Aside: common mistakeis

Losing a too much time to stupid organizational mistakes --

• indexing,
• passing wrong copies of variables to code,
• leaving random debugging code in miscellaneous functions,
• running wrong debugging branch in code,
• running wrong version of function.

Code has gotten complex enough that managing all of these little sharp edges is too expensive. Need to start being more disciplined. Solutions

• indexing: use functions for common indexing tasks
  • one_d_to_three_d
  • to_block_index
• miscellaneous variables: clean-up workspace every evening
• Debugging cruft:
  • if code is an in-line test, refactor questionable code into functions; write a real test.
  • if code is a plot, imagesc, or printf, always wrap in a DEBUGGING block, even if its a two-minute test.
• Wrong debugging branch:
  • if using method = 1 give names like method = MARKOV
  • display warnings for non-standard methods.
• wrong version of function: depends on future of old code
  • deprecation: add block to old reference code: fprintf('running legacy code (press enter)'); pause;
  • reference implementation: rename to *_ref.

Discussion: start_index and branch_index

In previous days, I was torn over whether I should add start_index to branch_index before evaluating.

The answer is yes, because it greatly simplifies the optimization of start_index, because you don't have to re-adjust the children's branch_indices.

how is markov blanket determined (pre or post offset indexing?)

Do we need to recurse after attach?

RAW VALUES should be stored, not derived values. branch_index is a raw value.

*Example: * C is attached to B. We want to attach B to A, with start index of 10. Assume branch_index is stored relative to the zero-index (as opposed to the first observed point). After attaching, we need to update C's branch point; if we later detach B from A, B's branch point need to be updated again. There's potential for drift to accumulate after all of these upates.

To avoid confusion, replace branch_index to branch_distance; will convert to an index value before computing. Also prior_indices needs updating? Or just eliminate

recursive Updating after attachment

Consider attachment:

Before

(D -> C -> B)    (A)

After

(D -> C -> B -> A) 

Visually,

             A   B C D         A B C D 
             |   | | |         | | | |
             |   | |/          | | |/
             |   | +           | | +  
             |   |/            | |/ 
             |   +       ==>   | +
             |                 |/         
             |                 +      

• After attaching B to A,
  • C's branch index changes, so...
    • C's branch distribution changes, so...
      • D's branch distribution changes so...
        • D's ML changes
      • C's ML changes.
    • C's prior_K doesnt change.

As far as stored fields go, it looks like branch distributions change recursively after attachment, etc.

Dependencies

Did some thinking about dependencies; what needs to be updated when parents are changed. Scan of the notes are available below. Also did a reference writeup.

Dependency hand-written notes

Refactoring

Changed Corr.branch_index to Corr.branch_distance.

Replaced all instances of Corrs(i).branch_index with get_branch_index(Corrs, i).

Replaced Corr.ll_indices* with Corr.ll_distances.

removed Corr.prior_indices

TODO: replace kernel(XX,YY) with eval_kernel(asdf) see att_set_start_index_2 TODO: replace Corrs cell array with Corrs structure array

TODO

Minor

• re-run tests with non-zero start index
• test three-levels of branching
  • reconstruction
  • ML vs. reference
• cleanup Corr fields
  • eliminate clean_ fields
  • group fields by processing stage
• map-out processing pipeline

Medium

• finish ground-truthing (Friday night, Saturday)
• implement recursive update
• code for inferring branching parameters.
• Finish training
  • infer branching parameters
  • re-write training ML
  • re-train prior parameters with full ML
• MCMCDA sampler

Posted by Kyle Simek

blog comments powered by Disqus