We develop a method to fit a 3D volumetric model of the human body to pixel-level body part labels obtained via a fully convolutional neural network trained on the Human80K dataset, which provides pixel-level annotations. We represent a human body as 3D cylinders, and use priors to constrain the size and orientation of the body parts, as well as to enforce non-intersection. We then use a graphics engine (e.g., OpenGL) to efficiently render the model onto the image plane -- using a camera model, which we also estimate -- resulting in pixel-level part labels.

Imagine having two videos known to contain similar content (e.g., sports) and wanting to extract clips in which both videos feature the same semantic action (e.g., a golf swing). In this research, we approach this problem as a linear integral optimization problem, and use the primal-dual method to solve it.

Human activity recognition comprises a range of open challenges and is a very active research area, spanning topics from visual recognition of individual behavior, pairwise interactions among individuals participating in different roles in a joint activity, coordinated sequences of actions as expressions of planned activity, and multiple groups of individuals interacting across broad time scales. In this research, we address the last of these, presenting a framework for automatically constructing an interpretation of high-level human activity structure as observed in surveillance video, across multiple, interleaved instances of activities.

We wish to obtain a 3D understanding of a moving scene from videos. This includes tracking everyone's position, as well as inferring their size and orientation. Since the camera is unknown, we must also infer it. The picture shows the result of running our algorithm on a well-known dataset.

Simultaneously tracking many objects with overlapping trajectories is hard because you do not know which detections belong to which objects. We have developed a new approach to this problem and has applied it to several kinds of data. For example, the image to the left shows tubes that are growing out of pollen specs (not visible) towards ovules (out of the picture) in an in vitro plant fertilization experiment. The paths of the pollen tubes are green, and the tracks automatically found are superimposed in red.