Lagrangian-based Video Analytics

The origins of Lagrangian methodology are in the analysis of general dynamical systems and they have been proven to be a powerful tool for analyzing computational fluid dynamics (CFD) systems (e.g., medical devices, and for meteorological applications). Lagrangian features provide new means to compactly characterize and extract time-dependent features, such as dynamic motion boundaries and areas of coherent motion, beyond an isolated analysis and tracing of single snapshots (i.e., here: video frames).

We aim for innovative ways to process and use dynamic patterns in video motion to quantify salient motion features and thus improve computer vision performance for tasks such as identification, segmentation, and classification. The proposed methodology provides a powerful set of data-driven descriptors for continuous and integral motion analysis on variable temporal scales (i.e., for short-term as well as long-term motion features).

At its core, our approach is based on characterizing motion as a sequence of optical flow fields to assemble a time-dependent vector field that encodes the dynamics of the sequence in space-time. The analysis of dynamic vector- or flow fields is based on integral field lines (or trajectories). For Lagrangian dynamics, it is assumed that these trajectories characterize the overall dynamics (i.e., ’motion’ in a general sense) and can provide quantitative information about to the observed phenomena and the underlying physical objects. Instead of considering individual trajectories only, this information can be compactly represented within so called Lagrangian fields, which are the basis for the Lagrangian video analytics and a dense representation of trajectory sets, their spatial relation, and their integral properties.


Tobias Senst, "Estimation and Analysis of Motion in Video Data", 17.06.2019

Crowd Violence Detection Using Global Motion-Compensated Lagrangian Features and Scale-Sensitive Video-Level Representation

Lagrangian theory provides a rich set of tools for analyzing non-local, long-term motion information in computer vision applications. Based on this theory, we present a specialized Lagrangian technique for the automated detection of violent scenes in video footage. We present a novel feature using Lagrangian direction fields that is based on a spatio-temporal model and uses appearance, background motion compensation, and long-term motion information. To ensure appropriate spatial and temporal feature scales, we apply an extended bag-of-words procedure in a late-fusion manner as classification scheme on a per-video basis. We demonstrate that the temporal scale, captured by the Lagrangian integration time parameter, is crucial for violence detection and show how it correlates to the spatial scale of characteristic events in the scene. Our experiments confirm that the inclusion of Lagrangian measures is a valuable cue for automated violence detection and increases the classification performance considerably compared to state-of-the-art methods.


Senst,T., Eiselein, V., Kuhn, A., Sikora,T. Crowd Violence Detection Using Global Motion-Compensated Lagrangian Features and Scale-Sensitive Video-Level Representation, IEEE Transactions on Information Forensics and Security, 2017

Senst,T., Eiselein, V., Sikora, T. A Local Feature based on Lagrangian Measures for Violent Video Classification, ICDP, 2015

AwardsBest Paper Award @ ICDP 

A Lagrangian Framework for Video Analytics

The extraction of motion patterns from image sequences based on the optical flow methodology is an important and timely topic among visual multi media applications. In this work we will present a novel framework that combines the optical flow methodology from image processing with methods developed for the Lagrangian analysis of time-dependent vector fields. The Lagrangian approach has been proven to be a valuable and powerful tool to capture the complex dynamic motion behavior within unsteady vector fields. To come up with a compact and applicable framework, this paper will provide the concepts on how to compute trajectory-based Lagrangian measures in series of optical flow fields, a set of basic measures to capture the essence of the motion behavior within the image and a compact hierarchical, feature-based description of the resulting motion features. The resulting toolbox will bee shown to be suitable for an automated image analysis as well as compact visual analysis of image sequences in its spatio-temporal context. We show its applicability for the task of motion feature description and extraction on different temporal scales, crowd motion analysis, and automated detection of abnormal events within video sequences.


Kuhn, A., Senst, T., Keller, I., Sikora, T., Theisel, H. A Lagrangian Framework for Video Analytics, MMSP, 2012

Detecting People Carrying Objects utilizing Lagrangian Dynamics

The availability of dense motion information in computer vision domain allows for the effective application of Lagrangian techniques that have their origin in fluid flow analysis and dynamical systems theory. A well established technique that has been proven to be useful in image- based crowd analysis are Finite Time Lyapunov Exponents (FTLE). Based on this, we present a method to detect people carrying object and describe a methodology how to apply established flow field methods onto the problem of describing individuals. Further, we reinterpret Lagrangian features in relation to the underlying motion process and show their applicability towards the appearance modeling of pedestrians. This definition allows to increase performance of state-of-the-art methods and is shown to be robust under varying parameter settings and different optical flow extraction approaches.


Senst, T., Kuhn, A., Theisel, H., Sikora, T. Detecting People Carrying Objects utilizing Lagrangian Dynamics, AVSS, 2012

Human Action Recognition using Lagrangian Descriptors

Human action recognition requires the description of complex motion patterns in image sequences. In general, these patterns span varying temporal scales. In this context, Lagrangian methods have proven to be valuable for crowd analysis tasks such as crowd segmentation. In this paper, we show that, besides their potential in describing large scale motion patterns, Lagrangian methods are also well suited to model complex individual human activities over variable time intervals. We use Finite Time Lyapunov Exponents and time-normalized arc length measures in a linear SVM classification scheme. The results demonstrate that our approach is promising and that human action recognition performance is improved by fusing Lagrangian measures.


Acar, E., Senst, T., Kuhn, A., Keller, I., Theisel, H., Albayrak, S., Sikora, T. Human Action Recognition using Lagrangian Descriptors, MMSP, 2012