Going Deeper into Action Recognition: A Survey

• Paper summary
• Going Deeper into Action Recognition: A Survey Herath et al. (2016)

Abstract

• The broad range of applications
  • video surveillance
  • human-computer interaction
  • retail analytics
  • user interface design
  • learning for robotics
  • web-video search and retrieval
  • medical diagnosis
  • quality-of-life improvement for elderly care
  • sports analytics
• Comprehensive review
  • handcrafted representations
  • deep learning based approaches

Introduction

What is an action?

• Human motions
  • from the simplest movement of a limb
  • to complex joint movement of a group of limbs and body
  • action seems to be hard to define
• Moeslund and Granum (2006) ¹; Poppe (2010) ²
  • action primitives as “an atomic movement that can be described at the limb level”
  • action: a diverse range of movements, from “simple and primitive ones” to “cyclic body movements”
  • activity: “a number of subsequent actions, representing a complex movement
    • Ex) left leg forward: action primitive of running
    • Ex) jumping hurdles: activity performed with the actions starting, running and jumping
• Turaga et al. (2008) ³
  • action: “simple motion patterns usually executed by a single person and typically lasting for a very short duration (order of tens of seconds)”
  • activity: “a complex sequence of actions performed by several humans who could be interacting with each other in a constrained manner.
    • Ex) actions: walking, running, or swimming
    • Ex) activity: tow persons shaking hands or a football team scoring a goal
• Wang et al. (2016) ⁴
  • action: “the change of transformation an action brings to the environment”
    • Ex) kicking a ball
• Authors
  • Action: “the most elementary and meaningful interactions” between humans and the environment
  • the category of the action: the meaning associated with this interaction

Taxonomy

1. Where to start from?

• A good representation
  • be easy to compute
  • provide description for a sufficiently large class of actions
  • reflect the similarity between two like actions
  • be robust to various variations (e.g., view-point, illumination)
• Earliest works
  • make use of 3D models
  • but 3D models is difficult and expensive
• Another solutions without 3D
  • Holistic representations
  • Local representations

1.1 Holistic representations

• Motion Energy Image (MEI) and Motion History Image (MHI)
• MEI equation

• $ D(x, y, t) $: a binary image sequence representing the detected object pixels

• Spatiotemporal volumes & spatiotemporal surfaces

• The holistic approaches are too rigid to capture possible variations of actions (e.g., view point, appearance, occlusions)
• Silhouette based representations are not capable of capturing fine details within the silhouette

2. Local Representation based Approaches

• Pipeline of action recognition using local representation
  • interest point detection → local descriptor extraction → aggregation of local descriptors.

2.1 Interest Point Detection

• Space-Time Interest Points (STIPs)
  • 3D-Harris detector: extend version of Harris corner detector

2.2 Local Descriptors

Edge and Motion Descriptors

• HoG3D: extended version of Histogram of Gradient Orientations
• HoF: Histogram of Optical Flow
• Motion Boundary Histogram (HBM)

Pixel Pattern Descriptors

• Volume local binary patterns (VLBP)
• Region Covariance Descriptor (RCD)

3. Deep Architectures for Action Recognition

• Four categories
  • Spatiotemporal networks
  • Multiple stream networks
  • Deep generative networks
  • Temporal coherency networks

3.1 Spatiotemporal networks

• Filters learned by CNN
  • in very first layers: low level features (ex. Gabor-like filters)
  • in top layers: high level semantics
• Direct approach
  • apply convolution operation with temporal information
  • 3D convolution Ji et al. (2013) ⁵
    • 3D kernels: extract features from both spatial and temporal dimensions
    • conv3d has fixed temporal domain (ex. fixed 10 frame input)
    • it is unclear why a similar assumption should be made across the temporal domain
• Various Fusion Schemes
  • Temporal pooling Ng et al. (2015) ⁶
  • Slow fusion Karpathy et al. (2014) ⁷

• C3D Tran et al. (2015) ⁸
• Factorizing conv3d into conv2d and conv1d Sun et al. (2015) ⁹
• Recurrent structure
  • Long-term Recurrent Convolutional Network Donahue et al. (2015) ¹⁰
  • Delving (GRU-RCN) Ballas et al. (2016) ¹¹
    • above paper is not mentioned in this survey paper

3.2 Multiple Stream Networks

• Fig(b) Two-stream network Simonyan and Zisserman (2014) ¹²
• Fig(c) Two-stream fusion network Feichtenhofer et al. (2016) ¹³

3.3 Deep Generative Models

less relevant to my works

3.4 Temporal Coherency Networks

less relevant to my works

4. A Quantitative Analysis

4.1 What is measured by action dataset?

• Comprehensive list of available datasets

• The complexity of a datasets
  • KTH and Weizmann (low complexity)
    • limited camera motion, almost zero background clutter
    • scope is limited
    • basic actions: walking, running and jumping
  • YouTube, movies and TV (ex. HMDB-51, UCF-101)
    • camera motion (and shake)
    • view-point variations
    • resolution inconsistencies
  • HMDB-51 and UCF-101 (medium complexity)
    • the actions are well cropped in the temporal domain
    • NOT well-suited: measuring the performance of action localization
    • exist subtle class
      • chewing and talking
      • playing violin and playing cello
  • Hollywood2 and Sports-1M (high complexity)
    • view-point/editing complexities
    • the action usually occur in a small portion of the clip
    • Sports-1M has scenes of spectators and banner adverts
  • HMDB-51, UCF-101, Hollywood2 and Sports-1M
    • cannot be distinguished by motion clues
    • the objects contributed to the actions become important
  • Deep learning need to very very much dataset
    • training on small and medium size datasets (KTH and Wiezmann) is difficult
    • Many researches exploit the pretrained model using Sports-1M dataset

4.2 Recognition Results

• column Type: Deep-net based(D), Representation based(R), Fused solution(F)

State-of-the-art solutions

• Deep-net solutions
  • The spatiotemporal networks
    • Karpathy et al. (2014) ⁷
    • Tran et al. (2015) ⁸
    • Varol et al. (2016) ¹⁴
  • Two-stream networks
    • Simonyan and Zisserman (2014) ¹²
    • Feichtenhofer et al. (2016) ¹³
  • More rigorous data augmentation
    • temporal crops by random clip sampling
    • frame skipping

4.3 What algorithmic changes to expect in future?

4.4 Bringing action recognition into life

We must fully understand the following areas in order to apply action recognition in real-life scenarios

• joint detection and recognition from a sequence
• constraining into a refined set of actions instead of big pool of classes

References