Object and Activity Detection with Weak Labels

Office of the Director of National Intelligence (ODNI)
Reference Code
How to Apply

Create and release your Profile on Zintellect – Postdoctoral applicants must create an account and complete a profile in the on-line application system.  Please note: your resume/CV may not exceed 2 pages.

Complete your application – Enter the rest of the information required for the IC Postdoc Program Research Opportunity. The application itself contains detailed instructions for each one of these components: availability, citizenship, transcripts, dissertation abstract, publication and presentation plan, and information about your Research Advisor co-applicant.

Additional information about the IC Postdoctoral Research Fellowship Program is available on the program website located at: https://orise.orau.gov/icpostdoc/index.html.

If you have questions, send an email to ICPostdoc@orau.org.  Please include the reference code for this opportunity in your email. 

Application Deadline
2/26/2021 6:00:00 PM Eastern Time Zone

Research Topic Description, including Problem Statement:

Building systems that can understand visual concepts and describe them coherently in natural language is fundamental to artificial intelligence. Advances in machine learning have had a profound impact on computer vision and natural language processing. In particular, there has been great progress in research on object detection, descriptions of images, and video of ordinary scenes and street views captured by personal cameras. These advances have relied heavily on visual features extracted from systems trained on a large volume of strong labels (boundary boxes drawn around the designated objects). The process for acquiring such data can be expensive and time consuming. An easier and less time-consuming approach to annotating an image or sequence of images is to provide weak labels -- determining what objects, entities and characteristics are present or describing the inferred and perceived activities in natural language. Using weak labels, including natural language descriptions for commercial overhead imagery and videos, would advance research in object and activity detection.

Example Approaches:

In [1], a multiple instance learning-based (MIL) deep learning system is able to capture and localize to cancerous regions within a mega-pixel image. Here, the system is trained on images with weak binary labels determining if the image contains cancerous cells or not. An example approach is to extend this MIL approach to a multi-class problem, where each object type is a class. See also [2] and [3].

In [4], a robust change captioning was proposed to describe in natural language the activity occurs between two scenes with possibly different viewpoints, and to localize to regions that explain the inferred activity. Here, the system is trained on pair of images and a caption describing the change. See also [5]. An example approach is to develop a system to correlate visual features within and across scenes with a described activity with the help of additional self-learning tasks to improve feature representations.


[1] Campanella, et al, “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images”, Nature Medicine, 2019.

[2] Ilse, Tomczak, and Welling, “Attention-based deep multiple instance Learning”, International Conference on Machine Learning, 2018.

[3] Carion, et al, “End-to-end object detection with transformers”, arXiv:2005.12872, 2020.

[4] Park, Darrell, Rohrbach; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.

[5] Gilton, et al, “Detection and description of change in visual stream”, arXiv:2003.12633, 2020.

Relevance to the Intelligence Community:

Using weak labels in object and activity detection would increase IC efficiency. "If we were to attempt to manually exploit the commercial satellite imagery we expect to have over the next 20 years, we would need eight million imagery analysts." (Robert Cardillo, NGA Director, GEOINT Symposium 2017)

Key Words: Image Description, Object Detection, Multiple Instance Learning, Weak Labels, Weak Supervision, Computer Vision, Natural Language Processing, Artificial Intelligence


Postdoc Eligibility

  • U.S. citizens only
  • Ph.D. in a relevant field must be completed before beginning the appointment and within five years of the application deadline
  • Proposal must be associated with an accredited U.S. university, college, or U.S. government laboratory
  • Eligible candidates may only receive one award from the IC Postdoctoral Research Fellowship Program

Research Advisor Eligibility

  • Must be an employee of an accredited U.S. university, college or U.S. government laboratory
  • Are not required to be U.S. citizens
Eligibility Requirements
  • Citizenship: U.S. Citizen Only
  • Degree: Doctoral Degree.
  • Discipline(s):
    • Communications and Graphics Design (2 )
    • Computer, Information, and Data Sciences (17 )
    • Earth and Geosciences (20 )
    • Engineering (27 )
    • Environmental and Marine Sciences (15 )
    • Life Health and Medical Sciences (46 )
    • Mathematics and Statistics (11 )
    • Nanotechnology (1 )
    • Other Non-S&E (2 )
    • Other Physical Sciences (12 )
    • Physics (16 )
    • Social and Behavioral Sciences (27 )