IARPA Janus  0.4.0
IARPA Janus Program API
Classes | Typedefs | Enumerations | Functions
Janus

Mandatory interface for Phase 2 of the Janus program. More...

Classes

struct  janus_image
 Common representation for still images and video frames. More...
 
struct  janus_attributes
 Attributes for a particular object in an image. More...
 

Typedefs

typedef enum janus_error janus_error
 Return type for functions that indicate an error status. More...
 
typedef uint8_t janus_data
 Data buffer type.
 
typedef enum janus_color_space janus_color_space
 Supported image formats.
 
typedef struct janus_image janus_image
 Common representation for still images and video frames. More...
 
typedef struct janus_attributes janus_attributes
 Attributes for a particular object in an image. More...
 
typedef struct janus_template_type * janus_template
 Contains the recognition information for an object. More...
 
typedef janus_datajanus_flat_template
 A finalized representation of a template suitable for comparison. More...
 
typedef size_t janus_template_id
 Unique identifier for a janus_flat_template. More...
 
typedef const char * janus_gallery_path
 Galleries are represented in persistent storage as folders on disk. More...
 
typedef struct janus_gallery_type * janus_gallery
 An opaque reference to a read-only janus_gallery_path. More...
 

Enumerations

enum  janus_error {
  JANUS_SUCCESS = 0, JANUS_UNKNOWN_ERROR, JANUS_OUT_OF_MEMORY, JANUS_INVALID_SDK_PATH,
  JANUS_OPEN_ERROR, JANUS_READ_ERROR, JANUS_WRITE_ERROR, JANUS_PARSE_ERROR,
  JANUS_INVALID_IMAGE, JANUS_INVALID_VIDEO, JANUS_MISSING_TEMPLATE_ID, JANUS_MISSING_FILE_NAME,
  JANUS_NULL_ATTRIBUTES, JANUS_MISSING_ATTRIBUTES, JANUS_FAILURE_TO_DETECT, JANUS_FAILURE_TO_ENROLL,
  JANUS_NOT_IMPLEMENTED, JANUS_NUM_ERRORS
}
 Return type for functions that indicate an error status. More...
 
enum  janus_color_space { JANUS_GRAY8, JANUS_BGR24 }
 Supported image formats. More...
 

Functions

JANUS_EXPORT janus_error janus_detect (const janus_image image, janus_attributes *attributes_array, const size_t num_requested, size_t *num_actual)
 Detect objects in a janus_image. More...
 
JANUS_EXPORT janus_error janus_initialize (const char *sdk_path, const char *temp_path, const char *algorithm)
 Call once at the start of the application, before making any other calls to the API. More...
 
JANUS_EXPORT janus_error janus_finalize ()
 Call once at the end of the application, after making all other calls to the API. More...
 
JANUS_EXPORT janus_error janus_allocate_template (janus_template *template_)
 Allocate memory for an empty template. More...
 
JANUS_EXPORT janus_error janus_augment (const janus_image image, janus_attributes *attributes, janus_template template_)
 Add an image to the template. More...
 
JANUS_EXPORT janus_error janus_free_template (janus_template template_)
 Free memory for a template previously allocated by janus_allocate_template. More...
 
JANUS_EXPORT size_t janus_max_template_size ()
 The maximum size of templates generated by janus_flatten_template. More...
 
JANUS_EXPORT janus_error janus_flatten_template (const janus_template template_, janus_flat_template flat_template, size_t *bytes)
 Create a finalized template representation for janus_verify, janus_write_gallery or janus_search. More...
 
JANUS_EXPORT janus_error janus_verify (const janus_flat_template a, const size_t a_bytes, const janus_flat_template b, const size_t b_bytes, float *similarity)
 Return a similarity score for two templates. More...
 
JANUS_EXPORT janus_error janus_write_gallery (const janus_flat_template *templates, const size_t *templates_bytes, const janus_template_id *template_ids, const size_t num_templates, janus_gallery_path gallery_path)
 Construct a gallery on disk. More...
 
JANUS_EXPORT janus_error janus_open_gallery (janus_gallery_path gallery_path, janus_gallery *gallery)
 Initialize a gallery from a folder created in a previous call to janus_write_gallery. More...
 
JANUS_EXPORT janus_error janus_close_gallery (janus_gallery gallery)
 Free a gallery previously initialized by janus_open_gallery. More...
 
JANUS_EXPORT janus_error janus_search (const janus_flat_template probe, const size_t probe_bytes, const janus_gallery gallery, const size_t num_requested_returns, janus_template_id *template_ids, float *similarities, size_t *num_actual_returns)
 Ranked search for a template against a gallery. More...
 
JANUS_EXPORT janus_error janus_cluster (const janus_gallery gallery, const double hint, janus_template_id *template_ids, int *cluster_ids)
 Cluster a gallery into a set of identities. More...
 

Detailed Description

Mandatory interface for Phase 2 of the Janus program.

Overview

A Janus application begins with a call to janus_initialize. New templates are constructed with janus_allocate_template and provided image data with janus_detect followed by janus_augment. Templates are finalized prior to comparison with janus_flatten_template, and freed after finalization with janus_free_template.

Finalized templates can be used for verification with janus_verify, or search with janus_search. Galleries are managed with janus_write_gallery, janus_open_gallery and janus_close_gallery.

A Janus application ends with a call to janus_finalize.

Thread Safety

All functions are marked one of:

Implementer Notes

Define JANUS_LIBRARY during compilation to export Janus symbols.

Typedef Documentation

Attributes for a particular object in an image.

Attributes associated with an object. Values of NaN (isnan) indicate that the attribute value is unknown.

Attribute Definitions

Attributes collected for images in the Janus program were annotated using crowd sourcing. As a result, attributes are defined in layman's terms, instead of strict scientific definitions. Below are the instructions given to workers for each attribute, which should act as a de-facto definition of the attribute. In some cases instructions evolved slightly over time, reflecting lessons learned from communicating with the workers.

As implementations of the Janus API leverage divergent algorithms and data sets, the precise semantics of each attribute may differ by implementation.

Face

Instructions with images.

We strongly prefer tight bounding boxes over loose ones. There's a bit of leeway here, especially because it is also important to capture the whole head, but if the bounding boxes are too loose we will have to reject the work. For the case of people wearing hats or headgear, or who just have large hair, try to approximate where the head would be (it's alright if you're not exact). As a general rule of thumb, if you can see a feature on a face (an eye, nose, or mouth) or if you can tell it's a forward facing head (but it's too far away to make out individual features), box it! Please err on the side of boxing heads, the exceptions being heads that are completely facing away and heads that are blocked so you cannot see any features. It's okay if boxes overlap! But, try to only box the parts of the head you can see. Non-living faces (portraits, faces on screens, etc.) are fair game, too! If they look realistic, box them. We understand that some images aren't worth your time. In the case of image shown below, where there are many small, blurry heads in the background, just box the person in focus (or press "Not visible" if no one is in focus). In cases like this one, where many faces are distinguishable, please press the "Crowd" button. We ask that you only do this in cases where there truly are crowds - at least 25 heads or so. These images will definitely be reviewed, and wrongly pressing this button will result in a rejection.

Right Eye

For this task, click on the middle of the right eye of the subject shown. Please note that the right eye of the subject will usually appear on the left. Also note that the middle of the eye is not necessarily the pupil, but rather the midpoint between outer edges of the eye. If the right eye is not visible or too blurry to identify, press "Not Visible" - unless the eye is obscured by dark glasses, in which case we ask that you estimate where the eye would be.

Left Eye

For this task, click on the middle of the left eye of the subject shown. Please note that the left eye of the subject will usually appear on the right. Also note that the middle of the eye is not necessarily the pupil, but rather the midpoint between outer edges of the eye. If the left eye is not visible or too blurry to identify, press "Not Visible" - unless the eye is obscured by dark glasses, in which case we ask that you estimate where the eye would be.

Nose Base

For this task, click on the middle of the nose base of the subject shown. This is considered to be the middle of where the nose meets the face. If the nose base is not visible or too blurry to identify, press "Not Visibleā€.

Forehead Occlusion

For this task, choose "Forehead Covered" when:

  • A scarf, hat, or hair covers at least 25% of the area between eyebrows and hairline.
  • The area of the forehead is not contained in the image (because the edge of the image cuts off this area). Choose "Not Covered" when:
  • More than 75% of the area between eyebrows and hairline is visible.
  • Another part of the person's face is blocking part of the forehead (e.g. the person's face is tilted upwards or to the side).
  • The image is too blurry to see the forehead region well.

Eye Occlusion

For this task, choose "Eyes Covered" when:

  • Something is in front of either eye. This will often be glasses (even clear glasses) or hair, but can be anything in the image.
  • The eye area is not contained in the image. Choose "Not Covered" when:
  • Both eyes are uncovered (even if they are closed).
  • Another part of the person's face is blocking either eye (e.g. the person's face is tilted upwards or turned to the side).
  • The image is too blurry to see the eye region well

Nose and Mouth Occlusion

For this task, choose "Covered" when:

  • Something is in front of the nose or the mouth, or both. This is often a microphone or a hand, but can be anything in the image. If any type of glasses are over the lower part of the nose, this should be marked as "covered."
  • The area of the nose and mouth is not contained in the image (the edge of the image cuts off this area). Choose "Not Covered" when:
  • Both the nose and mouth are uncovered (facial hair is not considered 'covering').
  • Another part of the person's face is blocking the nose or mouth (e.g. the person's face is tilted downwards or turned to the side).
  • The image is too blurry to see the nose and mouth region well.

Indoor or Outdoor

For this task, choose "Indoor" when:

  • The scene appears to be indoors, and is inside a building or other covered structure, including any vehicle with a roof.
  • The scene might be indoors or outdoors, and there seems to be artificial light present. Choose "Outdoor" when:
  • The scene appears to be outdoors, and is not under a roof or inside a building, vehicle or structure.
  • The scene might be indoors or outdoors, and the photo appears to have only natural light (sunlight).

Age

For this task, estimate the approximate age of the person shown. The age categories in years and corresponding values in the janus_metadata , are:

0 - 19 20 - 34 35 - 49 50-64 65+ Unknown
1 2 3 4 5 0

If image quality is low, please take your best guess.

Skin Tone

Skin tone is generalized into 6 categories described below, along with their corresponding values in the janus_metadata.

Light Pink Light Yellow Medium Pink/Brown Medium Yellow/Brown Medium-Dark Brown Dark Brown
1 2 3 4 5 6

Facial Hair

For this task, there are four possible types of facial hair. Select the facial hair type that is closest to the description below.

No Facial Hair Moustache Goatee Beard
0 1 2 3
typedef enum janus_error janus_error

Return type for functions that indicate an error status.

All error values are positive integers, with the exception of JANUS_SUCCESS = 0 which indicates no error.

A finalized representation of a template suitable for comparison.

Ideally the comparison should occur without a memory copy. Alternatively, the implementation may temporarily unmarshall this buffer into a more suitable data structure.

See also
janus_template
typedef struct janus_gallery_type* janus_gallery

An opaque reference to a read-only janus_gallery_path.

Initialize with janus_open_gallery and free with janus_close_gallery. Used to perform searches with janus_search and clustering with janus_cluster.

typedef const char* janus_gallery_path

Galleries are represented in persistent storage as folders on disk.

A janus_gallery_path is the path to the gallery folder. Galleries are created by janus_write_gallery and accessed by janus_open_gallery.

typedef struct janus_image janus_image

Common representation for still images and video frames.

Pixels are stored in row-major order. In other words, pixel layout with respect to decreasing memory spatial locality is channel, column, row. Thus pixel intensity can be retrieved as follows:

1 janus_data get_intensity(janus_image image, size_t channel, size_t column,
2  size_t row)
3 {
4  const size_t columnStep = (image.image_format == JANUS_COLOR ? 3 : 1);
5  const size_t index = row*image.step + column*columnStep + channel;
6  return image.data[index];
7 }

Coordinate (0, 0) corresponds to the top-left corner of the image. Coordinate (width-1, height-1) corresponds to the bottom-right corner of the image.

typedef struct janus_template_type* janus_template

Contains the recognition information for an object.

Create a new template with janus_allocate_template.

See also
janus_flat_template
typedef size_t janus_template_id

Unique identifier for a janus_flat_template.

Associate a template with a unique identifier during janus_write_gallery. Retrieve the unique identifier from janus_search and janus_cluster.

Enumeration Type Documentation

Supported image formats.

Enumerator
JANUS_GRAY8 

1 channel grayscale, 8-bit depth.

JANUS_BGR24 

3 channel color (BGR order), 8-bit depth.

Return type for functions that indicate an error status.

All error values are positive integers, with the exception of JANUS_SUCCESS = 0 which indicates no error.

Enumerator
JANUS_SUCCESS 

No error

JANUS_UNKNOWN_ERROR 

Catch-all error code

JANUS_OUT_OF_MEMORY 

Memorry allocation failed

JANUS_INVALID_SDK_PATH 

Incorrect location provided to janus_initialize

JANUS_OPEN_ERROR 

Failed to open a file

JANUS_READ_ERROR 

Failed to read from a file

JANUS_WRITE_ERROR 

Failed to write to a file

JANUS_PARSE_ERROR 

Failed to parse file

JANUS_INVALID_IMAGE 

Could not decode image file

JANUS_INVALID_VIDEO 

Could not decode video file

JANUS_MISSING_TEMPLATE_ID 

Expected a missing template ID

JANUS_MISSING_FILE_NAME 

Expected a missing file name

JANUS_NULL_ATTRIBUTES 

Null janus_attributes

JANUS_MISSING_ATTRIBUTES 

Not all required attributes were provided

JANUS_FAILURE_TO_DETECT 

Could not localize a face within the provided image

JANUS_FAILURE_TO_ENROLL 

Could not construct a template from the provided image and attributes

JANUS_NOT_IMPLEMENTED 

Optional functions may return this value in lieu of a meaninful implementation

JANUS_NUM_ERRORS 

Idiom to iterate over all errors

Function Documentation

JANUS_EXPORT janus_error janus_allocate_template ( janus_template template_)

Allocate memory for an empty template.

Memory is managed by the implementation and guaranteed until janus_free_template.

Add images to the template with janus_augment.

1 janus_template template_;
2 janus_error error = janus_allocate_template(&template_);
3 assert(!error);
Parameters
[in]template_An uninitialized template.
Remarks
This function is reentrant.
JANUS_EXPORT janus_error janus_augment ( const janus_image  image,
janus_attributes attributes,
janus_template  template_ 
)

Add an image to the template.

The attributes should be provided from a prior call to janus_detect. As a special case, if janus_attributes::frame_rate is greater than zero, the image should be treated as the first frame in a video sequence. Subsequent calls to this function may then pass NULL for attributes, in which case the implementation should use the attributes from a previous call to this function as the seed for tracking, or as a prior for localizing the object in the current frame. In this way, janus_detect need only be called once for videos, at the start of the frame sequence.

This function may write to attributes, reflecting additional information gained during augmentation.

Augmented templates can then be passed to janus_flatten_template for verification or janus_write_gallery for gallery construction.

Parameters
[in]imageThe image containing the detected object to be recognized.
[in,out]attributesLocation and metadata associated with a single detected object to recognize.
[in,out]template_The template to contain the object's recognition information.
Remarks
This function is reentrant.
JANUS_EXPORT janus_error janus_close_gallery ( janus_gallery  gallery)

Free a gallery previously initialized by janus_open_gallery.

Parameters
[in]galleryThe gallery to close.
Remarks
This function is reentrant.
JANUS_EXPORT janus_error janus_cluster ( const janus_gallery  gallery,
const double  hint,
janus_template_id template_ids,
int *  cluster_ids 
)

Cluster a gallery into a set of identities.

The output of this function is two arrays, template_ids and cluster_ids of equal length, serving as a mapping between templates and clusters.

Clustering Hint

Clustering is generally considered to be an ill-defined problem, and most algorithms require some help determining the appropriate number of clusters. The hint parameter helps influence the number of clusters, though the implementation is free to ignore it.

  • If hint is in the range [-1, 1] then it is a clustering aggressiveness, with -1 favoring more clusters (fewer templates per cluster), and 1 favoring fewer clusters (more templates per cluster).
  • If hint is greater than 1 then it is a clustering count, indicating the suggested number of clusters.
  • The suggested default value for hint is 0.

Gallery Size

The size of the gallery is number of templates passed to janus_write_gallery.

Note
The implementation of this function is optional, and may return JANUS_NOT_IMPLEMENTED.
Parameters
[in]galleryThe gallery to cluster.
[in]hintA hint to the clustering algorithm, see Clustering Hint.
[out]template_idsA pre-allocated array provided by the calling application large enough to hold Gallery Size elements.
[out]cluster_idsA pre-allocated array provided by the calling application large enough to hold Gallery Size elements.
Remarks
This function is thread_safe.
JANUS_EXPORT janus_error janus_detect ( const janus_image  image,
janus_attributes attributes_array,
const size_t  num_requested,
size_t *  num_actual 
)

Detect objects in a janus_image.

Each object is represented by a janus_attributes. In the case that the number of detected objects is greater than num_requested, the implementation may choose which detections to exclude, potentially returning early before scanning the entire image. Detected objects can then be used in janus_augment.

Detection Guarantees

The first num_actual elements of attributes_array will be populated by decreasing janus_attributes::detection_confidence.

Each of the num_actual detections will have values for at least the following attributes:

Any attribute of the num_actual detections without a value will be set to NaN.

Parameters
[in]imageImage to detect objects in.
[out]attributes_arrayPre-allocated array of uninitialized janus_attributes. Expected to be at least num_requested * sizeof(janus_attributes) bytes long.
[in]num_requestedLength of attributes_array.
[out]num_actualThe number of detections made by the system. If num_actual <= num_requested, then num_actual is the length of attributes_array populated with detected objects by the implementation. Otherwise, num_actual > num_requested, then attributes_array is fully populated and there were additional detections that weren't returned.
Remarks
This function is thread_safe.
JANUS_EXPORT janus_error janus_finalize ( )

Call once at the end of the application, after making all other calls to the API.

Remarks
This function is thread_unsafe and should only be called once.
See also
janus_initialize
JANUS_EXPORT janus_error janus_flatten_template ( const janus_template  template_,
janus_flat_template  flat_template,
size_t *  bytes 
)

Create a finalized template representation for janus_verify, janus_write_gallery or janus_search.

Parameters
[in]template_The recognition information to construct the finalized template from.
[in,out]flat_templateA pre-allocated buffer provided by the calling application no smaller than janus_max_template_size to contain the finalized template.
[out]bytesSize of the buffer actually used to store the template.
Remarks
This function is reentrant.
JANUS_EXPORT janus_error janus_free_template ( janus_template  template_)

Free memory for a template previously allocated by janus_allocate_template.

Call this function on a template after it is no longer needed.

Parameters
[in]template_The template to deallocate.
Remarks
This function is reentrant.
See also
janus_allocate_template
JANUS_EXPORT janus_error janus_initialize ( const char *  sdk_path,
const char *  temp_path,
const char *  algorithm 
)

Call once at the start of the application, before making any other calls to the API.

Parameters
[in]sdk_pathPath to the read-only directory containing the janus-compliant SDK as provided by the implementer.
[in]temp_pathPath to an existing empty read-write directory for use as temporary file storage by the implementation. This path is guaranteed until janus_finalize.
[in]algorithmAn empty string indicating the default algorithm, or an implementation-defined string indicating an alternative configuration.
Remarks
This function is thread_unsafe and should only be called once.
See also
janus_finalize
JANUS_EXPORT size_t janus_max_template_size ( )

The maximum size of templates generated by janus_flatten_template.

Should be less than or equal to 32 MB.

Remarks
This function is thread_safe.
JANUS_EXPORT janus_error janus_open_gallery ( janus_gallery_path  gallery_path,
janus_gallery gallery 
)

Initialize a gallery from a folder created in a previous call to janus_write_gallery.

Janus galleries are stored as folders on disk and accessed using the opaque janus_gallery type. Close the gallery when it is no longer needed with janus_close_gallery.

Parameters
[in]gallery_pathRead-only folder populated by a previous call to janus_write_gallery.
[out]galleryPointer to an uninitialized gallery.
Remarks
This function is reentrant.
JANUS_EXPORT janus_error janus_search ( const janus_flat_template  probe,
const size_t  probe_bytes,
const janus_gallery  gallery,
const size_t  num_requested_returns,
janus_template_id template_ids,
float *  similarities,
size_t *  num_actual_returns 
)

Ranked search for a template against a gallery.

template_ids and similarities should be pre-allocated buffers large enough to contain requested_returns elements. actual_returns will be less than or equal to requested_returns, depending on the contents of the gallery.

The returned similarities may be normalized by the implementation based on the contents of the gallery. Therefore, similarity scores returned from searches against different galleries are not guaranteed to be comparable.

Parameters
[in]probeProbe to search for.
[in]probe_bytesSize of probe.
[in]galleryGallery to search against.
[in]num_requested_returnsThe desired number of returned results.
[out]template_idsBuffer to contain the janus_template_id of the top matching gallery templates.
[out]similaritiesBuffer to contain the similarity scores of the top matching templates.
[out]num_actual_returnsThe number of populated elements in template_ids and similarities. This value could be zero.
Remarks
This function is thread_safe.
See also
janus_verify
JANUS_EXPORT janus_error janus_verify ( const janus_flat_template  a,
const size_t  a_bytes,
const janus_flat_template  b,
const size_t  b_bytes,
float *  similarity 
)

Return a similarity score for two templates.

Higher scores indicate greater similarity.

The returned similarity score is symmetric. In other words, swapping the order of a and b will not change similarity.

Parameters
[in]aThe first template to compare.
[in]a_bytesSize of template a.
[in]bThe second template to compare.
[in]b_bytesSize of template b.
[out]similarityHigher values indicate greater similarity.
Remarks
This function is thread_safe.
See also
janus_search
JANUS_EXPORT janus_error janus_write_gallery ( const janus_flat_template templates,
const size_t *  templates_bytes,
const janus_template_id template_ids,
const size_t  num_templates,
janus_gallery_path  gallery_path 
)

Construct a gallery on disk.

Access the constructed gallery with janus_open_gallery.

Note
The caller is advised to memory map templates in the event that they approach or exceed the available system memory.
Parameters
[in]templatesArray of templates to comprise the gallery.
[in]templates_bytesArray of sizes for each template.
[in]template_idsArray of janus_template_id for each template.
[in]num_templatesLength of templates, templates_bytes, and template_ids.
[in]gallery_pathPath to an empty read-write folder to store the gallery.
Remarks
This function is reentrant.