Video Metadata Extraction

This algorithm uses an image processing algorithm to extract metadata from each frame of a video, and returns a json file containing each frames timestamp and the resulting metadata.

note: This algorithm takes longer than most other algorithms as it recursively calls an image algorithm over each frame of a video file, make sure to add a large custom timeout to your algorithm call before using videos longer than 3 minutes.

note: This algorithm is royalty free, but the image processing algorithms it can call might not be, check your account's credit balance before running this algorithm on a custom video.

Changelog

0.6.0 - August 1st, 2018:

• Added a early termination watchdog thread that detects processing jobs that are forecasted to take longer than the maximum timeout of 50 minutes.
• various bug fixes that should improve stability and early termination in the event of a detected failure.

0.5.0 - May 23, 2017:

• Added Smart Video Downloader support to resolve a compressed gzip request problem.
• with Smart Video Downloader added, input_file may now point to any web hosted URL, including youtube, vimeo, etc.

Input

input_file: String,
output_file: String,
algorithm: String,
advanced_input: Json,
fps: Double,
image_compression: Int,
video_compression: Int

• input_file - (required) - The input url of the video file, can be any standard video container format (mp4, mkv, avi, etc), can be an http/https url or a data connector uri(data://, s3://, dropbox://, etc).
• output_file - (required) - The output url of the json file, must be a data connector uri (data://, s3://, dropbox://, etc).
• algorithm - (required) - The image processing algorithmia algorithm uri, if no default mode is found for that algorithm, advanced_input must be defined.
• advanced_input - (optional) - if you have advanced requirements or want to use an algorithm with no default parameters, See Advanced Input.
• fps - (optional) - If you have a desired fps sampling rate, input it here. defaults to the input video's fps.
• image_compression - (optional) - if you want to improve performance of processing, it's possible to compress each frame using the jpeg compression algorithm, the value provided is the associated compression ratio - from 0 (uncompressed) to 100 (most compressed) default jpeg compression is 20.
• video_compression - (optional) - by default, the output video file is raw and uncompressed, if you desire to compress your output video using the libx264 codec, provide a compression ratio value for this element, from 0 (uncompressed) to `100 (most compressed)

Advanced Input

If advanced_input is found, this algorithm will parse it as a json object; finding and replacing keywords relating to uploading and downloading files in both batch and single form with paths that the Video Metadata Extraction algorithm uses.

The keywords are:

$SINGLE_INPUT - whenever you see a single image input path for an algorithm, this keyword will work

$BATCH_INPUT - similar to single input keyword, but works for arrays of input files (Nuditydetectioni2v for example)

Example:

IllustrationTagger algorithm default input

{
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Trudeaujpg.jpg/348px-Trudeaujpg.jpg",
    "tags": ["1boy", "male", "sky", "water", "safe"]
}

Video Metadata Extraction with advanced Illustration Tagger input

{  
   "input_file": "data://path/to/file.mp4",
   "output_file": "data://save/data.json,
   "algorithm": "algo://deeplearning/IllustrationTagger/0.2.3",
   "advanced_input": {
  "image": "$SINGLE_INPUT",
  "tags": ["1boy", "male", "sky", "water", "safe"]
},
   "fps": 12.5
}

Output

{  
    "output_file": String
}

• output_file - the complete algorithmia URI for where the output json file has been uploaded to, it's format can be seen below.

formatting

The resulting output json file has the following format:

{  
   "frame_data":[  
      {  
         "timestamp":Double,
         "data":Json
      }      ...
   ]
}

• frame_data - the main json key, contains an array of json objects containing timestamps and data.
• timestamp - the timestamp of the frame in seconds from the start (frame 0 is 0.0, etc).
• data - the resulting output json data from the image processing algorithm for that particular frame, can be any json parsable format.

for example, output from nuditydetectioni2v:

{  
   "frame_data":[  
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.0
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.0666446023989338
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.1332892047978676
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.1999338071968014
      }
   ]
}

Default Options

This algorithm has default parameters set up for a number of algorithms, this list will grow as new image altering algorithms are released:

• Nudity Detection - uses version 0.2.4, in batch mode (sfw/NudityDetectioni2v)
• illustration Tagger - uses version 0.2.3, using default tags and threashold (deeplearning/SalNet)

Examples

Nuditydetectioni2v default input

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_test_nudity.json",
   "algorithm":"algo://sfw/nuditydetectioni2v"
}

Places365 Classifier advanced input

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_test_places.json",
   "algorithm":"algo://deeplearning/places365classifier/0.1.9",
   "advanced_input":{  
      "image":"$SINGLE_INPUT",
      "numResults": 15
   }
}

IllustrationTagger default input with fps

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_illustrated.json",
   "algorithm":"algo://deeplearning/IllustrationTagger",
   "fps": 14.24
}

Credits

All video processing is handled by FFMpeg, which is a fantastic video/image manipulation tool that can be found here