Video Metadata Extraction

This algorithm uses an image processing algorithm to extract metadata from each frame of a video, and returns a json file containing each frames timestamp and the resulting metadata.

note: This algorithm takes longer than most other algorithms as it recursively calls an image algorithm over each frame of a video file, make sure to add a large custom timeout to your algorithm call before using videos longer than 3 minutes.

note: This algorithm is royalty free, but the image processing algorithms it can call might not be, check your account's credit balance before running this algorithm on a custom video.

Changelog

0.6.0 - August 1st, 2018:

Added a early termination watchdog thread that detects processing jobs that are forecasted to take longer than the maximum timeout of 50 minutes.
various bug fixes that should improve stability and early termination in the event of a detected failure.

0.5.0 - May 23, 2017:

Added Smart Video Downloader support to resolve a compressed gzip request problem.
with Smart Video Downloader added, input_file may now point to any web hosted URL, including youtube, vimeo, etc.

Input

input_file: String,
output_file: String,
algorithm: String,
advanced_input: Json,
fps: Double,
image_compression: Int,
video_compression: Int

input_file - (required) - The input url of the video file, can be any standard video container format (mp4, mkv, avi, etc), can be an http/https url or a data connector uri(data://, s3://, dropbox://, etc).
output_file - (required) - The output url of the json file, must be a data connector uri (data://, s3://, dropbox://, etc).
algorithm - (required) - The image processing algorithmia algorithm uri, if no default mode is found for that algorithm, advanced_input must be defined.
advanced_input - (optional) - if you have advanced requirements or want to use an algorithm with no default parameters, See Advanced Input.
fps - (optional) - If you have a desired fps sampling rate, input it here. defaults to the input video's fps.
image_compression - (optional) - if you want to improve performance of processing, it's possible to compress each frame using the jpeg compression algorithm, the value provided is the associated compression ratio - from 0 (uncompressed) to 100 (most compressed) default jpeg compression is 20.
video_compression - (optional) - by default, the output video file is raw and uncompressed, if you desire to compress your output video using the libx264 codec, provide a compression ratio value for this element, from 0 (uncompressed) to `100 (most compressed)

Advanced Input

If advanced_input is found, this algorithm will parse it as a json object; finding and replacing keywords relating to uploading and downloading files in both batch and single form with paths that the Video Metadata Extraction algorithm uses.

The keywords are:

$SINGLE_INPUT - whenever you see a single image input path for an algorithm, this keyword will work

$BATCH_INPUT - similar to single input keyword, but works for arrays of input files (Nuditydetectioni2v for example)

Example:

IllustrationTagger algorithm default input

{
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/9/9a/Trudeaujpg.jpg/348px-Trudeaujpg.jpg",
    "tags": ["1boy", "male", "sky", "water", "safe"]
}

Video Metadata Extraction with advanced Illustration Tagger input


{  
   "input_file": "data://path/to/file.mp4",
   "output_file": "data://save/data.json,
   "algorithm": "algo://deeplearning/IllustrationTagger/0.2.3",
   "advanced_input": {
  "image": "$SINGLE_INPUT",
  "tags": ["1boy", "male", "sky", "water", "safe"]
},
   "fps": 12.5
}

Output

{  
    "output_file": String
}

output_file - the complete algorithmia URI for where the output json file has been uploaded to, it's format can be seen below.

formatting

The resulting output json file has the following format:

{  
   "frame_data":[  
      {  
         "timestamp":Double,
         "data":Json
      }      ...
   ]
}

frame_data - the main json key, contains an array of json objects containing timestamps and data.
timestamp - the timestamp of the frame in seconds from the start (frame 0 is 0.0, etc).
data - the resulting output json data from the image processing algorithm for that particular frame, can be any json parsable format.

for example, output from nuditydetectioni2v:

{  
   "frame_data":[  
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.0
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.0666446023989338
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.1332892047978676
      },
      {  
         "data":{  
            "confidence":1,
            "nude":false
         },
         "timestamp":0.1999338071968014
      }
   ]
}

Default Options

This algorithm has default parameters set up for a number of algorithms, this list will grow as new image altering algorithms are released:

Nudity Detection - uses version 0.2.4, in batch mode (sfw/NudityDetectioni2v)
illustration Tagger - uses version 0.2.3, using default tags and threashold (deeplearning/SalNet)

Examples

Nuditydetectioni2v default input

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_test_nudity.json",
   "algorithm":"algo://sfw/nuditydetectioni2v"
}

Places365 Classifier advanced input

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_test_places.json",
   "algorithm":"algo://deeplearning/places365classifier/0.1.9",
   "advanced_input":{  
      "image":"$SINGLE_INPUT",
      "numResults": 15
   }
}

IllustrationTagger default input with fps

{  
   "input_file":"data://media/videos/kenny_test.mp4",
   "output_file":"data://.algo/temp/kenny_illustrated.json",
   "algorithm":"algo://deeplearning/IllustrationTagger",
   "fps": 14.24
}

Credits

All video processing is handled by FFMpeg, which is a fantastic video/image manipulation tool that can be found here