A Python script that uses FFMPEG to generate multiple videos at different quality settings, to aid in A/B testing. A second Python script is included to easily slice a segment of a video for such a test.

A screenshot of a video and its transcoded result, side by side

Links

  1. License
  2. Download
  3. The Video A/B Testing Script
  4. The Video Slicer Script
  5. Development Branch on Gitlab

License

The MIT License (MIT)

Copyright (c) 2016 Elliott Karpilovsky

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Download

The video-ab.zip file contains the scripts, as well as tests. While the two main scripts only require FFMPEG, the tests additionally require the Python Image Library.

The MD5 sum is provided as a quick check that the file downloaded properly. However, it does not validate the download. Verify by using the signature.

The Video A/B Testing Script

The script video_ab.py takes three required parameters: an input movie, an output directory, and a codec (one of: Theora, VP8, VP9, X264, X265, XVid, MPEG2). It then generates multiple versions of the movie in the output directory, each one transcoded at a different quality setting. The original video is placed side-by-side the transcoded version for easy comparison. A stats.txt file is also generated, containing information on the commands used to create the videos, as well as comparative bitrates.

The Video A/B Tool uses quality numbers from 0.0 (lowest quality) to 1.0 (highest quality) when describing videos. Internally, these quality numbers are mapped to codec settings, depending on the codec. The user can change the qualities on the command line, but realize that they may not have an effect on the underlying video; e.g., 1.0 and 0.999 both are mapped to to VP9’s maximum setting of CRF=1, which can only vary between 1 and 63. The stats.txt file shows the underlying codec settings for a video at a given quality.

Finally, the composition of original and the transcoded video can be changed. For example, the left half can be set to the original, and the right half set to the transcoded, as shown below.

A screenshot of a video and its transcoded result; left half of the video is the original, right half is transcoded

Flags

  • --ffmpeg-bin, -b: the location of the FFMPEG binary. The default value is “ffmpeg”.

  • --min-quality: the minimum quality to generate. The default value is 0.0.

  • --max-quality: the maximum quality to generate. The default value is 1.0.

  • --inc-quality: the amount to increment the quality between videos. The default value is 0.1. Note that a quality will be skipped if it maps to the same codec settings as a previous quality. E.g., no matter how small the incremental quality is set, the script will generate no more than 63 videos for VP9 (which can vary its CRF value between 1 and 63).

  • --mode, -m: the method to compose the original and transcoded video. The default value is “side-right”, which puts the two videos together, with the transcoded video on the right. “side-left” can be specified to put it on the left. If the two should overlay, the values “top”, “bottom”, “left”, and “right” can be used, specifying that 50% of the video on that side should be the transcoded video.

  • --force, -f: force overwriting of output. If not set, the program will exist instead of overwriting an existing file.

Examples

  • python video_ab.py /path/to/input.mkv /path/to/output vp9

  • python video_ab.py --mode right --inc-quality 0.3 /path/to/input.mkv /path/to/output x265

Caveats

  1. The calculated bitrate of the video includes the Matroska multimedia container overhead.

  2. The final output videos are losslessly compressed, to ensure artifacts are not introduced by the program. However, their file sizes are very large. It is recommended to perform A/B testing on a short video clip. See the video_slicer.py script to easily cut a video.

The Video Slicer Script

The script video_slicer.py takes two required parameters: an input video and an output destination. Based on command line flags, it will extract a segment of the input video, based on offset and length.

Flags

  • --ffmpeg-bin, -b: the location of the FFMPEG binary. The default value is “ffmpeg”.

  • --offset, -o: where the video segment should start. It can be expressed in either HH:MM:SS format, or as the keyword “random”, indicating that a random offset should be chosen.

  • --seconds, -s: the length of the sliced video

  • --ffmpeg_args: additional arguments to pass to FFMPEG while slicing. The default is “-map 0 -c copy”, which tells it to copy everything within the segment, without transcoding.

  • --force, -f: force overwriting of output. If not set, the program will exist instead of overwriting an existing file.

Examples

  • python video_slicer.py --offset 00:15:31 /path/to/input.mkv /path/to/output.mkv

  • python video_slicer.py --offset random --seconds 15 /path/to/input.mkv /path/to/output.mkv