Video thumbnailing

jcrawford · October 10, 2022, 11:39pm

I’m trying to write a plugin that generates thumbnails for videos during import. Using NReco’s ffmpeg wrapper for convenience, this is what I have so far:

using System.IO;
using System.Windows.Media.Imaging;

namespace Soletude.Stagsi.Plugins
{
    public class VidThumbta
    {

        public static void StagsiPlugin(IPluginService service)
        {
            service.RegisterImageLoaderV1(RenderVid, new ImageLoaderV1 { FileExtensions = new[] { "mp4", "m4v", "avi", "mov" } });
        }

        private static BitmapSource RenderVid(Stream input, IImageLoadingV1 parameters)
        {

            var resultImageStream = new MemoryStream();
            var ffMpeg = new NReco.VideoConverter.FFMpegConverter();
            ffMpeg.GetVideoThumbnail(parameters.FilePath, resultImageStream);

            JpegBitmapDecoder decoder = new JpegBitmapDecoder(resultImageStream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default);
            BitmapSource bmp = decoder.Frames[0];

            return bmp;
        }
    }
}

Unfortunately it’s not working at all. Here’s what I see in the logs:

Init plugin: C:\Program Files (x86)\Stagsi\Plugins\videoThumbnailer.dll : Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at ##.#Jn(PluginService , Assembly )
   at Soletude.Stagsi.Plugins.PluginService.#Rh(String[] )
22-10-10 17:27:36.225: Found plugin - C:\Program Files (x86)\Stagsi\Plugins\NReco.VideoConverter.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\psd2pixels.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\PsdPlugin.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\TxtPlugin.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\WpfPlugin.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\XamlTune.dll
Found plugin - C:\Program Files (x86)\Stagsi\Plugins\XamlTunePlugin.dll

So it seems like it’s probably failing to find one of its dependencies? But the only dependencies are components of Stagsi and the NReco.VideoConverter.dll which is present in the plugins folder along with my built DLL.

Unfortunately I know very little about .NET development so I’m not sure how to debug this further - any ideas? is it possible to get more logging out of Stagsi about loading plugins?

jcrawford · October 11, 2022, 1:50am

I’ve also tried loading this plugin by just putting the .cs file into the Plugins directory… this avoids problems with loading the plugin (Stagsi starts without logging any errors) but it seems like the plugin is never actually run, based on logging I added. Is it possible that Stagsi has a whitelist of file types that it attempts to thumbnail?

norkov · October 12, 2022, 7:01am

Hey, thanks for your attempt on a Stagsi plugin!

Sadly, I couldn’t get it to work either but the problem seems to be within NReco.

First, plugins like this are easier to handle as plain .cs files placed inside your database’s Plugins. The fact you don’t see it mentioned in the Stagsi’s log doesn’t mean it isn’t loaded - try making a typo and you will see an error box about failed compilation upon starting Stagsi. Also, you should see your FileExtensions listed in Import’s Open Dialog’s file filter.

But one thing you have forgotten is the ref block. From the docs:

with explicit reference from // ref: c:\path\to.dll comment(s) in the end of the file (ignoring blank lines)

So you should have something like this:

...
namespace Soletude.Stagsi.Plugins
{
    public class VidThumbta
...
}
// ref: C:\Users\User\Desktop\NReco.VideoConverter.dll

Again, you can confirm Stagsi is parsing your file by referring to invalid DLL in the ref comment.

Now, there are several ways to see if your plugin is actually effective. One is drawing something on a bitmap and returning it to Stagsi. See how it’s done in TxtPlugin for example. In other words, in the beginning or instead of your RenderVid():

var visual = new DrawingVisual();

using (var dc = visual.RenderOpen()) {
    dc.DrawRectangle(Brushes.White, null, new Rect(new Size(1000, 1000)));
    dc.DrawText(new FormattedText(parameters.FilePath, CultureInfo.CurrentCulture, FlowDirection.LeftToRight, new Typeface("Arial"), 10.0, Brushes.Black), new Point(0, 0));
}

var target = new RenderTargetBitmap(1000, 1000, 96.0, 96.0, PixelFormats.Default);
target.Render(visual);
return target;

If you see a rectangle with the path to the imported file then it works. You can press F3 in Stagsi to enlarge the image.

Next you can try moving parts of your original function before the code above. You will see it stops working (Stagsi no longer takes your thumbnail with the file path) at this point:

var resultImageStream = new MemoryStream();
var ffMpeg = new NReco.VideoConverter.FFMpegConverter();
ffMpeg.GetVideoThumbnail(parameters.FilePath, resultImageStream);

var visual = new DrawingVisual();
// ...

Yet you have confirmed parameters.FilePath holds the correct path, and in fact you can just replace that with a hardcoded string - the problem will remain.

Why GetVideoThumbnail() fails? To this I have no answer. You can try attaching Visual Studio debugger to Stagsi to debug the plugin. Or you can first write a trivial console program using NReco and port the code to the plugin once you confirm it works standalone. (For the reference, NReco’s documentation is here.)

Please keep me posted.

Is it possible that Stagsi has a whitelist of file types that it attempts to thumbnail?

Not really. If it had one, the plugin system would be meaningless.

is it possible to get more logging out of Stagsi about loading plugins?

No, Stagsi has certain unfortunate cases of silencing exceptions and plugin processing during import is one of them. So out of the box we don’t know the exception NReco throws, you have to use VS to see it.

jcrawford · October 13, 2022, 8:38pm

Thanks for your detailed help! I ditched the NReco wrapper and decided to do the whole thing without depending on any libraries, which of course avoids any issues with loading them but also just felt easier to debug. I now have a partially working solution, code below.

There’s one problem and a couple of smaller issues.

Problem: It wasn’t generating thumbnails for larger files, so I noticed the setting for the no thumbnail threshold. Unfortunately setting it higher than 2,147,483,647 (max 32 bit signed integer) causes “Unable to load settings” on startup, and the “Reset” option there also seems to behave oddly (settings.json was unchanged but the program seemed to no longer try to load it, logged a warning about Bootstrap.json not being loadable). This seems like probably a straightforward problem with e.g. the data structure the JSON is being parsed into? I’d really like to use a value of like 5GB for my use case.

And for the two more minor issues:

Right now terminal windows show up from the ffprobe/ffmpeg processes, which is sort of annoying. I haven’t really looked into fixing this yet because the terminal windows briefly popping up during import tells you whether or not ffprobe/ffmpeg started, and that’s been handy while debugging this.
I depend on using the parameters.FilePath, which the docs seem to suggest is not a good idea (they say it may be blank). Unfortunately ffmpeg can’t decode MP4s from stdin because it needs to seek around a couple of times during decoding, so I think this is the best approach right now (another option would be writing the stream to a temporary file for ffmpeg to use… that’s going to be slow).

Code:

using System;
using System.IO;
using System.Windows.Media.Imaging;
using System.Windows.Media;
using System.Windows;
using System.Globalization;
using System.Diagnostics;
using System.Text;
using System.Collections.Generic;

namespace Soletude.Stagsi.Plugins
{
    public class VidThumbta
    {

        public static void StagsiPlugin(IPluginService service)
        {
            service.RegisterImageLoaderV1(RenderVid, new ImageLoaderV1 { FileExtensions = new[] { "mp4", "m4v", "avi", "mov" } });
        }

        private static BitmapSource RenderVid(Stream input, IImageLoadingV1 parameters)
        {
            // Step 1: run ffprobe to determine if it is a video type.
            var outputBuilder = new StringBuilder();
            var errorBuilder = new StringBuilder();
            var startInfo = new ProcessStartInfo("ffprobe.exe");
            startInfo.RedirectStandardOutput = true;
            startInfo.UseShellExecute = false;
            var argumentBuilder = new List<string>();
            argumentBuilder.Add("-hide_banner");
            argumentBuilder.Add("-loglevel fatal");
            argumentBuilder.Add("-show_format");
            argumentBuilder.Add("-print_format json");
            argumentBuilder.Add("-read_intervals \"%+2\"");
            argumentBuilder.Add("\"" + parameters.FilePath + "\""); // input from stdin
            startInfo.Arguments = String.Join(" ", argumentBuilder.ToArray());

            var ffprobeProcess = new Process();
            ffprobeProcess.StartInfo = startInfo;
            ffprobeProcess.EnableRaisingEvents = true;
            ffprobeProcess.OutputDataReceived += new DataReceivedEventHandler
            (
                delegate(object sender, DataReceivedEventArgs e)
                {
                    // append the new data to the data already read-in
                    outputBuilder.Append(e.Data);
                }
            );
            ffprobeProcess.Start();
            ffprobeProcess.BeginOutputReadLine();
            //input.CopyTo(ffprobeProcess.StandardInput.BaseStream);
            ffprobeProcess.WaitForExit();
            ffprobeProcess.CancelOutputRead();
            string probeOutput = outputBuilder.ToString();

            // Check if this is a format we can work on
            if(!probeOutput.Contains("mp4")) {
                // Returning null tells Stagsi we can't thumbnail this file
                return null;
            }

            // Now generate the thumbnail
            var outputBuffer = new MemoryStream();
            var ffmpegErrorBuilder = new StringBuilder();
            var ffmpegStartInfo = new ProcessStartInfo("ffmpeg.exe");
            ffmpegStartInfo.RedirectStandardOutput = true;
            ffmpegStartInfo.UseShellExecute = false;
            var ffmpegArgumentBuilder = new List<string>();
            //ffmpegArgumentBuilder.Add("-loglevel fatal");
            ffmpegArgumentBuilder.Add("-ss 00:00:03.000");
            ffmpegArgumentBuilder.Add("-i \"" + parameters.FilePath + "\""); // input from stdin
            ffmpegArgumentBuilder.Add("-vframes 1");
            ffmpegArgumentBuilder.Add("pipe:.jpg");
            ffmpegStartInfo.Arguments = String.Join(" ", ffmpegArgumentBuilder.ToArray());

            var ffmpegProcess = new Process();
            ffmpegProcess.StartInfo = ffmpegStartInfo;
            ffmpegProcess.Start();
            var decoder = new JpegBitmapDecoder(ffmpegProcess.StandardOutput.BaseStream, BitmapCreateOptions.PreservePixelFormat, BitmapCacheOption.Default);
            BitmapFrame thumb = decoder.Frames[0];

            ffmpegProcess.WaitForExit();
            string ffmpegOutput = ffmpegErrorBuilder.ToString();

            return thumb;
        }
    }
}

norkov · October 14, 2022, 8:21am

I also wanted to suggest you just call ffmpeg via the command line but feared it would be too sluggish.

CreateProcess() has a special flag that hides the console window (CREATE_NO_WINDOW = 0x08000000). See SO and MSDN. We do this via Pinvoke, I think we had trouble with the native way in Net. You can do that too or you can call ffmpeg via a wrapper like NirCmd that does the hiding for you (nircmd execmd ffmpeg -i ...). I have also seen some small dedicated tool somewhere that does the same job.

This is a future-proof note. We currently don’t have any cases when FilePath may be null so your approach will work fine.

Just wanted to point out that your plugin is called for every file extension. These are only for the UI and order of trying plugins during import. As pointed out in the docs, you should check if the input at least looks like a video file you support. Running a separate tool just for that is probably quite slow (if you import thousands of videos in a batch). What our core plugins do is they read a few bytes in the beginning and compare with magic signatures.

I dug into our code and figured that this is a leftover from our (very) old implementation when we had no plugin system and relied on ImageMagick Net port to generate thumbnails (we needed support for PSD that Net doesn’t offer) and it had terrible memory leaks so we set up this limit. In the current version Stagsi reads the entire file into memory if the imported file is below NoThumbThreshold, then passes around this buffer to meta-data extractor and hash calculator rather than passing a stream (working with memory is faster than seeking). If the file is too big, its hash equals last of NoThumbThreshold bytes, and meta-data is not extracted at all. However, plugins don’t receive this buffer, they always receive the stream and yet this setting prevents them from being called along with the meta-data collection function. This is an obvious error.

I can’t promise we fix this soon but if you continue to use Stagsi in the coming months then I’ll see what can be done. In the meantime, you can generate thumbnails semi-manually for such large files as described in the docs (for example, via on-demand Tools script or via a separate program that reads Stagsi’s database, determines videos without thumbnails and generates them). You should see that bumping NoThumbThreshold to be gigabytes in size is a bad idea, it’ll kill performance.

norkov · February 13, 2024, 8:54pm

Version 1802 now provides FFMpegPlugin (and its source code).