opencv - 如何将 Xamarin.android 与 OpenCV 集成

Question

我正在尝试在 Visual Studio 中将 Xamarin.android 与 OpenCV 连接起来，文档真的很差，谁能给我提供一些操作步骤

Answer 1

在 Xamarin.Android 上使用 OpenCV 的方法不止一种:

1 - 使用 OpenCV4Android 的绑定(bind):OpenCV4Android 是使用 JNI 的 Android (Java) 的 OpenCV (C++) 包装器。通过绑定(bind)，我们有一个 Java 和 C# 之间的接口(interface)(更多信息在 https://learn.microsoft.com/en-us/xamarin/android/platform/binding-java-library/ 中)。

它在 https://github.com/jeremy-ellis-tech/Xamarin.Android.OpenCV 中实现，其中使用了 OpenCV 3.1.0。 您可以按照安装说明和“减少 .dll 大小”来生成一个 dll 并引用它，或者您可以在您的 Visual Studio 解决方案中插入文件夹“/src/OpenCV.Binding”的 Visual Studio 项目并添加一个引用您的项目到这个新项目。

如果您想使用较新的版本，您可以下载较新版本的 OpenCV4Android(OpenCV SourceForge 中名为 opencv-version-android-sdk.zip 的文件，例如:链接 https://sourceforge.net/projects/opencvlibrary/files/4.1.0/ 上的 OpenCV4Android 4.1.0)和将先前项目中文件夹“/src/OpenCV.Binding/Jars”的内容替换为提取的 OpenCV4Android 文件夹“/sdk/native/libs”和“/sdk/native/3rdparty/libs”的内容。

还有这个项目的一个分支的 NuGet:https://www.nuget.org/packages/Xamarin.OpenCV.Droid可以简化安装和使用，但我没用过这个，所以我不能说它是否有效。

由于此方法是 OpenCV4Android 的绑定(bind)而不是纯 OpenCV，您将使用 OpenCV4Android 的文档 (https://opencv.org/android/)。还值得一提的是，通过这种方式，我们拥有三层编程语言(C# - Java - C++)，因此我们在方法调用方面存在性能损失(JNI 是一种负担)。因此，建议您尽量少调用。

2 - 使用 OpenCV C++ 的包装器:通过这种方式，我们将使用 C++ 共享库 (.so) 并从 C# (https://learn.microsoft.com/en-us/xamarin/android/platform/native-libraries) 调用其方法。 为此，我们需要编写 OpenCV 方法的 PInvoke，这些方法很多，这意味着很多时间。所以我们将使用别人已经制作的东西。

我们有 OpenCvSharp，它是 OpenCV 到 .NET 的包装器，显然运行良好。问题:它与 ARM 不兼容，因此无法在智能手机上运行。 然而，一个好心人将它改编为在 ARM 设备上运行:https://github.com/Kawaian/OpenCvSharp .

如何快速使用它:在您的解决方案中插入文件夹“/src/OpenCvSharp”的项目并引用它。您将“/src/OpenCvSharp.Android/Native”的内容复制到项目的“lib”或“libs”文件夹中。然后将“.so”文件配置为“始终复制”到 OutPut 目录，并将其构建操作配置为“AndroidNativeLibrary”(如果您的项目是应用程序)或“嵌入式 native 库”(如果您的项目是 Android 库)。

另一种方法是安装 NuGet ( https://www.nuget.org/packages/Kawaian.OpenCVSharp/ )，这会更容易一些，但也需要将“.so”文件复制到“lib”或“libs”并进行配置。

这个包装器使用 OpenCV 3.2.0。我正在研究一种方法来更新此项目的 OpenCV 版本，但目前它有效。

这种方式的最大优势是性能(比较两种实现时，我的应用程序提高了约 30%)。但一个缺点是缺少已经制作好的 Android.Bitmap - OpenCV.Mat 转换方法。我通过适配 OpenCV4Android 的转换方法实现了它们:

// Method adapted from the original OpenCV convert at https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/modules/java/generator/src/cpp/utils.cpp
///<summary>
///This function converts an image in the OpenCV Mat representation to the Android Bitmap.
///The input Mat object has to be of the types 'CV_8UC1' (gray-scale), 'CV_8UC3' (RGB) or 'CV_8UC4' (RGBA).
///The output Bitmap object has to be of the same size as the input Mat and of the types 'ARGB_8888' or 'RGB_565'.
///This function throws an exception if the conversion fails.
///</summary>
///<param name="srcImage">srcImage is a valid input Mat object of types 'CV_8UC1', 'CV_8UC3' or 'CV_8UC4'</param>
///<param name="dstImage">dstImage is a valid Bitmap object of the same size as the Mat and of type 'ARGB_8888' or 'RGB_565'</param>
///<param name="needPremultiplyAlpha">premultiplyAlpha is a flag, that determines, whether the Mat needs to be converted to alpha premultiplied format (like Android keeps 'ARGB_8888' bitmaps); the flag is ignored for 'RGB_565' bitmaps.</param>
public static void MatToBitmap(Mat srcImage, Bitmap dstImage, bool needPremultiplyAlpha = false)
{
    var bitmapInfo = dstImage.GetBitmapInfo();
    var bitmapPixels = dstImage.LockPixels();
 
    if (bitmapInfo.Format != Format.Rgba8888 && bitmapInfo.Format != Format.Rgb565)
        throw new Exception("Invalid Bitmap Format: It is of format " + bitmapInfo.Format.ToString() + " and is expected Rgba8888 or Rgb565");
    if (srcImage.Dims() != 2)
        throw new Exception("The source image has " + srcImage.Dims() + " dimensions, while it is expected 2");
    if (srcImage.Cols != bitmapInfo.Width || srcImage.Rows != bitmapInfo.Height)
        throw new Exception("The source image and the output Bitmap don't have the same amount of rows and columns");
    if (srcImage.Type() != MatType.CV_8UC1 && srcImage.Type() != MatType.CV_8UC3 && srcImage.Type() != MatType.CV_8UC4)
        throw new Exception("The source image has the type " + srcImage.Type().ToString() + ", while it is expected CV_8UC1, CV_8UC3 or CV_8UC4");
    if (bitmapPixels == null)
        throw new Exception("Can't lock the output bitmap");
 
    if (bitmapInfo.Format == Format.Rgba8888)
    {
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4, bitmapPixels);
        if (srcImage.Type() == MatType.CV_8UC1)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC1 -> RGBA_8888");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.GRAY2RGBA);
        }
        else if (srcImage.Type() == MatType.CV_8UC3)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC3 -> RGBA_8888");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGB2RGBA);
        }
        else if (srcImage.Type() == MatType.CV_8UC4)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC4 -> RGBA_8888");
            if (needPremultiplyAlpha)
                Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGBA2mRGBA);
            else
                srcImage.CopyTo(tmp);
        }
    }
    else
    {
        // info.format == ANDROID_BITMAP_FORMAT_RGB_565
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC2, bitmapPixels);
        if (srcImage.Type() == MatType.CV_8UC1)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC1 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.GRAY2BGR565);
        }
        else if (srcImage.Type() == MatType.CV_8UC3)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC3 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGB2BGR565);
        }
        else if (srcImage.Type() == MatType.CV_8UC4)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC4 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGBA2BGR565);
        }
    }
    dstImage.UnlockPixels();
    return;
}

// Method adapted from the original OpenCV convert at https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/modules/java/generator/src/cpp/utils.cpp
///<summary>
///This function converts an Android Bitmap image to the OpenCV Mat.
///'ARGB_8888' and 'RGB_565' input Bitmap formats are supported.
///The output Mat is always created of the same size as the input Bitmap and of the 'CV_8UC4' type,it keeps the image in RGBA format.
///This function throws an exception if the conversion fails.
///</summary>
///<param name="srcImage">srcImage is a valid input Bitmap object of the type 'ARGB_8888' or 'RGB_565'</param>
///<param name="dstImage">dstImage is a valid output Mat object, it will be reallocated if needed, so it may be empty.</param>
///<param name="needUnPremultiplyAlpha">unPremultiplyAlpha is a flag, that determines, whether the bitmap needs to be converted from alpha premultiplied format (like Android keeps 'ARGB_8888' ones) to regular one; this flag is ignored for 'RGB_565' bitmaps.</param>
public static void BitmapToMat(Bitmap srcImage, Mat dstImage, bool needUnPremultiplyAlpha = false)
{
    var bitmapInfo = srcImage.GetBitmapInfo();
    var bitmapPixels = srcImage.LockPixels();
 
    if (bitmapInfo.Format != Format.Rgba8888 && bitmapInfo.Format != Format.Rgb565)
        throw new Exception("Invalid Bitmap Format: It is of format " + bitmapInfo.Format.ToString() + " and is expected Rgba8888 or Rgb565");
    if (bitmapPixels == null)
        throw new Exception("Can't lock the source bitmap");
 
    dstImage.Create((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4);
 
    if (bitmapInfo.Format == Format.Rgba8888)
    {
        Android.Util.Log.Info("nBitmapToMat", "RGBA_8888 -> CV_8UC4");
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4, bitmapPixels);
        if (needUnPremultiplyAlpha)
            Cv2.CvtColor(tmp, dstImage, ColorConversionCodes.mRGBA2RGBA);
        else
            tmp.CopyTo(dstImage);
    }
    else
    {
        // info.format == ANDROID_BITMAP_FORMAT_RGB_565
        Android.Util.Log.Info("nBitmapToMat", "RGB_565 -> CV_8UC4");
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC2, bitmapPixels);
        Cv2.CvtColor(tmp, dstImage, ColorConversionCodes.BGR5652RGBA);
    }
 
    srcImage.UnlockPixels();
    return;
}