json - 使用 System.Text.Json 高效替换大型 JSON 的属性

Question

我正在处理大型 JSON，其中一些元素包含以 Base 64 编码的大型(最多 100MB)文件。例如:

{ "name": "One Name", "fileContent": "...base64..." }

我想将 fileContent 属性值存储在磁盘中(以字节为单位)并将其替换为文件的路径，如下所示:

{ "name": "One Name", "fileRoute": "/route/to/file" }

是否有可能通过 System.Text.Json 使用流或任何其他方式来避免必须在内存中处理非常大的 JSON 来实现这一点？

Answer 1

您的基本要求是转换包含属性 "fileContent": "...base64..." 的 JSON至 "fileRoute": "/route/to/file"同时还写入 fileContent 的值out 到一个单独的二进制文件 ，而不具体化 fileContent 的值作为一个完整的字符串。

目前尚不清楚这是否可以通过 System.Text.Json 的 .NET Core 3.1 实现来完成。 .就算可以，也不容易。简单地生成一个 Utf8JsonReader来自Stream需要工作，请参阅 Parsing a JSON file with .NET core 3.0/System.text.Json 。这样做之后，有一个方法 Utf8JsonReader.ValueSequence 返回最后处理的 token 的原始值作为 ReadOnlySequence<byte>输入有效载荷的切片。但是，该方法似乎并不易于使用，因为它仅在 token 包含在多个段中时才有效，不能保证值的格式正确，并且不会对 JSON 转义序列进行转义。

Newtonsoft 在这里根本无法工作，因为 JsonTextReader 始终完全具体化每个原始字符串值。

作为替代方案，您可以考虑由 JsonReaderWriterFactory 返回的读者和作者.这些读者和作者被 DataContractJsonSerializer 使用并即时将 JSON 转换为 XML read和 written .由于这些读者和作者的基类是 XmlReader和 XmlWriter ，它们支持通过 XmlReader.ReadValueChunk(Char[], Int32, Int32) 读取 block 中的字符串值.更好的是，它们支持通过 XmlReader.ReadContentAsBase64(Byte[], Int32, Int32) 读取 block 中的 Base64 二进制值。 .

给定这些读者和作者，我们可以使用流式转换算法来转换 fileContent节点到 fileRoute节点，同时将 Base64 二进制文件提取到单独的二进制文件中。

首先，介绍以下基于 Combining the XmlReader and XmlWriter classes for simple streaming transformations 的 XML 流转换方法。 来自 Mark Fussell和 this answer给 Automating replacing tables from external files :

public static class XmlWriterExtensions
{
    // Adapted from this answer https://stackoverflow.com/a/28903486
    // to https://stackoverflow.com/questions/28891440/automating-replacing-tables-from-external-files/
    // By https://stackoverflow.com/users/3744182/dbc

    /// <summary>
    /// Make a DEEP copy of the current xmlreader node to xmlwriter, allowing the caller to transform selected elements.
    /// </summary>
    /// <param name="writer"></param>
    /// <param name="reader"></param>
    /// <param name="shouldTransform"></param>
    /// <param name="transform"></param>
    public static void WriteTransformedNode(this XmlWriter writer, XmlReader reader, Predicate<XmlReader> shouldTransform, Action<XmlReader, XmlWriter> transform)
    {
        if (reader == null || writer == null || shouldTransform == null || transform == null)
            throw new ArgumentNullException();

        int d = reader.NodeType == XmlNodeType.None ? -1 : reader.Depth;
        do
        {
            if (reader.NodeType == XmlNodeType.Element && shouldTransform(reader))
            {
                using (var subReader = reader.ReadSubtree())
                {
                    transform(subReader, writer);
                }
                // ReadSubtree() places us at the end of the current element, so we need to move to the next node.
                reader.Read();
            }
            else
            {
                writer.WriteShallowNode(reader);
            }
        }
        while (!reader.EOF && (d < reader.Depth || (d == reader.Depth && reader.NodeType == XmlNodeType.EndElement)));
    }

    /// <summary>
    /// Make a SHALLOW copy of the current xmlreader node to xmlwriter, and advance the XML reader past the current node.
    /// </summary>
    /// <param name="writer"></param>
    /// <param name="reader"></param>
    public static void WriteShallowNode(this XmlWriter writer, XmlReader reader)
    {
        // Adapted from https://learn.microsoft.com/en-us/archive/blogs/mfussell/combining-the-xmlreader-and-xmlwriter-classes-for-simple-streaming-transformations
        // By Mark Fussell https://learn.microsoft.com/en-us/archive/blogs/mfussell/
        // and rewritten to avoid using reader.Value, which fully materializes the text value of a node.
        if (reader == null)
            throw new ArgumentNullException("reader");
        if (writer == null)
            throw new ArgumentNullException("writer");

        switch (reader.NodeType)
        {   
            case XmlNodeType.None:
                // This is returned by the System.Xml.XmlReader if a Read method has not been called.
                reader.Read();
                break;

            case XmlNodeType.Element:
                writer.WriteStartElement(reader.Prefix, reader.LocalName, reader.NamespaceURI);
                writer.WriteAttributes(reader, true);
                if (reader.IsEmptyElement)
                {
                    writer.WriteEndElement();
                }
                reader.Read();
                break;

            case XmlNodeType.Text:
            case XmlNodeType.Whitespace:
            case XmlNodeType.SignificantWhitespace:
            case XmlNodeType.CDATA:
            case XmlNodeType.XmlDeclaration:
            case XmlNodeType.ProcessingInstruction:
            case XmlNodeType.EntityReference:
            case XmlNodeType.DocumentType:
            case XmlNodeType.Comment:
                //Avoid using reader.Value as this will fully materialize the string value of the node.  Use WriteNode instead,
                // it copies text values in chunks.  See: https://referencesource.microsoft.com/#system.xml/System/Xml/Core/XmlWriter.cs,368
                writer.WriteNode(reader, true);
                break;

            case XmlNodeType.EndElement:
                writer.WriteFullEndElement();
                reader.Read();
                break;

            default:
                throw new XmlException(string.Format("Unknown NodeType {0}", reader.NodeType));
        }
    }
}

public static partial class XmlReaderExtensions
{
    // Taken from this answer https://stackoverflow.com/a/54136179/3744182
    // To https://stackoverflow.com/questions/54126687/xmlreader-how-to-read-very-long-string-in-element-without-system-outofmemoryex
    // By https://stackoverflow.com/users/3744182/dbc
    public static bool CopyBase64ElementContentsToFile(this XmlReader reader, string path)
    {
        using (var stream = File.Create(path))
        {
            byte[] buffer = new byte[8192];
            int readBytes = 0;

            while ((readBytes = reader.ReadElementContentAsBase64(buffer, 0, buffer.Length)) > 0)
            {
                stream.Write(buffer, 0, readBytes);
            }
        }
        return true;
    }
}

接下来，使用 JsonReaderWriterFactory ，引入以下方法从一个 JSON 文件流式传输到另一个，重写 fileContent需要的节点:

public static class JsonPatchExtensions
{
    public static string[] PatchFileContentToFileRoute(string oldJsonFileName, string newJsonFileName, FilenameGenerator generator)
    {
        var newNames = new List<string>();

        using (var inStream = File.OpenRead(oldJsonFileName))
        using (var outStream = File.Open(newJsonFileName, FileMode.Create))
        using (var xmlReader = JsonReaderWriterFactory.CreateJsonReader(inStream, XmlDictionaryReaderQuotas.Max))
        using (var xmlWriter = JsonReaderWriterFactory.CreateJsonWriter(outStream))
        {
            xmlWriter.WriteTransformedNode(xmlReader, 
                r => r.LocalName == "fileContent" && r.NamespaceURI == "",
                (r, w) =>
                {
                    r.MoveToContent();
                    var name = generator.GenerateNewName();
                    r.CopyBase64ElementContentsToFile(name);
                    w.WriteStartElement("fileRoute", "");
                    w.WriteAttributeString("type", "string");
                    w.WriteString(name);
                    w.WriteEndElement();
                    newNames.Add(name);
                });
        }

        return newNames.ToArray();
    }
}

public abstract class FilenameGenerator
{
    public abstract string GenerateNewName();
}

// Replace the following with whatever algorithm you need to generate unique binary file names.

public class IncrementalFilenameGenerator : FilenameGenerator
{
    readonly string prefix;
    readonly string extension;
    int count = 0;

    public IncrementalFilenameGenerator(string prefix, string extension)
    {
        this.prefix = prefix;
        this.extension = extension;
    }

    public override string GenerateNewName()
    {
        var newName = Path.ChangeExtension(prefix + (++count).ToString(), extension);
        return newName;
    }
}

然后调用如下:

var binaryFileNames = JsonPatchExtensions.PatchFileContentToFileRoute(
    oldJsonFileName, 
    newJsonFileName,
    // Replace the following with your actual binary file name generation algorithm
    new IncrementalFilenameGenerator("Question59839437_fileContent_", ".bin"));

来源:

• Combining the XmlReader and XmlWriter classes for simple streaming transformations 来自 Mark Fussell

• Automating replacing tables from external files 。

• XmlReader - How to read very long string in element without System.OutOfMemoryException 。

• Parse huge OData JSON by streaming certain sections of the json to avoid LOH 。

• Mapping Between JSON and XML 。

演示 fiddle here .