performance - 如何处理具有恒定内存的两条长线？

Question

输入文件由两行组成，每行包含许多数字

1 2 3 4... 
5 6 7 8... 

我想处理每一行的数据，如下所示:

doSomething :: [Int] -> [Int] -> Int
doSomething [] y = 0 -- stop execution. I'm concerted in memory behavior only.
doSomething (x:xs) y = doSomething xs y
main = do
    inputdata <- getContents
    let (x:xs) = lines inputdata
        firstLine = map (read ::String->Int) $ words $ x
        secondLine = map (read ::String->Int) $ words $ head xs
    print $ doSomething firstLine secondLine

当我运行这个程序时，堆分析显示如下:


如果我不使用 secondLine ( xs )，那么这个程序会以恒定内存运行。
列表中的每个条目firstLine被 GC 处理然后丢弃。

为什么消耗的内存这么大？
我看到分配的内存量约为 100MB，但实际输入数据大小为 5MB。

是否 head xs强制将整个第一行读入内存，
甚至 secondLine根本不用？这是主要原因吗
堆分析图的内存增加？

如何处理具有恒定内存的两条线，例如后面的​​执行图？

如果 3) 的答案取决于处理顺序，我怎样才能先处理第二行，然后再处理第一行？

Answer 1

Q1) Why the consumed memory is so large? I see the amount of allocated memory is ~100MB, but actual input data size is 5MB.

String在 Haskell 中是 [Char] 的类型别名，因此在实际字节中，编译器还必须为内存中的每个字符保留一个构造函数、一个指向装箱字符的指针和一个指向下一个构造函数的指针，结果比 C 风格的字符串占用了 10 倍以上的内存。更糟糕的是，文本多次存储在内存中。

Q3) How can I process two lines with constant memory, like latter execution graph?
Q4) If the answer to Q3) depends on processing order, how can I process second line first, and then first line?

不，你不能从 lines 开始处理第二行。函数必须评估第一行中的每个字节以命中 '\n' 换行符。

Q2) Does head xs force reading entire first line into memory, even secondLine is not used at all? And is this the main cause of memory increase of heap profiling graph?

这不是 head这可以防止第一行被 GCed。如果调整 doSomething 的类型签名，仍然会发生空间泄漏。并通过 xs直接到它。关键是(未优化的)编译器不会知道 secondLine在 doSomething 之前未使用终于命中了第一个模式，所以程序保留了对 xs 的引用.顺便说一句，如果使用 -O2 编译，您的程序将使用常量内存运行。

导致你程序空间泄漏的原因主要是这行:

let (x:xs) = lines inputdata

当 x或 xs被丢弃，这个 let 子句将内联在转储的核心中。只有当它们都被稍后引用时，Core 才会出现奇怪的行为:它构造一个元组，通过模式匹配来破坏它，然后再次将这两个部分构造成一个元组，因此保持对 secondLine 的引用。该程序实际上保留了对元组 (x, xs) 的引用所以第一行永远不会被GCed。

带有 secondLine 的核心注释掉:

Rec {
doSomething_rjH
doSomething_rjH =
  \ ds_d1lv y_alG ->
    case ds_d1lv of _ {
      [] -> I# 0;
      : x_alH xs_alI -> doSomething_rjH xs_alI y_alG
    }
end Rec }

main
main =
  >>=
    $fMonadIO
    getContents
    (\ inputdata_app ->
       print
         $fShowInt
         (doSomething_rjH
            (map
               (read $fReadInt)
               (words
                  (case lines inputdata_app of _ {
                     [] -> case irrefutPatError "a.hs:6:9-32|(x : xs)"# of wild1_00 { };
                     : x_auf xs_aug -> x_auf
                   })))
            ([])))

main
main = runMainIO main

有空间泄漏的核心:

Rec {
doSomething_rjH
doSomething_rjH =
  \ ds_d1ol y_alG ->
    case ds_d1ol of _ {
      [] -> I# 0;
      : x_alH xs_alI -> doSomething_rjH xs_alI y_alG
    }
end Rec }

main
main =
  >>=
    $fMonadIO
    getContents
    (\ inputdata_app ->
       -- *** Construct ***
       let {
         ds_d1op
         ds_d1op =
           case lines inputdata_app of _ {
             [] -> irrefutPatError "a.hs:6:9-30|x : xs"#;
             : x_awM xs_awN -> (x_awM, xs_awN)
           } } in
       -- *** Destruct ***
       let {
         xs_awN
         xs_awN = case ds_d1op of _ { (x_awM, xs1_XwZ) -> xs1_XwZ } } in
       let {
         x_awM
         x_awM = case ds_d1op of _ { (x1_XwZ, xs1_XwU) -> x1_XwZ } } in
       -- *** Construct ***
       let {
         ds1_d1oq
         ds1_d1oq = (x_awM, xs_awN) } in
       print
         $fShowInt
         -- *** Destruct ***
         (doSomething_rjH
            (map
               (read $fReadInt)
               (words (case ds1_d1oq of _ { (x1_Xx1, xs1_Xx3) -> x1_Xx1 })))
            (map
               (read $fReadInt)
               (words
                  (head (case ds1_d1oq of _ { (x1_Xx1, xs1_Xx3) -> xs1_Xx3 }))))))

main
main = runMainIO main

替换 let case .. of 的子句子句将修复空间泄漏:

doSomething :: [Int] -> [Int] -> Int
doSomething [] _ = 0 -- stop execution. I'm concerted in memory behavior only.
doSomething (_:xs) y = doSomething xs y

main :: IO ()
main = do
  inputdata <- getContents
  case lines inputdata of
    x:xs -> do
      let
        firstLine = map (read ::String->Int) $ words x
        secondLine = map (read ::String->Int) $ words $ head xs
      print $ doSomething firstLine secondLine

被倾倒的核心。这次没有发生“先建后毁”的模式:

Rec {
doSomething_rjG
doSomething_rjG =
  \ ds_d1o6 ds1_d1o7 ->
    case ds_d1o6 of _ {
      [] -> I# 0;
      : ds2_d1o8 xs_alG -> doSomething_rjG xs_alG ds1_d1o7
    }
end Rec }

main
main =
  >>=
    $fMonadIO
    getContents
    (\ inputdata_apn ->
       case lines inputdata_apn of _ {
         [] -> patError "a.hs:(8,3)-(13,46)|case"#;
         : x_asI xs_asJ ->
           print
             $fShowInt
             (doSomething_rjG
                (map (read $fReadInt) (words x_asI))
                (map (read $fReadInt) (words (head xs_asJ))))
       })

main
main = runMainIO main