我用 Haskell 编写了一个程序,它必须加载并解析 UTF8 格式的大文本文件。该文件表示一个字典,每行都有键:值对。在我的程序中,我想要一个 Data.Map 容器来进行快速字典搜索。我的文件大约有 40MB,但将其加载到我的程序后,使用了 1.5 GB RAM,并且从未释放。我做错了什么?内存使用量是否符合预期?
这是我的程序的代码示例:
模块主要在哪里
import Engine
import Codec.Archive.Zip
import Data.IORef
import System.IO
import System.Directory
import qualified System.IO.UTF8 as UTF8
import qualified Data.ByteString.Lazy as B
import qualified Data.ByteString.UTF8 as BsUtf
import qualified Data.Map as Map
import Graphics.UI.Gtk
import Graphics.UI.Gtk.Glade
maybeRead :: Read a => BsUtf.ByteString -> Maybe a
maybeRead s = case reads $ BsUtf.toString s of
[(x, "")] -> Just x
_ -> Nothing
parseToEntries :: [BsUtf.ByteString] -> [(BsUtf.ByteString, Int)]
parseToEntries [] = []
parseToEntries (x:xs) = let (key, svalue) = BsUtf.break (==':') x
value = maybeRead svalue
in case value of
Just x -> [(key, x)] ++ parseToEntries xs
Nothing -> parseToEntries xs
createDict :: BsUtf.ByteString -> IO (Map.Map BsUtf.ByteString Int)
createDict str = do
let entries = parseToEntries $ BsUtf.lines str
dict = Map.fromList entries
return (dict)
main :: IO ()
main = do
currFileName <- newIORef ""
dictZipFile <- B.readFile "data.db"
extractFilesFromArchive [] $ toArchive dictZipFile
dictFile <- UTF8.readFile "dict.txt"
dict <- createDict $ BsUtf.fromString dictFile
...
searchAccent :: Map.Map BsUtf.ByteString Int -> String -> Int
searchAccent dict word = let sword = BsUtf.fromString $ map toLower word
entry = Map.lookup sword dict
in case entry of
Nothing -> -1
Just match -> 0
最佳答案
快速回答。
主要问题是 System.IO.UTF8.readFile
将文件读入 String
。
假设的瓶颈在这里:
dictFile <- UTF8.readFile "dict.txt"
dict <- createDict $ BsUtf.fromString dictFile
处理 UTF-8 文本时,最好使用 Data.Text
而不是 ByteString
。
尝试这样的事情:
import qualified Data.Text.Lazy as LT
import qualified Data.Text.Lazy.Encoding as LT
...
dictFile <- B.readFile "dict.txt"
dict <- createDict $ LT.decodeUtf8 dictFile
另一个瓶颈是解析数字:您将 ByteString
转换为 String
,然后读取
它。
最好使用Data.Text.Lazy.Read
:
import qualified Data.Text.Lazy.Read as LT
maybeRead :: LT.Text -> Maybe Int
maybeRead s = case LT.decimal s of
Left _ -> Nothing
Right i -> Just i
关于string - 为什么 Haskell 在处理字符串时分配大量内存?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/10036028/