unix - 如何根据多个列值拆分文件

Question

我需要获取这个 test_file 并将其拆分为 col5 和 col6 的每个独特组合都有一个单独的文件。 另一个警告是该文件需要在 150,000 条记录后进行拆分。此外，还需要从文件中提取命名约定: "$5"_"$6"_P"sysdate"_IU"$4"60""[文件号]".zip

测试文件.csv

col1, col2, col3, col4, col5, col6 ..... col32
1234, 6789, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1235, 1233, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1236, 4423, 1, 01/31/2017 00:00:00, 1000, 5678 ..... col32
1237, 3323, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1238, 0808, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1239, 2222, 1, 01/31/2017 00:00:00, 2000, 1234 ..... col32
1231, 4535, 1, 01/31/2017 00:00:00, 2000, 1234 ..... col32
1232, 8080, 1, 01/31/2017 00:00:00, 2000, 5678 ..... col32
1233, 7878, 1, 01/31/2017 00:00:00, 2000, 5678 ..... col32

结果应如下所示:

1000_1234_P20170203_IU20170131_60_1.ZIP
col1, col2, col3, col4, col5, col6 ..... col32
1234, 6789, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1235, 1233, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1237, 3323, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32
1238, 0808, 1, 01/31/2017 00:00:00, 1000, 1234 ..... col32

1000_5678_P20170203_IU20170131_60_1.ZIP
col1, col2, col3, col4, col5, col6 ..... col32
1236, 4423, 1, 01/31/2017 00:00:00, 1000, 5678 ..... col32

2000_1234_P20170203_IU20170131_60_1.ZIP
col1, col2, col3, col4, col5, col6 ..... col32
1239, 2222, 1, 01/31/2017 00:00:00, 2000, 1234 ..... col32
1231, 4535, 1, 01/31/2017 00:00:00, 2000, 1234 ..... col32

2000_5678_P20170203_IU20170131_60_1.ZIP
col1, col2, col3, col4, col5, col6 ..... col32
1232, 8080, 1, 01/31/2017 00:00:00, 2000, 5678 ..... col32
1233, 7878, 1, 01/31/2017 00:00:00, 2000, 5678 ..... col32

Answer 1

从这里开始:

awk -F', *' -v sysdate="$(date +'%Y%m%d')" '
NR==1 { hdr = $0; next }
(cnt[$5,$6]++ % 150000) == 0 { sfx[$5,$6]++ }
{
    split($4,d,/[\/ ]/)
    out = $5 "_" $6 "_P" sysdate "_IU" d[3] d[1] d[2] "_60_" sfx[$5,$6] ".zip"
    if (!seen[out]++) {
        print hdr > out
    }
    print > out
}
' file

并进行适合的按摩。如果您不使用 GNU awk，则可能需要 close() 文件，以避免出现“打开的文件太多”错误。