linux - 如何以最快的速度复制文件？

标签 linux bash unix ubuntu scp

我正在 machineA 上运行我的 shell 脚本，它将文件从 machineB 和 machineC 复制到 machineA。

如果文件不在machineB 中，那么它肯定在machineC 中。所以我将首先尝试从 machineB 复制，如果它不在 machineB 中，那么我将转到 machineC 复制相同的文件。

在 machineB 和 machineC 中会有一个这样的文件夹 YYYYMMDD inside this folder - 

/data/pe_t1_snapshot

因此无论日期是上述文件夹中此格式 YYYYMMDD 的最新日期 - 我将选择该文件夹作为我需要开始复制文件的完整路径 -

假设这是 /data/pe_t1_snapshot 中的最新日期文件夹 20140317 那么这将是我的完整路径 - 

/data/pe_t1_snapshot/20140317

我需要从那里开始复制 machineB 和 machineC 中的文件。我需要从 machineB 和 machineC 复制 400 文件到 machineA 并且每个文件大小是 1.5 GB.

目前我有下面的 shell 脚本，它工作正常，因为我正在使用 scp 但不知何故它需要 ~2 小时 来复制 400我猜 machineA 中的文件对我来说太长了。 :(

下面是我的shell脚本-

#!/bin/bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200
SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    # delete all the files first
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
    done

    # delete all the files first
    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.
    done
fi

我正在复制 PRIMARY 文件夹中的 PRIMARY_PARTITION 文件和 machineA 中 SECONDARY 文件夹中的 SECONDARY_PARTITION 文件。

有什么方法可以在 machineA 中更快地移动文件。我可以一次复制 10 个文件还是并行复制 5 个文件以加快此过程或任何其他方法？

注意:machineA 在 SSD 上运行

更新:-

我试过的并行 Shell 脚本，shell 脚本的顶部与上面显示的相同。

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    find "$PRIMARY" -mindepth 1 -delete
    for el in "${PRIMARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.) &
          WAITPID="$WAITPID $!"        
    done

    find "$SECONDARY" -mindepth 1 -delete
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.) &
          WAITPID="$WAITPID $!"        
    done
     wait $WAITPID
     echo "All files done copying."
fi

我在使用并行 shell 脚本时遇到的错误- 

channel 24: open failed: administratively prohibited: open failed
channel 25: open failed: administratively prohibited: open failed
channel 26: open failed: administratively prohibited: open failed
channel 28: open failed: administratively prohibited: open failed
channel 30: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 32: open failed: administratively prohibited: open failed
channel 36: open failed: administratively prohibited: open failed
channel 37: open failed: administratively prohibited: open failed
channel 38: open failed: administratively prohibited: open failed
channel 40: open failed: administratively prohibited: open failed
channel 46: open failed: administratively prohibited: open failed
channel 47: open failed: administratively prohibited: open failed
channel 49: open failed: administratively prohibited: open failed
channel 52: open failed: administratively prohibited: open failed
channel 54: open failed: administratively prohibited: open failed
channel 55: open failed: administratively prohibited: open failed
channel 56: open failed: administratively prohibited: open failed
channel 57: open failed: administratively prohibited: open failed
channel 59: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 61: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
mux_client_request_session: session request failed: Session open refused by peer
channel 64: open failed: administratively prohibited: open failed
mux_client_request_session: session request failed: Session open refused by peer
channel 68: open failed: administratively prohibited: open failed
channel 72: open failed: administratively prohibited: open failed
channel 74: open failed: administratively prohibited: open failed
channel 76: open failed: administratively prohibited: open failed
channel 78: open failed: administratively prohibited: open failed

最佳答案

你可以试试这个命令

rsync

来自

man rsync

您会看到:rsync 远程更新协议(protocol)允许 rsync 通过网络连接仅传输两组文件之间的差异，使用此包随附的技术报告中描述的高效校验和搜索算法。

关于linux - 如何以最快的速度复制文件？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/22903743/

上一篇：linux - 我怎样才能知道某个特定的软件包是否已经安装在 Arch Linux 中

下一篇：清除串口缓冲区

相关文章：

c# - MonoDevelop + NAudio + Ubuntu Linux 告诉我找不到 Winmm.dll？

Java native 接口(interface) GetStringUTFChars fatal error

linux - 我是否需要为并发下载设置下载速度限制(使用 cURL)，或者它们将平分？

linux - 像 'core local storage' 这样的东西可能吗？

bash - 用不同文件中的列替换文件中的列，同时保留格式

linux - 下面的 awk 有什么问题

c++ - 将程序 A 的输出重定向到程序 B 的输入，反之亦然

linux - 如何在 Linux 上获得总体 CPU 使用率(例如 57%)

linux - 文件列表

wordpress - 上传的文件无法移动到 wp-content/uploads

©2024 IT工具网  联系我们