multithreading - 多个wget -r同时访问一个站点?

标签 multithreading wget

任何带有选项的命令/wget?

对于多线程下载,是否需要递归同时进行网站?

最佳答案

我找到了一个不错的解决方案。

http://www.linuxquestions.org/questions/linux-networking-3/wget-multi-threaded-downloading-457375/阅读原件

wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &

copied as many times as you deem fitting to have as much processes downloading. This isn't as elegant as a properly multithreaded app, but it will get the job done with only a slight amount of over head. the key here being the "-N" switch. This means transfer the file only if it is newer than what's on the disk. This will (mostly) prevent each process from downloading the same file a different process already downloaded, but skip the file and download what some other process hasn't downloaded. It uses the time stamp as a means of doing this, hence the slight overhead.

It works great for me and saves a lot of time. Don't have too many processes as this may saturate the web site's connection and tick off the owner. Keep it around a max of 4 or so. However, the number is only limited by CPU and network bandwidth on both ends.

关于multithreading - 多个wget -r同时访问一个站点?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/4745799/

相关文章:

python - 多线程 Python 代码中的 Emacs pdb 和断点问题

multithreading - 在Perl中实现看门狗

multithreading - 使用Caliburn.Micro.ReactiveUI 1.2.2时,错误的线程访问UI对象

wget - 连接到 https 页面时 wget 速度较慢

linux - wget 强制重试直到有连接

multithreading - 什么是互斥体?

Java 并发 : Count characters of String

c - wget 命令将输入​​作为文件,并且文件必须包含 url 和字符串参数

linux - 在 Linux 中使用 wget 或其他命令获取整个域内容?

git - Docker 使用经过身份验证的 git 调用