Java TCP 套接字 : data transfer is slow

Question

我用 ServerSocket 设置了一个服务器，用客户端机器连接到它。它们通过交换机直接联网，ping 时间<1ms。

现在，我尝试通过套接字的输出流将“大量”数据从客户端推送到服务器。传输 0.6Gb 需要 23 分钟。我可以通过 scp 在几秒钟内推送一个更大的文件。

知道我可能做错了什么吗？我基本上只是循环并在套接字上调用 writeInt。速度问题与数据的来源无关，即使我只是发送一个常量整数而不是从磁盘读取也是如此。

我尝试将两侧的发送和接收缓冲区设置为 4Mb，没有骰子。我为读取器和写入器使用缓冲流，没有骰子。

我错过了什么吗？

编辑:代码

这里是我做 socket 的地方

System.out.println("Connecting to " + hostname);

    serverAddr = InetAddress.getByName(hostname);

    // connect and wait for port assignment
    Socket initialSock = new Socket();
    initialSock.connect(new InetSocketAddress(serverAddr, LDAMaster.LDA_MASTER_PORT));
    int newPort = LDAHelper.readConnectionForwardPacket(new DataInputStream(initialSock.getInputStream()));
    initialSock.close();
    initialSock = null;

    System.out.println("Forwarded to " + newPort);

    // got my new port, connect to it
    sock = new Socket();
    sock.setReceiveBufferSize(RECEIVE_BUFFER_SIZE);
    sock.setSendBufferSize(SEND_BUFFER_SIZE);
    sock.connect(new InetSocketAddress(serverAddr, newPort));

    System.out.println("Connected to " + hostname + ":" + newPort + " with buffers snd=" + sock.getSendBufferSize() + " rcv=" + sock.getReceiveBufferSize());

    // get the MD5s
    try {
        byte[] dataMd5 = LDAHelper.md5File(dataFile),
               indexMd5 = LDAHelper.md5File(indexFile);

        long freeSpace = 90210; // ** TODO: actually set this **

        output = new DataOutputStream(new BufferedOutputStream(sock.getOutputStream()));
        input  = new DataInputStream(new BufferedInputStream(sock.getInputStream()));

这是我进行服务器端连接的地方:

    ServerSocket servSock = new ServerSocket();
    servSock.setSoTimeout(SO_TIMEOUT);
    servSock.setReuseAddress(true);
    servSock.bind(new InetSocketAddress(LDA_MASTER_PORT));

    int currPort = LDA_START_PORT;

    while (true) {
        try {
            Socket conn = servSock.accept();
            System.out.println("Got a connection.  Sending them to port " + currPort);
            clients.add(new MasterClientCommunicator(this, currPort));
            clients.get(clients.size()-1).start();

            Thread.sleep(500);

            LDAHelper.sendConnectionForwardPacket(new DataOutputStream(conn.getOutputStream()), currPort);

            currPort++;
        } catch (SocketTimeoutException e) {
            System.out.println("Done listening.  Dispatching instructions.");
            break;
        }
        catch (IOException e) {
            e.printStackTrace();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

好的，这就是我要传送超过 ~0.6Gb 数据的地方。

public static void sendTermDeltaPacket(DataOutputStream out, TIntIntHashMap[] termDelta) throws IOException {
    long bytesTransferred = 0, numZeros = 0;

    long start = System.currentTimeMillis();

    out.write(PACKET_TERM_DELTA); // header     
    out.flush();
    for (int z=0; z < termDelta.length; z++) {
        out.writeInt(termDelta[z].size()); // # of elements for each term
        bytesTransferred += 4;
    }

    for (int z=0; z < termDelta.length; z++) {
        for (int i=0; i < termDelta[z].size(); i++) {
            out.writeInt(1);
            out.writeInt(1);
        }
    }

到目前为止看起来很简单......

Answer 1

当您传输大量数据时，您不想写入单个字节。

import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;

public class Transfer {

    public static void main(String[] args) {
        final String largeFile = "/home/dr/test.dat"; // REPLACE
        final int BUFFER_SIZE = 65536;
        new Thread(new Runnable() {
            public void run() {
                try {
                    ServerSocket serverSocket = new ServerSocket(12345);
                    Socket clientSocket = serverSocket.accept();
                    long startTime = System.currentTimeMillis();
                    byte[] buffer = new byte[BUFFER_SIZE];
                    int read;
                    int totalRead = 0;
                    InputStream clientInputStream = clientSocket.getInputStream();
                    while ((read = clientInputStream.read(buffer)) != -1) {
                        totalRead += read;
                    }
                    long endTime = System.currentTimeMillis();
                    System.out.println(totalRead + " bytes read in " + (endTime - startTime) + " ms.");
                } catch (IOException e) {
                }
            }
        }).start();
        new Thread(new Runnable() {
            public void run() {
                try {
                    Thread.sleep(1000);
                    Socket socket = new Socket("localhost", 12345);
                    FileInputStream fileInputStream = new FileInputStream(largeFile);
                    OutputStream socketOutputStream = socket.getOutputStream();
                    long startTime = System.currentTimeMillis();
                    byte[] buffer = new byte[BUFFER_SIZE];
                    int read;
                    int readTotal = 0;
                    while ((read = fileInputStream.read(buffer)) != -1) {
                        socketOutputStream.write(buffer, 0, read);
                        readTotal += read;
                    }
                    socketOutputStream.close();
                    fileInputStream.close();
                    socket.close();
                    long endTime = System.currentTimeMillis();
                    System.out.println(readTotal + " bytes written in " + (endTime - startTime) + " ms.");
                } catch (Exception e) {
                }
            }
        }).start();
    }
}

这会在我的机器上用 19 秒的时间复制 1 GiB 的数据。这里的关键是使用 InputStream.read和 OutputStream.write接受字节数组作为参数的方法。缓冲区的大小并不是很重要，它应该比 5 大一点。用上面的 BUFFER_SIZE 进行实验，看看它如何影响速度，但也要记住，它可能因您运行的每台机器而异这个程序上。 64 KiB 似乎是一个很好的折衷方案。