Java 8 IO 流陷阱

Oct 269, 26119
Oct 219, 21029

Java

问

今天下午，测试人员在测试 MySQL 备库修改存储配置时，必现如下错误：

description: “CONSUME_MSG_FAIL”

details: “CloudRuntimeException(id=44d32759dcd54f1db2bec0d748864274, message=Shell command failed:↵xfs_growfs /data, causeOfStr=java.io.IOException:

Cannot run program “/bin/bash” (in directory “/root”): error=24, Too many open files)”

最后一行报错信息是 Too many open files

答

原因

Too many open files

不同说法：

句柄数超出系统限制

程序打开的文件/socket连接数量超过系统设定值

进程在某个时刻打开了超过系统限制的文件数量以及通讯链接数

解决

ulimit -a

其中 open files (-n) 65536 表示允许单个进程最大允许打开的文件数量是 65536

ps axu | grep java

lsof -p ${PID} | wc -l

小结

从前三个步骤看出，4196 << 65536，因此不需要增大进程可以打开的文件数

同时需要进一步查看 4196 个句柄的持有者是谁？

ls -lh /proc/${PID}/fd

发现 slave.index 的句柄泄漏了，需要定位到代码中

定位

原先代码

private List<String> getAllBinLogs(Comparator<String> comparator) {
    ...
    String content = readFile(binLogIndexPath);
    return ...;
}

private String readFile(String filePath) {
    Path path = Paths.get(filePath);
    String content = null;
    try {
        content = Files.lines(path).collect(Collectors.joining());
    } catch (IOException e) {
        e.printStackTrace();
    }
    return content;
}

追溯到 Files.lines(Path path) 的源码可以看到：

public static Stream<String> lines(Path path, Charset cs) throws IOException {
    BufferedReader br = Files.newBufferedReader(path, cs);
    try {
        return br.lines().onClose(asUncheckedRunnable(br));
    } catch (Error|RuntimeException e) {
        try {
            br.close();
        } catch (IOException ex) {
            try {
                e.addSuppressed(ex);
            } catch (Throwable ignore) {}
        }
        throw e;
    }
}

现在关注该方法上的两段注释：

Read all lines from a file as a {@code Stream}. Unlike {@link readAllLines(Path, Charset) readAllLines}, this method does not read all lines into a {@code List}, but instead populates lazily as the stream is consumed.

The returned stream encapsulates a {@link Reader}. If timely disposal of file system resources is required, the try-with-resources construct should be used to ensure that the stream’s {@link Stream#close close} method is invoked after the stream operations are completed.

提到了 readAllLines(Path, Charset) 方法，看一下：

public static List<String> readAllLines(Path path, Charset cs) throws IOException{
    try (BufferedReader reader = newBufferedReader(path, cs)) {
        List<String> result = new ArrayList<>();
        for (;;) {
            String line = reader.readLine();
            if (line == null)
                break;
            result.add(line);
        }
        return result;
    }
}

可以看到，该方法清晰地使用了 try-with-resources 语法糖，保证文件的关闭

与此相反，Files.lines(Path path) 并没有保证文件的关闭

修复代码

private List<String> getAllBinLogs(Comparator<String> comparator) {
        ...
        List<String> content;
        try {
            content = Files.readAllLines(Paths.get(binLogIndexPath));
        } catch (IOException e) {
            throw new CloudRuntimeException(e);
        }
        return ...;
    }

验证

没有泄漏的句柄，Over

Java 8 IO 流 陷阱

问

答

原因

解决