IOException: error=24, Too many open files

Problem

We experienced a couple of jobs that failed today. Looking at the log, it gave us the message of Too Many Open Files.

Error Message

java.io.IOException: Cannot run program "tail": java.io.IOException: error=24, Too many open files 
...
[anonymous] ERROR (DasExceptionFilter.java:43) - Unhandled exception handling /admin/application-log 
...
java.lang.RuntimeException: exception on executing command '[tail, -n, 1000, /opt/datameer/<path>/logs/conductor.log]'

Troubleshooting

If you are interested in knowing what files are currently open you can use the following command.

lsof -p <pid of jvm>

You can also see what the ulimit for open file is using:

ulimit -n

Background

This error is coming from the operating system because of not having enough file descriptors.You might need to increase the max number of open files - global level or per-user limits (or both). 

Solution

To raise the global limit for file descriptors you need to edit, depending on your operating system, the sysctl fs.file-max.

Setting file handles

To do this on a user level you need to edit the ulimit.

Setting shell limits

Usually we recommend to set the values mentioned within in given knowledge base articles.