“Too many open files” When Compiling on macOS

Problem

The following error message is output by LLVM Clang 10.0.1 when compiling from the command-line. The Clion IDE builds the project just fine, of course.

$ cmake --build build
...
In file included from /opt/local/include/boost/spirit/home/x3.hpp:19:
In file included from /opt/local/include/boost/spirit/home/x3/operator.hpp:10:
In file included from /opt/local/include/boost/spirit/home/x3/operator/sequence.hpp:12:
/opt/local/include/boost/spirit/home/x3/operator/detail/sequence.hpp:25:10: fatal error: cannot open file '/opt/local/include/boost/fusion/include/as_deque.hpp': Too many open files
#include <boost/fusion/include/as_deque.hpp>
         ^
1 error generated.

Too many open files… Really? What decade is this?!

Solution

According to Wilson Mar’s article Maximum limits (in macOS file descriptors), this issue is caused by a low default limit for the number of files that can be open simultaneously. Digging a little bit deeper into the launchctl and setrlimit man pages, its important to note this limit is specific to a single process. What follows are step-by-step instructions for detecting and resolving this issue.

First, check the existing soft and hard limits for the maximum number of open files.
launchctl limit maxfiles maxfiles 256 unlimited
In this case, the 256 file soft limit is the issue. This limit is much too low.
For the running session, remedy the problem by setting higher limits with launchctl.
sudo launchctl limit maxfiles 65536 2147483647
This sets the soft limit to 65,536 open files and the hard limit to 2,147,483,647 open files.

Don’t exceed the maximum number of 2,147,483,647 for either limit here.^[1]

To persist this setting, create a launchd.plist file which will be used to launch the command every time the machine boots.^[2]

/Library/LaunchDaemons/limit.maxfiles.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
        "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>65536</string>
      <string>2147483647</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

Increasing this limit with ulimit from a startup shell script is quick and do-able. However, such an approach only applies to a single user, doesn’t account for changing shells, and is more likely to accidentally be overwritten or deleted.

Ensure the file is owned by the root user and belongs to the wheel group.
```
sudo chown root:wheel /Library/LaunchDaemons/limit.maxfiles.plist
```
Set permissions on the file so that it is readable and writeable by the owner and only readable by group members and everyone else.
```
sudo chmod 644 /Library/LaunchDaemons/limit.maxfiles.plist
```

Create a system service from the script.^[3]^[4]

sudo launchctl bootstrap system /Library/LaunchDaemons/limit.maxfiles.plist

Set the service to run at boot.

sudo launchctl enable system/limit.maxfiles

Verify that the limits have been updated.

launchctl limit maxfiles
	maxfiles    65536          2147483647

Restart your terminal application to take advantage of the increased limits.
Check to make sure the changes have taken effect for your shell.
```
ulimit -S -n
65536
```

Problem

Solution

Conclusion