New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OSX crash complaining of operation in progress in another thread when fork() was called
#32499
Comments
This is apparently due to some new security changes made in High Sierra that are breaking lots of Python things that use |
in progress in another thread when fork() was called
@nitzmahone, the question is why it works fine when I revert to Ansible 2.3. Doesn't it mean it's a regression introduced in 2.4? |
It'd be really nice to have this fixed because I need new stuff from 2.4 but I'm stuck with 2.3 because of the bug. At least acknowledging it's an Ansible regression instead of blaming OS X would be a step in the right direction. |
It doesn't appear to be an Ansible regression- 2.3.0 fails for us the exact same way on High Sierra. Are you sure you didn't upgrade to High Sierra between 2.3.x and 2.4.1 (since it was released right around that time)? This is the best examination of the underlying issue we've found: https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/ - so far the bugs opened on Python core for this issue have been closed (eg https://bugs.python.org/issue30837). Regardless, the issue appears to be well outside the Ansible codebase, and the behavior doesn't appear to have changed in any recent releases, so I don't think there's much we can do. |
@nitzmahone Winrm has issue still $ ansible all -m ping
descartes | UNREACHABLE! => {
"changed": false,
"msg": "ssl: HTTPSConnectionPool(host='descartes', port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x108f7e510>: Failed to establish a new connection: [Errno 61] Connection refused',))",
"unreachable": true
}
schrodinger | UNREACHABLE! => {
"changed": false,
"msg": "ssl: HTTPSConnectionPool(host='schrodinger', port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x108d20290>: Failed to establish a new connection: [Errno 61] Connection refused',))",
"unreachable": true
}
[WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last): File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 276, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 256, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq)) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/protocol.py", line
207, in send_message return self.transport.send_message(message) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/transport.py", line
202, in send_message raise WinRMTransportError('http', error_message) WinRMTransportError:
(u'http', u'Bad HTTP response returned from server. Code 500')
[WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last): File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 276, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 256, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq)) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/protocol.py", line
207, in send_message return self.transport.send_message(message) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/transport.py", line
202, in send_message raise WinRMTransportError('http', error_message) WinRMTransportError:
(u'http', u'Bad HTTP response returned from server. Code 500')
[WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last): File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 276, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 256, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq)) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/protocol.py", line
207, in send_message return self.transport.send_message(message) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/transport.py", line
202, in send_message raise WinRMTransportError('http', error_message) WinRMTransportError:
(u'http', u'Bad HTTP response returned from server. Code 500')
oracle | FAILED! => {
"failed": true,
"msg": "winrm send_input failed"
}
babbage | FAILED! => {
"failed": true,
"msg": "winrm send_input failed"
}
kitchenl1 | FAILED! => {
"failed": true,
"msg": "winrm send_input failed"
}
[WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last): File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 276, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 256, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq)) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/protocol.py", line
207, in send_message return self.transport.send_message(message) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/transport.py", line
202, in send_message raise WinRMTransportError('http', error_message) WinRMTransportError:
(u'http', u'Bad HTTP response returned from server. Code 500')
standup | FAILED! => {
"failed": true,
"msg": "winrm send_input failed"
}
kitchenl2 | UNREACHABLE! => {
"changed": false,
"msg": "ssl: HTTPSConnectionPool(host='kitchenl2', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x108f21b10>, 'Connection to kitchenl2 timed out. (connect timeout=30)'))",
"unreachable": true
}
[WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last): File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 276, in _winrm_exec
self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-
packages/ansible/plugins/connection/winrm.py", line 256, in _winrm_send_input
protocol.send_message(xmltodict.unparse(rq)) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/protocol.py", line
207, in send_message return self.transport.send_message(message) File
"/usr/local/Cellar/ansible/2.4.1.0/libexec/lib/python2.7/site-packages/winrm/transport.py", line
202, in send_message raise WinRMTransportError('http', error_message) WinRMTransportError:
(u'http', u'Bad HTTP response returned from server. Code 500')
shannon | FAILED! => {
"failed": true,
"msg": "winrm send_input failed"
}
|
|
That fixed it for me. I can finally use 2.4. Thanks. 🎉 |
Further comment: this is almost certainly a bug in the python which apple distributes. Apple has changed their fork implementation to disallow forking with active threads. Taking the latest python distributed from python.org seems to fix this error without needing to set the environment variable (tested on python 3.6). |
Add OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES to the environment to workaround an issue with running Ansible on MacOS 10.13 (High Sierra). See ansible/ansible#32499
For anybody else who comes across this running Ansible from Vagrant, you can work around this by adding this hack to your
|
I'm admittedly way out of my depth here, but AFAICT the above isn't as reassuring as it seems. Discussion in https://bugs.python.org/issue33725 suggests that
If that's true, maybe this should be solvable in the Ansible codebase after all? FWIW, I do see the error using a shiny new Python 3.7.0 from homebrew (unless I apply the environment variable workaround). |
Trying to remember what this bug is about I came across this article which gives a good explanation. Essentially, to safely support threaded applications we should eliminate all calls to fork except a) when immediately followed by an exec() b) at the very start of a program when creating a special worker which can then do subsequent forks but which never runs threads. Unfortunately "threaded applications" means all applications because there are now many libraries which privately embed threading. |
* Verified working on OpenShift 3.10 cluster * Added workaround for ansible/ansible#32499 * Tweaked Bitucket resource usage
I upgraded my system to Mojave
I am getting this error which is not resolved by 2.3.3 :002 > User.first
objc[56844]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[56844]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
exited rails console |
The Did you get it to work @Sohair63? |
@swaathi please could you open a new bug report with full information - especially all software versions (MacOS,Python,Ansible etc), the exact playbook you are using that triggers the crash and exactly how you run it. I'll upgrade soon and then attempt to debug if I can reproduce. Put a link to your new bug from #49207 N.B. you have to set the environment variable in the shell before you launch the ansible process - it's not enough to set it in the ansible process. That has occasionally been a problem with debugging this situation. |
@swaathi Yes it is working fine for me now, but not sure how it got fixed. |
since I upgraded my system to Mojave |
ISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
CONFIGURATION
I have system variables set in
.bash_profile
:OS / ENVIRONMENT
Mac OS X High Sierra with latest updates. Ansible is installed using
brew
.SUMMARY
Can't do an etcd lookup since upgrading to 2.4. Worked fine before 2.4.
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
Ansible just hangs after outputting that error message.
The text was updated successfully, but these errors were encountered: