dap cli tool and Windows compatibility

[pgo586]

I can't remember whether the answer to this question has already been given earlier, but I'm finally getting to try to experiment with the CLI (previously, I wrote nodejs scripts for retrieving snapshots/incrementals from DAP using the low level REST API). I'm trying to run a few CLI commands in a Git Bash terminal on my Windows desktop, in which I'd like to initialize (and later sync) a test Postgres database locally (I already created an empty database called 'CD2Test') but my first call to initdb produces the following: 

    dap initdb --connection-string postgresql://user:password@localhost/CD2Test --namespace can
vas --table accounts
2023-07-10 14:50:38,637 - INFO - Query started with job ID: 6e5a3dd3-9da2-4d4e-84b0-4bfec4b30f22
2023-07-10 14:50:38,638 - INFO - Query job still in status: waiting. Checking again in 5 seconds...
2023-07-10 14:50:43,823 - INFO - Query job still in status: running. Checking again in 5 seconds...
2023-07-10 14:50:49,384 - INFO - Data has been successfully retrieved:
{"id": "6e5a3dd3-9da2-4d4e-84b0-4bfec4b30f22", "status": "complete", "expires_at": "2023-07-11T19:50:38Z", "objects": [{"id": "6e5a3dd3-9da2-4d4e-84b0-4bfec4b30f22/part-00000-daee8ad4-0f47-4b51-9bb6-9fc172bd20dc-c000.json.gz"}], "schema_version": 2, "at": "2023-07-10T18:21:11Z"}
2023-07-10 14:50:51,298 - INFO - Downloading [object 1/1 - job 6e5a3dd3-9da2-4d4e-84b0-4bfec4b30f22]
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002C653100790>
Traceback (most recent call last):
File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Python310\lib\asyncio\base_events.py", line 745, in call_soon
self._check_closed()
File "C:\Python310\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002C653100790>
Traceback (most recent call last):
File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Python310\lib\asyncio\base_events.py", line 745, in call_soon
self._check_closed()
File "C:\Python310\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002C653100790>
Traceback (most recent call last):
File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Python310\lib\asyncio\base_events.py", line 745, in call_soon
self._check_closed()
File "C:\Python310\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000002C653100790>
Traceback (most recent call last):
File "C:\Python310\lib\asyncio\proactor_events.py", line 116, in __del__
self.close()
File "C:\Python310\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Python310\lib\asyncio\base_events.py", line 745, in call_soon
self._check_closed()
File "C:\Python310\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed


So the obvious question comes to mind: is this CLI compatible with Windows in the first place? Aside from that, what could the above error mean? Any tips will be appreciated (disclaimer: I'm a python and postgres newbie)

[LeventeHunyadi]

While the DAP client library is (generally) OS-independent, we don't explicitly test on Windows. It's hard to tell what could be the root cause. Event loop is how the built-in Python library asyncio schedules tasks but it doesn't seem clear from the above error message why that scheduling failed. I happen to have a Python 3.10 installation on a Windows machine, I will check out if I can reproduce the above error. Is it a one-off outcome, or can you consistently reproduce it even if you retry?

[pgo586]

Hi @LeventeHunyadi , thanks for your reply. The problem can be consistently reproduced. From a simple Google search, I could see that some users had python work-arounds for this particular type of error -that presumably avoid it on Windows-. I guess this may in turn indicate that such  work-arounds would need to be incorporated in the CLI/client library in order to achieve Windows compatibility. Looking forward to your own experimentation with this. Thanks!

[LeventeHunyadi]

Interesting. Some posts I came across are suggesting to switch to the select event policy, which is the default for UNIX-flavor OSes: 

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Typically, in Windows programming, you use the proactor pattern in the (low-level) Winsock (Windows sockets) API for asynchronous calls, which should be the default for Python asyncio on Windows. Perhaps it's receiving less attention from the Python community, and the implementation may not be as fleshed out as the UNIX version.

[LeventeHunyadi · Accepted solution]

The above patch has been applied to the mainline code base.

I can confirm this has eliminated the issue on a Windows OS with Python 3.10. Before the patch, I would get the same exception:

RuntimeError: Event loop is closed

 After the patch, I could successfully replicate both a smaller and a larger table with initdb.

The patch will likely be included in the next DAP client library release.