CD2 dap issues

[RichardCarballo]

Hey all, CD2 has been a big pain implementing for my organization, once we finally got the DAP tool working now we have v1.1 out with this mandatory namespace issues now and a mountain of new errors.
With this new 1.1 dap tool, has anyone received and error like:

AssertionError: Data should not be empty
ERROR:asyncio:Exception in callback _SelectorSocketTransport._write_send()
handle: <Handle _SelectorSocketTransport._write_send()>
Traceback (most recent call last):
File "C:\Users\.......\AppData\Local\Programs\Python\Python312\Lib\asyncio\events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "C:\Users\........\AppData\Local\Programs\Python\Python312\Lib\asyncio\selector_events.py", line 1137, in _write_send
assert self._buffer, 'Data should not be empty'

I am getting this quite often and looking for a fix.

[sgergely]

Hello Richard,

I'm sorry to hear about the challenges you've faced with implementing CD2 for your organization. As the new product manager overseeing the CD2 API and CLI client, I'm here to help resolve any issues you encounter on the short and long term.

After reviewing the error message you provided, it appears to be related to a Python issue. To assist you further, could you please provide additional details about your environment?

1. Could you confirm the version of Python you're using? (I noticed the path indicates Python 3.12, is that correct?)
2. Have there been any recent changes to your Python setup? Was the DAP tool functioning properly for you before?
3. Can you share the specific DAP command you're attempting to run when you encounter this error?

Your insights will help us diagnose and address the issue effectively. Please feel free to provide any additional information you think may be relevant. We're here to support you in resolving this issue.

Looking forward to your response!

[RichardCarballo]

Hello! sorry for late response.
1. Python 3.12.3 (tags/v3.12.3:f6650f9, Apr 9 2024, 14:05:25) [MSC v.1938 64 bit (AMD64)] on win32

2. the old dap tool was working gloriously, had to migrate to new server since server version was outdated and was forced to install the new DAP tool. so this was a fresh server fresh install.

3. dap initdb --table submissions --namespace canvas (using dap tool 1.1)

3.1 why is this namespace option mandatory now? 

[RichardCarballo]

Just fyi, if i let the error continue for hours it will load 1.875 million rows into submissions but syncdb will error out saying it was never initiated and to run initdb first and then i will have to delete the table and repeat this nightmare every time because it will violate PK restraints without dumping the table first

So the question is, why is dap triggering a python error:
assert self._buffer, 'Data should not be empty'

[RichardCarballo · Accepted solution]

This is not a permanent fix but a bypass until resolved, rolling back to python 3.11.X resolved the issue. Seems to be compatibility issue between DAP 1.1 and python 3.12.x

[sgergely]

Hi Richard,

Thanks for posting a workaround!

I'm going to look into this Python 3.12.x version issue as it come up for others as well as I can see.

[Brian2345]

I can confirm that this is still and issue between DAP 1.2 and python 3.12.x

I rolled back to python 3.11.9 and was able to get it to work.

It seemed to be an issue on some tables but not others. Without doing a row count and confirming, it seems to be an issue with larger tables. Smaller (sub 10k-ish records) seem to go ok.

edit: Just saw DAP 1.3 released. Will give that a try when time allows.

[sgergely]

Hey @Brian2345, we are aware of this issue and we are trying to find a solution for but it is quite hard for us to replicate. If you can provide details how to replicate it with a more like 100% success rate compared to our 1% we would be happy to collect the info from you!

The workaround we have found for this issue is that you run DAP CLI from under WSL because it is a linux underneath. The issue is likely related to how the Python standard library interacts with I/O completion ports in WinSock—code that resides outside of our direct control.

I have added this as a known issue to our release notes and my DAP CLI 1.3.0 release announcement blogpost as well.
Please let me know if this workaround works for you! 

[Brian2345]

I don't have all the logs anymore since rolling back the version of python, but here's what I do have if it helps at all. A snippet of the error and the info section from the log.

It consistently occurred for large tables such as assessment_questions. All while calling initdb, not syncdb, since it during our initial setup.

2024-11-15 10:50:59,415 - INFO - Python version: 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
2024-11-15 10:50:59,415 - INFO - Platform: uname_result(system='Windows', node='nodenamehere', release='10', version='10.0.19045', machine='AMD64')
2024-11-15 10:51:00,274 - INFO - Dependency versions: {'aiofiles': '24.1.0', 'aiohttp': '3.10.10', 'aiohttp-retry': '2.8.3', 'json_strong_typing': '0.3.4', 'PyJWT': '2.9.0', 'pysqlsync': '0.7.1', 'tsv2py': '0.7.1', 'types-aiofiles': '24.1.0.20240626'}
2024-11-15 10:51:00,519 - INFO - Database version: PostgreSQL 16.4 (Ubuntu 16.4-0ubuntu0.24.04.2) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0, 64-bit