DAP 1.0.0 syncdb error in PostgreSQL - table not initialized, use "initdb"

[bruno_guimaraes]

I used the initdb command to create a table. The table is created successfully with some data. Then when I try to use the syncdb command the following error appears:

INFO:pysqlsync.postgres:PostgreSQL version 16.0.2 final
2024-03-06 15:48:52,993 - ERROR - table not initialized, use `initdb`

Has anyone seen this error and know how to resolve it?

[RobertSznyida]

Hi @bruno_guimaraes,

there is an ongoing discussion here with StevenWalls (he brought it up in the comments section), he has an issue that sounds similar to yours. It is specific to the environment (operating system and python version).

In short, `syncdb` doesn't work if there was some failure during `initdb`. So it would be helpful to see the output of the `initdb` command that you previously executed. And knowing the operating system and Python version would be also helpful.

If you had the same error during `initdb` as what StevenWalls had ("Fatal Python error: none_dealloc: deallocating None"), AND you are using Python version lower than 3.12 on Windows, then my suggestion is upgrading your Python version to 3.12, if that is an option for you, while we analyze the issue further.

It is likely that your database is an inconsistent state now, so before retrying the `initdb` again, you need to drop the one specific table that is being initialized.

Let us know if this was helpful or you need further assistance.

[michael60]

Hi Bruno,

I am also having this issue with Python 3.12 and Postres 16.02 on Windows Server 2016.

-M

[LeventeHunyadi]

I recommend that you upgrade to the latest version of DAP client library (version 1.1.0 at the time of writing), which includes a patch for the original root cause, see the release notes. Feel free to open a support ticket if the issue persists after the upgrade, and make sure to include any debug logs. You may need to manually delete the database tables that got into an inconsistent state if you had a failed initdb, as suggested in the response by Róbert.

[michael60]

Thanks. I installed 1.1.0 and removed DAP schemas from Postgres and tried initdb again on accounts. It states success but no data in canvas.accounts table:

2024-03-25 14:02:15,028 - INFO - Data has been successfully retrieved:
{"id": "6debc9ad-e2ce-4550-9109-e0b3cd34565f", "status": "complete", "expires_at": "2024-03-26T17:59:08Z", "objects": [{"id": "6debc9ad-e2ce-4550-9109-e0b3cd34565f/part-00000-55d0211c-293b-461c-8c18-e113049a2fe3-c000.tsv.gz"}], "schema_version": 2, "at": "2024-03-25T16:42:05Z"}

Maybe I need Windows 11:-(??

[LeventeHunyadi]

Can you share a full debug log? (Alternatively, feel free to open a support ticket.) The message Data has been successfully retrieved means that the result-set has been packaged by DAP API, and has been downloaded by the client library. Unfortunately, it tells nothing about whether that downloaded output data has been successfully inserted into the database, which is what you are having the issue with.

DAP client library is largely OS-independent; we don't have much OS-specific code. A few notable differences are the tab-separated values (TSV) parser, and the database driver, which are C libraries wrapped in Python (so-called extensions). However, they don't depend on Windows version. When compiling these libraries, the compilation target is typically an older OS release such that it runs on a wide variety of systems.

[michael60]

Attached is my dap log ater running this:
>dap --loglevel debug --logfile c:\temp\dap.log initdb --connection-string postgresql://xxxxxxxxxx@localhost:5432/postgres --namespace canvas --table accounts

[LeventeHunyadi]

Unfortunately, this log seems incomplete or the process terminated prematurely. The relevant lines in the debug log are as follows:

2024-03-25 15:25:30,208 - DEBUG - init: insert data from resources saved to disk into database table (sql_op_init.py:87)
2024-03-25 15:25:30,212 - DEBUG - processing file 1 of 1 (sql_op_init.py:100)

It seems as if inserting the data into the database is starting but then the process ends abruptly (assuming there are no more log records). You might want to open a support ticket such that the team can investigate in more detail.

[RobertSznyida]

@michael60 I'm thinking there might be more detailed log coming to the console when running the client, compared to what gets logged to the log file. The "Fatal Python error" that I mentioned in my earlier reply gets printed to the console but doesn't get to the log file, because the application terminates immediately when the fatal error occurs. You might have a similar situation.

So, if you could also send the console output that you get when running the client, that might help to understand what's going on. You could copy the console output to a text file and attach it here as well.

[michael60]

Thanks Robert. I'm attaching the console output from a new Windows 11, Python 3.12, Postgres 16.2 machine we set up for this purpose hoping the issue was OS (Windows 10). 

Everything seems to execute flawlessly right up until the Database insert (copy) of the tsv? 

[michael60]

Also - I can manually open and see the contents of the tsv file which looks ok to me.

[RobertSznyida]

@michael60 Thank you, I checked your console output, I would've expected some kind of error message that could help but unfortunately there is none.

What you could try is upgrading the `tsv2py` dependency of the client to the latest version (0.6.2), which has a new check detecting the presence of AVX2 instruction set on the machine, and then check the console output again.

pip install --upgrade tsv2py

Building tsv2py from source might also solve the issue, Levente gives more details in his comments ubnder this post: https://community.canvaslms.com/t5/Data-and-Analytics-Group/DAP-CLI-Tool-Imports-Structure-but-No-Da...

[NoorRashid]

Yes, I got this too!

Update to the new stable release and reinitialise the db, should work!