CD2 DAP Client Issues when running via CRON

[ismail_orabi]

I created a shell script that uses the DAP client to run the CD2 sync commands. The script works fine when run manually, but when I try to schedule it with CRON it fails with the stacktrace below.

DAP Client Version: 0.3.10

I simplified the script to run a single "dap list" command and I still receive the error

CRONOUTPUT: 2023-08-15 15:31:03,679 - ERROR - malformed HTTP response:
CRONOUTPUT: Forbidden; invalid authentication data.
CRONOUTPUT: 2023-08-15 15:31:03,680 - ERROR - malformed HTTP response
CRONOUTPUT: Traceback (most recent call last):
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/__main__.py", line 138, in console_entry
CRONOUTPUT:     main()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/__main__.py", line 130, in main
CRONOUTPUT:     asyncio.run(dapCommand.execute(args))
CRONOUTPUT:   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
CRONOUTPUT:     return loop.run_until_complete(main)
CRONOUTPUT:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
CRONOUTPUT:     return future.result()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/commands.py", line 31, in execute
CRONOUTPUT:     executed = await super().execute(args)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/base.py", line 49, in execute
CRONOUTPUT:     if await subcommand.execute(args):
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/base.py", line 45, in execute
CRONOUTPUT:     await self._execute_impl(args)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/commands.py", line 153, in _execute_impl
CRONOUTPUT:     tables = await session.get_tables(args.namespace)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 363, in get_tables
CRONOUTPUT:     table_list = await self._get(f"/dap/query/{namespace}/table", TableList)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 193, in _get
CRONOUTPUT:     await self.authenticate()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 328, in authenticate
CRONOUTPUT:     properties = await self._post_auth_request(self._credentials.basic_credentials)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 278, in _post_auth_request
CRONOUTPUT:     return await self._process(response, TokenProperties, suppress_output=True)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 298, in _process
CRONOUTPUT:     raise DAPClientError("malformed HTTP response")
CRONOUTPUT: dap.api.DAPClientError: malformed HTTP response

[IanGoh]

It looks like the root issue is authentication.  So you might want to concentrate on checking that works.

[reynlds]

Client version 3.1.0 is pretty old...an upgrade may be in order. I'm running 0.3.9.1.

Also, do you have your credentials set in your user profile? If not, you'll have to pass them as part of the script. I use cron also and have exported these items (~/.bash_profile) to support the sync process:

declare -x DAP_API_URL="https://api-gateway.instructure.com/"
declare -x DAP_CLIENT_ID="XXXXX"
declare -x DAP_CLIENT_SECRET="XXXXX"
declare -x DAP_CONNECTION_STRING="postgresql://postgres_username:postgres_password@postgres_db_hostname:5432/postgres_db"

[ThayerMerritt]

Hello, 

While I'm not sure that I'm receiving the same error exactly, I've run into similar issues as well. Echoing reynlds I presume this is related to cron running as a system-level user rather than your local user. This would prevent environment variables from being passed correctly, assuming that you're using those. Your interactive shell/terminal would pick these up. Since DAP doesn't have access to your env variables, which contain your credentials without passing the user information, a forbidden: invalid authentication credentials would be expected. I want to say that cron/crontab can have the username specified in the command. I hope this steers you in a productive direction!

Good luck!

[morris_gart]

Did you every resolve this?  I am wanting to do the same thing but with no luck.

[ThayerMerritt]

Hi @morris_gart 

I solved the problem by running DAP in a docker container (named CD2Update for the purposes of the script below). 
Added the configuration variables to the container configuration and executed the container as a bash script using cron containing the command:

#!/bin/bash
docker start CD2Update
docker exec CD2Update /scripts/sync.sh
docker stop CD2Update

the sync script contains values to make sure Python, DAP, and all dependencies needed to run are installed & up-to-date, and then runs the sync job for all tables. Once complete the container shuts down. 

It's not elegant and does require docker knowledge and the ability to create and run containers, but it worked for me. I execute every 6 hours.