CD2 DAP Client Issues when running via CRON

[ismail_orabi]

I created a shell script that uses the DAP client to run the CD2 sync commands. The script works fine when run manually, but when I try to schedule it with CRON it fails with the stacktrace below.

DAP Client Version: 0.3.10

I simplified the script to run a single "dap list" command and I still receive the error

CRONOUTPUT: 2023-08-15 15:31:03,679 - ERROR - malformed HTTP response:
CRONOUTPUT: Forbidden; invalid authentication data.
CRONOUTPUT: 2023-08-15 15:31:03,680 - ERROR - malformed HTTP response
CRONOUTPUT: Traceback (most recent call last):
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/__main__.py", line 138, in console_entry
CRONOUTPUT:     main()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/__main__.py", line 130, in main
CRONOUTPUT:     asyncio.run(dapCommand.execute(args))
CRONOUTPUT:   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
CRONOUTPUT:     return loop.run_until_complete(main)
CRONOUTPUT:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
CRONOUTPUT:     return future.result()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/commands.py", line 31, in execute
CRONOUTPUT:     executed = await super().execute(args)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/base.py", line 49, in execute
CRONOUTPUT:     if await subcommand.execute(args):
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/base.py", line 45, in execute
CRONOUTPUT:     await self._execute_impl(args)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/commands/commands.py", line 153, in _execute_impl
CRONOUTPUT:     tables = await session.get_tables(args.namespace)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 363, in get_tables
CRONOUTPUT:     table_list = await self._get(f"/dap/query/{namespace}/table", TableList)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 193, in _get
CRONOUTPUT:     await self.authenticate()
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 328, in authenticate
CRONOUTPUT:     properties = await self._post_auth_request(self._credentials.basic_credentials)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 278, in _post_auth_request
CRONOUTPUT:     return await self._process(response, TokenProperties, suppress_output=True)
CRONOUTPUT:   File "/usr/local/lib/python3.10/dist-packages/dap/api.py", line 298, in _process
CRONOUTPUT:     raise DAPClientError("malformed HTTP response")
CRONOUTPUT: dap.api.DAPClientError: malformed HTTP response

[IanGoh]

It looks like the root issue is authentication.  So you might want to concentrate on checking that works.

[reynlds]

Client version 3.1.0 is pretty old...an upgrade may be in order. I'm running 0.3.9.1.

Also, do you have your credentials set in your user profile? If not, you'll have to pass them as part of the script. I use cron also and have exported these items (~/.bash_profile) to support the sync process:

declare -x DAP_API_URL="https://api-gateway.instructure.com/"
declare -x DAP_CLIENT_ID="XXXXX"
declare -x DAP_CLIENT_SECRET="XXXXX"
declare -x DAP_CONNECTION_STRING="postgresql://postgres_username:postgres_password@postgres_db_hostname:5432/postgres_db"

[ThayerMerritt]

Hello, 

While I'm not sure that I'm receiving the same error exactly, I've run into similar issues as well. Echoing reynlds I presume this is related to cron running as a system-level user rather than your local user. This would prevent environment variables from being passed correctly, assuming that you're using those. Your interactive shell/terminal would pick these up. Since DAP doesn't have access to your env variables, which contain your credentials without passing the user information, a forbidden: invalid authentication credentials would be expected. I want to say that cron/crontab can have the username specified in the command. I hope this steers you in a productive direction!

Good luck!