CD2: dap client basics

[dtod]

Trying to get the dap client running to no avail.

dap schema --table wikis
2023-04-20 09:53:35,688 - WARNING - Received error in response: {'message': 'The client is not authorized to perform this operation.'}
2023-04-20 09:53:35,692 - ERROR - {'message': 'The client is not authorized to perform this operation.'}

and 

dap initdb --namespace canvas --table accounts
2023-04-20 09:58:07,159 - ERROR -

I'm guessing a key/secret issue, but there's nothing obvious.

[ColinMurtaugh]

What version of the dap client do you have (dap --version)? Has your key expired (check in identity.instructure.com)? 

[dtod]

dap 0.3.7

No, the key has not expired. I'm sure it's something elementary, but no idea what.

[dtod]

Is there a way to specify the postgresql schema, i.e. not public?

[LeventeHunyadi]

By default, DAP client library creates a new schema for each DAP namespace, e.g. all Canvas table data goes into the PostgreSQL schema canvas, i.e. canvas.accounts, canvas.assignments, canvas.submissions, etc. This mapping is not configurable at the moment.

[dtod]

FYI - I tested the same key and secret in Postman, so it's not that per se. Probably something about the environment I'm using.

[phanley]

You may have done this already, but

dap --loglevel debug schema --table wikis

 will print out messages about the authentication flow, including the token response which might pinpoint where the problem is

Also, depending on which version you started using and which version you're using now, are you using env for DAP_CLIENT_ID & DAP_CLIENT_SECRET or the deprecated DAP_API_KEY (which is just the two combined)?  I don't use POSTMAN and haven't  worked with the API outside of the dap client, so it could be the case that the DAP_API_KEY value works with postman, but it doesn't work anymore with dap cli (for the last few versions), now the client_id and client_secret need to be the seperated api_key.

[LeventeHunyadi]

Support for DAP_API_KEY has been removed from DAP client library. I recommend using DAP_CLIENT_ID and DAP_CLIENT_SECRET for both the DAP client library and direct DAP API calls. Even though DAP_API_KEY may continue to work for a while in DAP API endpoint calls, it should be replaced with DAP_CLIENT_ID and DAP_CLIENT_SECRET.

For details, check out the official documentation how DAP_CLIENT_ID and DAP_CLIENT_SECRET work with direct API calls.

[dtod]

Thank you both. I've validated the key and secret using Postman and curl, and even native python code, but still getting a failure with the dap client:

~ # dap --loglevel debug --base-url https://api-gateway.instructure.com --client-id "${CLIENT_ID}" --client-secret "${SECRET}" schema --table wikis
2023-04-25 17:10:09,005 - DEBUG - Using selector: EpollSelector (selector_events.py:54)
2023-04-25 17:10:09,006 - DEBUG - Client region: us-east-1 (api.py:78)
2023-04-25 17:10:09,006 - DEBUG - Get schema of table: wikis (api.py:343)
2023-04-25 17:10:09,006 - DEBUG - Authenticating to DAP in region us-east-1 (api.py:289)
2023-04-25 17:10:09,338 - DEBUG - GET/POST response payload:
{'message': 'The client is not authorized to perform this operation.'} (api.py:269)
2023-04-25 17:10:09,338 - WARNING - Received error in response: {'message': 'The client is not authorized to perform this operation.'} (api.py:275)
2023-04-25 17:10:09,341 - DEBUG - Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/dap/__main__.py", line 126, in console_entry
    main()
  File "/usr/lib/python3.10/site-packages/dap/__main__.py", line 115, in main
    asyncio.run(dapCommand.execute(args))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/lib/python3.10/site-packages/dap/commands/commands.py", line 31, in execute
    executed = await super().execute(args)
  File "/usr/lib/python3.10/site-packages/dap/commands/base.py", line 49, in execute
    if await subcommand.execute(args):
  File "/usr/lib/python3.10/site-packages/dap/commands/base.py", line 45, in execute
    await self._execute_impl(args)
  File "/usr/lib/python3.10/site-packages/dap/commands/commands.py", line 181, in _execute_impl
    await session.download_table_schema(
  File "/usr/lib/python3.10/site-packages/dap/api.py", line 360, in download_table_schema
    versioned_schema = await self.get_table_schema(namespace, table)
  File "/usr/lib/python3.10/site-packages/dap/api.py", line 344, in get_table_schema
    versioned_schema = await self._get(
  File "/usr/lib/python3.10/site-packages/dap/api.py", line 169, in _get
    return await self._process(response, response_type)
  File "/usr/lib/python3.10/site-packages/dap/api.py", line 276, in _process
    raise error_object
dap.dap_error.ServerError: {'message': 'The client is not authorized to perform this operation.'}
(__main__.py:140)
2023-04-25 17:10:09,341 - ERROR - {'message': 'The client is not authorized to perform this operation.'} (__main__.py:141)

On the same server with curl:

curl --request POST https://api-gateway.instructure.com/ids/auth/login --user "${CLIENT_ID}:${SECRET}" --data-urlencode 'grant_type=client_credentials'
{"access_token":"eyJhbG...etc....etc.

[LeventeHunyadi]

I am afraid I am at a complete loss what could be happening. I have tried both methods, and both yield success: one downloads the schema for the Canvas table wikis, and the other returns a JWT access token, both of which are the expected behavior. However, your log timestamps for the DAP client execution trace seem to be from a while ago, are you sure you use the right client ID and secret, the exact same as for curl?

DAP client library defines some standard environment variable names that it recognizes even if you don't pass base URL, client ID and secret values as command-line parameters:

export DAP_API_URL=https://api-gateway.instructure.com
export DAP_CLIENT_ID=us-east-1#0c59cade-...-2ac120002
export DAP_CLIENT_SECRET=xdEC0lI...4X4QBOhM

I would double-check if you don't set those by accident (to some other value). (Your example uses custom environment variable names CLIENT_ID and SECRET.)