Any update on the next release of the DAP api? We are excited to use DAP with SQL server.

[J-J-Jason · Accepted solution]

It looks like a new dap version was released about three hours ago.

https://pypi.org/project/instructure-dap-client/

What's new?
https://data-access-platform-api.s3.amazonaws.com/client/RELEASE_NOTES.html#version-1-0-0-2024-02-28

[JasonBrandt]

Yay! We will be digging in shortly!!

[reynlds]

Time to upgrade blindly! No guts, no glory!!!

[kyle_cole]

Product Specialist for Data and Analytics here. Im excited that 1.0 is released now 😀 Please let me know how it goes! 

[ColinMurtaugh]

I'm making some progress modifying my existing AWS Lambda-based pipeline to the new version. 

I am getting an error that I think is caused by this bug in one of the new dependencies: https://github.com/MagicStack/asyncpg/issues/952#issuecomment-1526168429

In my case I am not using IAM authentication but I do have my RDS proxy set to require TLS. The DAP client running in Lambda is not able to connect to the RDS proxy if "require TLS" is turned on, but it is able to connect if it is not. Interestingly I don't seem to have this problem when running the code locally. 

Hopefully I'll be able to work around it one way or another.

--Colin 

[LeventeHunyadi]

Does downgrading to an earlier version of asyncpg help? I don't think we are depending on any features in the latest asyncpg version, we simply bumped the required version to the latest available at the time the most recent version of the DAP client library written.

[ColinMurtaugh]

Aha - it wasn't asyncpg (I had 0.29.0 installed in both the working and broken environments) -- but I had different versions of aws-lambda-powertools installed. Turns out that 2.30.1 works but 2.34.2 doesn't. I'll need to dig deeper into that to figure out why, but at least I have working code again!

--Colin

[J-J-Jason]

The old version 0.3.18 has been yanked so I would probably use a fresh virtual environment instead of upgrading so that I have a copy of the previous version around just in case.

[GrantHicks]

Will this work for mssql now? Looks like the docs still say there is only support for postgres and mysql.

[kyle_cole]

@GrantHicks 

Only Postgres and MySQL are supported dialects. 

[YaejuneLee]

this is very cool bro

[reynlds]

As of updating this library, I'm now getting this error:

ERROR - [Errno 28] No space left on device

I've got a 250MB /tmp mount setup on my host which it's maxing out, erroring, flushing and moving on to the next table. I don't see any changes in the release notes, unless this is part of the "more efficient and reliable" "mechanism for replicating the data from DAP to target database". I also don't see any requirement to have a large /tmp mount for any reason.

It would be helpful for someone to confirm this. Thanks.

[reynlds]

More errors:

asyncpg.exceptions.NotNullViolationError: null value in column "name" of relation "course_sections" violates not-null constraint
DETAIL: Failing row contains (29818, null, 17349, null, 2022-03-07 16:08:02.961, 2022-03-07 16:10:13.093, active, null, null, null, null, null, null, null, null, null)


pysqlsync.formation.object_types.FormationError: operation not permitted; cannot drop values in an enumeration: dimdim

[LeventeHunyadi]

The first error seems to be unrelated to the client library, it seems as if DAP API would be returning records in the output that are not compliant with the declared table schema.

The second error is also a schema compatibility issue. In earlier versions of DAP API, the enumeration web_conferences__conference_type had a duplicate value but with a different character case: DimDim and dimdim. This has already been addressed in DAP API long ago (and the all lowercase version was removed) but if you have a database table schema from that time, the DAP API schema and the local database schema are now different. The straightforward solution is to drop the table (dropdb), and fetch a new snapshot (initdb), which will continue to work with successive incremental updates (syncdb). The more complex solution is to re-cast the column in the expected SQL type such that it becomes

CREATE TYPE "canvas"."web_conferences__conference_type" AS ENUM ('__dap_unspecified__', 'AdobeConnect', 'BigBlueButton', 'CiscoWebex', 'DimDim', 'Wimba', 'Wiziq');

I recommend reporting these errors to Instructure support such that they can be properly tracked.

[jsimon3]

We built a C# wrapper that can quickly ingest the data and spit out TSQL as well
there are examples in the related repos too.
https://github.com/uvadev/PullCanvasData2/pkgs/nuget/PullCanvasData2

[fentl]

Is it just me or does `dap list` now require a `--namespace <NAMESPACE>` parameter?  

Why?  Does that even make sense in the context of what `dap list` is supposed to be used?

[J-J-Jason]

You have to specify because there are a few different namespaces. canvas is where most of the tables live, but the web_logs table is in the canvas_logs namespace and I think there is also a catalogs namespace. (The namespace parameter was always there.)

[LeventeHunyadi]

The reason is consistency.

In the beginning, there was a single namespace called canvas. Even though the namespace parameter has always existed (even in version 0.2.x), if you passed no value at the command line, it automatically assumed the default value of canvas. This was OK until the table web_logs was introduced, which is in the namespace canvas_logs, and Catalog tables came along, which are in the namespace catalog. (We can't put them all into the same namespace because multiple namespaces have tables of the same name but different schema/content, e.g. canvas.users is different from catalog.users.) Specifically, people have started looking for web_logs in canvas, and they were disappointed not to find web logs data in their account, when in fact all they needed was to pass the namespace name as a command line parameter.

By making it explicit that you need a namespace, our hope is that people find what they are looking for, especially as we extend DAP with more data sets, e.g. New Quizzes. (If and when additional data sets are added is subject to the product roadmap.)

[fentl]

Is there the possibility of adding a `namespaces` command similar to `list` so that a client can enumerate the namespaces (or just so we can know about them since I'm not finding any documentation on them on the dap client pages)

[fentl]

There but not required previously, now it's required and breaks scripts that call it without the namespace parameter.