CD2: DAP Client Plans to support non-postgresql databases

[ismail_orabi]

Will the DAP client support other databases (specifically MS SQL Server)? If so, is there an estimated timeline for this? It seems the DAP client is the best tool for building an automated pipeline to a data warehouse and does a lot of the work that would otherwise be tedious via the API.

[LeventeHunyadi · Accepted solution]

Yes, we plan on supporting multiple database engines, and DAP client library has been engineered with this in mind. However, there is currently no timeline for adding support for other database engines such as Microsoft SQL. Currently, we are working on consolidating an extension interface such that contributors can easily add integrations for other database engines. DAP client library database integration is based on SQLAlchemy, which is an abstraction layer supporting many databases. However, there are some PostgreSQL-specific pieces that we need to get rid of or hide behind an interface before new engines like Oracle, Microsoft SQL, MySQL, etc. can be seamlessly integrated.

Would you be interested in contributing Microsoft SQL support, or beta-testing such an integration if it were available?

[laned]

We'd be willing at the very least to test MSSQL integration. We may be positioned to help with isolating bugs and such as well.

[tyler_clair]

We would possibly be interested in helping with the Oracle side.

[pkreemer]

Hi, I'd be interested in testing any beta (or alpha) version of a MS SQL plugin for DAP! It's great to see the plugin framework documented now (https://data-access-platform-api.s3.amazonaws.com/client/plugins.html) but even better to see some early form of MS SQL plugin. 🙂

thanks-

Paul (WA state community colleges)

[LeventeHunyadi]

The plugin interface for DAP client library was not finalized for the version series 0.3.x. In fact, we have recently overhauled the internal database engine integration API to improve robustness and increase performance. It is now much simpler to write an integration for a new database engine than it used to be.

We are currently working towards a 0.4.x series of the DAP client library. Our internal beta supports PostgreSQL, Microsoft SQL and MySQL. Unfortunately, we don't have a date yet when this new version is going to be publicly available.

[sgergely]

Hello @ismail_orabi, @pkreemer, @laned, @tyler_clair!

I'm happy to announce that we have released the DAP CLI 1.3.0 where we have added support for MS SQL.

Please check it out and let us know how it works for you!

[AlanWallace]

Great news on the MS SQL support!  Our destination database, unfortunately, is Oracle, and we would be interested in testing anything that you come up with.