Clarification Needed on Data Extraction from Canvas Data 2 API

[HarshithaGP]

Hi Team,

I am developer working at Fivetran.
I am currently working on pulling data from Canvas using the API and have encountered some issues. According to the Canvas Data 2 API documentation, I need to specify a date from which to pull the data. However, if I set a date beyond 2022-10-15, I receive the following error:

"message": "The ‘since’ parameter in the request (2022-10-08T00:00:00Z) precedes the earliest data available (2022-10-28T21:04:02.145Z)."

I’ve noticed that this initial date varies across different tables, leading to some confusion about whether this is the starting date for data extraction. Specifically, I would like to clarify whether we can only retrieve data from previous years relative to the current date. (For example since two years of data relative to the current date, Next year if we call the API again, will this initial date is going to change or it will remain same) Is this something your source will work like this? OR Will this date be fixed forever?

Additionally, I see that Canvas Data 2 was introduced in 2023, but I am unsure how data will be available from 2022 if the service starts in 2023.

Could you please explain how this will work?

Thank you for your assistance!

[sor1]

I believe that the date that you are referencing is used to manage incremental data requests.   One takes a table snapshot -and then subsequently runs incremental snapshots using the date of the previously successful request in the request.   So the first incremental would start with the date of the table snapshot and then the second incremental would start with the timestamp of the first incremental. 

One then needs to apply the incremental records against the base snapshot  (adds, delete, etc).  

This process is focused on helping us get to a reasonably close copy of the current table.   There isn't any way to recreate the table, say three months ago,  unless you had a full snapshot prior to that time and had  incremental updates since then.  

[LeventeHunyadi · Accepted solution]

DAP API does not support time travel, it only allows you to retrieve a current state with a snapshot query, and then use incremental queries to keep that state up to date with the state in DAP (Data Access Platform, a.k.a. CD 2) as data is inserted/updated/deleted. Specifically, you can't pass an arbitrary timestamp to an incremental query, you must always use the at or until timestamp returned by a previous snapshot or incremental query, Otherwise, you will get cryptic errors. Other than providing the ability to chain incremental queries, the timestamp passed in the field since or received in the field until has no business meaning, and varies across tables and institution accounts.