I'm getting what looks like a few corrupted snapshot files in each run

[pgo586]

As I run my nodejs script for complete snapshot (jsonl) retrievals (I've written this API-based script a few months ago, which is able to retrieve all the DAP tables in either snapshot or incremental form), and then attempt to copy all the retrieved files to another location, I've been seeing corrupted files in the last month or so (perhaps a couple per run sometimes). The symptom is that the copy command does not succeed and so I have to conclude that the (gz) files are corrupt (also when trying to decompress, I get an input/output error). Can Instructure and others comment on this? Is this a bug that is in the process of getting fixed perhaps? 

[LeventeHunyadi]

If you are having any issues interacting with the DAP API, make sure you always share a job ID such that the support team and/or engineer on call/duty can look into the specifics. They should be able to inspect the files generated and saved to S3 buckets, and see if anything is wrong with them.

A job ID is always returned in DAP API responses when you submit a snapshot or incremental query.

[pgo586]

Unfortunately @LeventeHunyadi there was NO problem with the API; simply corrupted files in my directory AFTER the complete snapshot had run without errors (otherwise I would definitely include job IDs). This is quite worrisome, as it essentially renders the whole snapshot useless (as it's incomplete) :-(. I had not had this problem until recently though. 

[LeventeHunyadi]

When you submit a snapshot or incremental query, DAP API assigns a job ID. The job ID spans the entire lifetime of the process. When the query has produced a result-set, the result-set is saved to an S3 bucket directory (prefix) specific to the job ID. When you retrieve a pre-signed URL, this is the location the pre-signed URL points to.

If you can share the job ID with the (support) team, the engineer on duty can look into the files that were generated and saved to S3. They can check if everything is OK with the files. Specifically, if the files are already malformed in S3, the engineering team has work to do: it's a server-side error. However, if the files got corrupted during the download process, it might be related to the download script you use: it's a client-side error.

It's always a good idea to save the job ID. For example, the official DAP client library always logs the job ID (to stdout or a file, depending on your choice).

[pgo586]

Thanks for the reply @LeventeHunyadi . I am actually quite familiar with the job Ids and how they're used in the API, and in fact, my script logs probably too much already :-). So when you asked for the job ids, remembering that those are logged as soon as they are obtained, I was afraid that it would be hard to associate the job id references with the eventual files being downloaded -given the absence of errors-, but I hadn't actually put the effort to do this yet.  I have just inspected carefully the log and I actually *do see* the job id logged with the table it is actually associated with. So here are the couple of tables (and their job ids) affected during a June 7th snapshot run by my script: 1- assessment_questions: 8887f837-bbbc-46ef-a4df-009700a27b82  (problem file: part 0007), 2- assignment_groups: 51eb4027-ecb5-4782-99f0-f38d209afa12 (problem file: part 00001)
Hope this helps tell whether this is a server or client error, as you described. 

[LeventeHunyadi]

I am afraid DAP has a retention time of 48 hours for query results. When the objects have expired, they are irreversibly deleted, you need to submit another query to DAP API, and a new result-set would be generated, which is — likewise — available for 48 hours.

[pgo586]

ok, I'll submit this info next time as soon as I see this problem then. Sorry for the bad timing on my part. 

[pgo586]

Hi again @LeventeHunyadi , I happen to have another corrupted file from yesterday's snapshot retrieval, and here's the information:

- table: attachment_associations t
- job id: fe90cafe-40be-440d-88a5-043966e9ee74, 
- download URL that produced a locally downloaded corrupted file: https://data-access-platform-output-prod-iad.s3.amazonaws.com/output/rootAccountId%3DWTbP67mC863Zx9q...

Please let me know if there's any other information you need to know. Thanks in advance for looking into it!

BTW: I'm not sure you'll get to see the complete URL above; it corresponds to part-00001 of the table though.