[ARCHIVED] duplicate records on files like account, courses and assignments

[ddaza]

hi, I'm dealing with this issue:

for some reason, in my canvasdata files, I'm seeing duplicate records for courses, accounts or assignments. Is this normal? I'd double check the course ID and it shows 2 records for the same ID. what could be wrong?#

[robotcars]

Hi  @ddaza 

Often times when there are duplicate rows in the files themselves it's the result of a crash during the batch process that creates the files. When this happens secondary jobs are spun up to recover and ensure that all data was delivered. This can usually be cleaned up with the import process or a few command line tools. Can you confirm if you have duplicate rows in the file or just in the database? If it's just the files, which affected the database, tomorrow's dump should be better. If you're importing duplicates to your database you may want to prep and clean the data before loading it into the database, or workout an import process that won't import duplicates, will flag errors on primary key.

[ddaza]

thanks.

Duplicate records are in canvasdata downloaded files with the CLI. I was wondering if the problem is just in latest week files but it seems that problem is persisting. I'd submitted a ticket with canvasdatahelp to review my canvasdata files.

[ddaza]

Ok, I got an email from canvasdatahelp telling me that files downloaded from canvas console don't have duplicate records, I downloaded the file and try to unpack it with the CLI, but I got this error

events.js:174

      throw er; // Unhandled 'error' event

      ^

Error: unknown compression method

    at Zlib.zlibOnError [as onerror] (zlib.js:162:17)

Emitted 'error' event at:

    at errorOrDestroy (internal/streams/destroy.js:107:12)

    at Unzip.onerror (_stream_readable.js:733:7)

    at Unzip.emit (events.js:198:13)

    at Zlib.zlibOnError [as onerror] (zlib.js:165:8)

I was assuming that maybe the file that I downloaded was wrong, so I did a CLI fetch to get the same file, now I can't unpack my files. 

[jsimon3]

Did you ever find a fix to this issue?

[ddaza]

Yes, I forgot to shared it on the community. When somebody is using the "fetch" option to download selected canvasdata files, prior to "unpack" these files, it's mandatory to deleted the older files. If we keep the older files (example: several fetched files from user_dim), the unpack option use all available files to create a new unpacked file. Because of this, everytime that I download new files and re-ran the unpack process including new file with old files, the unpacked result file included records from all the files.

I hope this explanation works.