Find the data causing an Invalid UTF-8 error on file import

[PeterOConnell]

When trying to test an upload for new courses I am getting an Invalid UTF-8 error from one of my files. Is there any way to identify the data that is causing the error? The file generates courses for our current academic year and uploads fine, but the test for next years courses is causing an error. It seems most likely there is an error somewhere in my data, but I don't know how to find it in the 1200ish courses I have to import.

[mzimmerman]

Hi @PeterOConnell 

In my experience, that error doesn't appear to be related to any particular characters or entries in the file.  Rather, it seems to be related to the encoding of the file itself.  I sometimes find that I need to open and then save the file in a different text editor, specifically saving it as UTF-8, though I confess that it happens infrequently enough that I don't remember specifically which editors cause/fix the problem.   This is mentioned in https://community.canvaslms.com/t5/Troubleshooting/Importing-SIS-CSV-files/ta-p/861 

[PeterOConnell]

I had found that if I opened the file and specifically saved it as a UTF-8 csv it worked, however we run a task through our system that then uses PowerShell to package all the files into one zip file ready for upload.

We have found a solution using SQL to produce a validation within our system to check where there are non UTF-8 characters in our data so hopefully will not have this issue in the future.