Large number of failed snapshot jobs

[jdh132]

We are encountering a large number of failed job when trying to pull table snapshots. The snapshots seem extremely slow when compared to mid-July. After an hour the token expires and I have altered the code to renew and keep querying jobs if this occurs.

- Received job state 'running' for job ID '24f5f523-7fe9-46f8-aa60-93c4d280e156'

- Received job state 'running' for job ID '24f5f523-7fe9-46f8-aa60-93c4d280e156'
- Received job state 'failed' for job ID '24f5f523-7fe9-46f8-aa60-93c4d280e156'
- Job is failed.

[Edina_Tipter]

@jdh132 it looks like there is a bug. In your example, in the assignment_groups, there was a value we could not process in CSV. We are retrying 3 times and then fail the job, thus the long waiting time. We will have to address this. Thank you for the feedback.
Do you remember encountering this with other tables as well?

[jdh132]

Hi @Edina_Tipter Sorry, I had no notifications that there was a reply. This error was for most of the tables in fact which was very surprising from the first time I ran a snapshot of all files. I'll attempt another pull on Monday then.

[Edina_Tipter]

Hi @jdh132 , We have released a fix to the issues we found when processing the csv. I would expect that this has solved the issue you reported but I would appreciate a feedback how the pulling of the new dataset went yesterday.

[jdh132]

Hi @Edina_Tipter I ran the snapshots for about 6 hours in the background yesterday and got about 35 of the 65 files before I quit for the day. It does appear the issue I had seen is fixed. The speed is still very slow and I'm downloading the remaining tables today to look through. The speed is still a limiting factor; In comparison to CD1 which I can download all tables in 30 minutes if I exclude the requests.

[Edina_Tipter]

Hi @jdh132 , Thank you for feedback. It means our fix has worked.
This quarter is dedicated to improve the speed. I assume, that the most time is spent querying the job status. Did you have the same experience?

[jdh132]

@Edina_Tipter That's good to hear. Yeah, I based our current script for testing on the bash script sample provided so it's polling every 10 seconds. I altered it for the breaking changes this summer.

[jdh132]

@Edina_Tipter I found another table today that is having the same issue

- Received job state 'running' for job ID '4bdc81ba-d34e-40ae-bd28-95fdd49f339a'
- Received job state 'running' for job ID '4bdc81ba-d34e-40ae-bd28-95fdd49f339a'
- Received job state 'null' for job ID '4bdc81ba-d34e-40ae-bd28-95fdd49f339a'
- Job timed out after 1 hour.
- renewing token...
- Received job state 'failed' for job ID '4bdc81ba-d34e-40ae-bd28-95fdd49f339a'
- Job is failed.

[jdh132]

btw.. it was the submissions table. The rest of the dump seemed fine though wiki_pages took well over an hour. The rest of the job was complete in 8 hours for a total of 14 hours to receive all snapshots.

[jdh132]

I also can't pull the submissions table through jsonl either. I'm guessing it's too large. The asset_user_accesses table is the same issue and can't be pulled in either format.

[Edina_Tipter]

@jdh123 Thank you for the feedback. Let us investigate this. One should be able to pull any table.

[jdh132]

@Edina_Tipter Thanks, Submissions is a table we use often and is the largest table in CD1. The breaking changes around how file downloads are handled seems to facilitate multiple files for each tables but I haven't see that implemented yet. This was one of the tables I might have expected this.

[Edina_Tipter]

@jdh132 Thank you for the observations. Indeed the API is designed to be able to serve splitting bigger files and we have such a ticket on our backlog. Nevertheless, until the end of the week we will release a quick fix so that you can download bigger files as well such as submissions.

Also, thank you for being mindful of our API throughput, not to block the pipeline with concurrent queries, but please feel free to try that out as well. 

[Edina_Tipter]

@jdh132 we will take a look at the timeout/failed issue.

Are you triggering the downloads one after the other in a chain or running them concurrently?

[jdh132]

@Edina_Tipter I'm running them sequentially. I could run them concurrently but didn't want to overwhelm your backend. If that's an option I could certainly do that and throttle to your recommendations.