The title kinda covers the question: how can I avoid sporadic 504 errors when conducting extensive API calls?
So, the basic situation is that I'm trying to generate a report of all instances of the appearance of a string withing the content of all courses within an account. Yeah, that in itself sounds like a lot, and it is. The calls are synchronous, though I intend to revise that once I get it working 100%, so there's not an issue of a call interfering with another.
Having said that, I am receiving inconsistent 504 errors, and I can't determine the reason for it. I know the connections are still good, so I can only assume it to be a glitch with the call handling, whether it be on my end or Instructure's, I'm not sure. As far as I can tell, every call I make is 100% valid as they do work, standalone or within a bulk queue. That the same code works individually and in groups leads me to believe that the issue isn't being caused by my code, after all, it that were the case the errors would be consistent when running with identical values, and they're not.
Searching around, I found plenty of people have had issues with making API calls for various reasons, but I couldn't find anything specifically for 504 Timeouts.
Has anyone encountered these with the Canvas API before? If so, how do you handle them?
My current solution is to simply resubmit the call with the header reports the error, but I'd rather avoid having to bloat the call queue any further. A single search of our Master shells took 15,486 calls (plus an additional 19 due to the timeouts) and 128 minutes to execute. The concern is that the account is constantly growing and those numbers are already high..