File names in CD2 objects data

[lmsstaff]

So I'm wondering why this is, but if the file names returned from the table query endpoint (/dap/query/canvas/table/{table name}/data) are to be sequential (part0, part1, part2, etc.), why then does this not seem to be the case consistently across all calls to the endpoint?

For example, in a call to conversations, I get back 2 objects, part0 and part3.  I don't know what happened to part1 and part2.  Is this by design?  There is no mention of the peculiarity in the CD2 docs or API docs that I can tell.  And these are snapshot data calls too, which only adds to the mystery.

As well, sometimes the part0 file is empty.  Why return an empty file?

Are other users encountering this and what is the reasoning for this?  Just confusing.

[ColinMurtaugh]

I've always just treated the filenames as opaque identifiers; I don't think the docs promise any particular sequence. 

IIRC the occasional empty file is a known issue that will be addressed at some point, but as far as I know they can be ignored and don't actually cause a problem. 

In any case, I've moved on to using the "initdb" and "syncdb" functions in the dap library which handle all of the file fetching and assembly for you. It does require that you have a postgresql-compatible database to feed into, but in my experience it works well. 

--Colin

[LeventeHunyadi]

The file names returned by the DAP Query API endpoints do not adhere to any standard, they can be arbitrary, and your code should not assume any specific naming scheme. Instead, you should always extract the names returned in the JSON payload of the HTTP POST response body.

Parts of a (large) resultset are produced by parallel tasks that don't communicate with each other. It is possible that one task produces no output and therefore has no corresponding file. It is also possible that a task produces some output but does not return records in which case you might get an empty file.

I recommend the official DAP client library, which automatically tackles all these special cases.

[lmsstaff]

Got it.  Thanks for the info.

In my little developer mind, there is always this little voice that asks those types of questions.  In this case, I take what I'm given and play by the API rules.  Thanks again for the feedback.