CanvasData2 errors -- some(?) files are apparently gzipped now?

[ColinMurtaugh]

Hi @oxana --

I'm getting errors (see below) using the nodejs dap CLI (version 1.0.18), and after digging in deeper I found that some of the data files are now gzipped.  It seems to me that the dap CLI isn't handling these files properly. 

I also noticed that the schema now includes a table called "access_tokens-DLQ"; I'm guessing that this related to an internal dead letter queue and isn't meant to be there. 

Thanks--

Colin

---------------------

Error details:

Error [ERR_STREAM_CANNOT_PIPE]: Cannot pipe, not readable
    at new NodeError (node:internal/errors:278:15)
    at WriteStream.Writable.pipe (node:internal/streams/writable:283:24)
    at Stream.through (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3474:14)
    at /Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3532:20
    at Array.reduce (<anonymous>)
    at Function._.pipeline (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3531:20)
    at Object.write (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/format/index.js:99:29)
    at async compactAndWrite (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/helpers/shared.js:45:3)
    at async Function.compactSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:71:5)
    at async Function.getFreshSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:59:5)
    at async handler (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:93:9)
    at async Object.withDir (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/tmp-promise/index.js:35:12)
    at async Function.updateExistingSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:99:5)
    at async startQuery (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:118:11) {
  code: 'ERR_STREAM_CANNOT_PIPE'
}

[oxana]

@ColinMurtaugh  are you on 1.0.18 cli version ?  what's the command you are running? 

[ColinMurtaugh]

Hi @oxana -- I am using the latest (1.0.18). I'm running "dap snapshot courses".  FWIW that command is also printing out the help text before giving the error. Here's the some more terminal output that shows the whole thing:

~/w/canvas-data-2 ❯❯❯ dap --version
1.0.18
~/w/canvas-data-2 ❯❯❯ dap snapshot courses
[2020-12-04T14:35:38.312Z info]: Requesting snapshot for courses...
[2020-12-04T14:35:38.649Z info]: Successfully sent a request for courses's snapshot
[2020-12-04T14:35:38.893Z warn]: Unsorted streams. Retrying...
[2020-12-04T14:35:38.894Z info]: Requesting snapshot for courses....
[2020-12-04T14:35:39.140Z info]: Successfully sent a request for courses's snapshot
dap snapshot [tables...]

Get snapshot of a table or set of tables

Options:
  --version      Show version number                                                       [boolean]
  --config, -c   JSON config file includes authorization token   [required] [default: "config.json"]
  --concurrency  The number of tables to fetch concurrently                            [default: 10]
  --format       The output format of data. Supported formats: csv, json            [default: "csv"]
  --output, -o   The directory to download to                                 [default: "snapshots"]
  -h, --help     Show help                                                                 [boolean]

Error [ERR_STREAM_CANNOT_PIPE]: Cannot pipe, not readable
    at new NodeError (node:internal/errors:278:15)
    at WriteStream.Writable.pipe (node:internal/streams/writable:283:24)
    at Stream.through (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3474:14)
    at /Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3532:20
    at Array.reduce (<anonymous>)
    at Function._.pipeline (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/highland/lib/index.js:3531:20)
    at Object.write (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/format/index.js:99:29)
    at async compactAndWrite (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/helpers/shared.js:45:3)
    at async Function.compactSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:71:5)
    at async Function.getFreshSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:59:5)
    at async handler (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:93:9)
    at async Object.withDir (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/tmp-promise/index.js:35:12)
    at async Function.updateExistingSnapshot (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:99:5)
    at async startQuery (/Users/cmurtaugh/workspace/canvas-data-2/node_modules/data-access-platform-cli/lib/commands/snapshot.js:118:11) {
  code: 'ERR_STREAM_CANNOT_PIPE'
}

I was getting a similar error when running my work-in-progress Python version of the client, and I was able to correct it by gunzipping the file data for files whose name ends in '.gz'. 

--Colin

[oxana]

@ColinMurtaugh could you use node version 10 or 12 please, looks like we could repro the error with node 13 but it goes away with 12, it's related to one of the unsupported libraries in the newer node version rather than gzipping. 

[ColinMurtaugh]

Ah, yeah - I've got node 15.3.0 currently. In any case, I mentioned the error more for the benefit of others who may run into the same thing (and this is one of the reasons that I'm really not interested in a JS/Node client -- the dependency-management nightmare isn't a world I want to live in!)

I've moved on to writing a Python client (though this will be much easier to do once API documentation is available! 🙂 ).

Thanks,

Colin