Activity Feed
- Posted Are incremental retrievals (via API) currently working? on Data Access Platform: Canvas Data 2 Alpha. 11-16-2022 01:13 PM
- Got a Kudo for Re: Too many failed jobs (34) after last maintenance. 11-02-2022 07:27 AM
- Posted Re: Too many failed jobs (34) after last maintenance on Data Access Platform: Canvas Data 2 Alpha. 11-02-2022 07:12 AM
- Kudoed Re: Too many failed jobs (34) after last maintenance for Edina_Tipter. 11-01-2022 09:40 AM
- Posted Too many failed jobs (34) after last maintenance on Data Access Platform: Canvas Data 2 Alpha. 10-27-2022 07:29 AM
- Posted Re: Are authentication tokens correctly encoding their expiration time? on Data Access Platform: Canvas Data 2 Alpha. 10-26-2022 11:43 AM
- Posted Are authentication tokens correctly encoding their expiration time? on Data Access Platform: Canvas Data 2 Alpha. 10-26-2022 10:09 AM
- Kudoed Re: Is the maintenance update over? I'm still unable to do any data retrievals for Edina_Tipter. 10-24-2022 09:35 AM
- Posted Is the maintenance update over? I'm still unable to do any data retrievals on Data Access Platform: Canvas Data 2 Alpha. 10-24-2022 09:19 AM
- Posted Re: Would like more information about hard-deletes on Data Access Platform: Canvas Data 2 Alpha. 10-19-2022 11:11 AM
- Posted Would like more information about hard-deletes on Data Access Platform: Canvas Data 2 Alpha. 10-18-2022 02:12 PM
- Got a Kudo for Re: Has anybody managed to create a complete script for data retrieval and can share it?. 06-30-2022 04:07 PM
- Kudoed Re: Has anybody managed to create a complete script for data retrieval and can share it? for meghan_lewis. 06-30-2022 07:48 AM
- Kudoed Re: Has anybody managed to create a complete script for data retrieval and can share it? for meghan_lewis. 06-30-2022 07:48 AM
- Posted Re: Has anybody managed to create a complete script for data retrieval and can share it? on Data Access Platform: Canvas Data 2 Alpha. 06-30-2022 07:47 AM
- Got a Kudo for Re: Unable to retrieve incremental updates for table (users). 06-30-2022 12:13 AM
- Posted Re: Unable to retrieve incremental updates for table (users) on Data Access Platform: Canvas Data 2 Alpha. 06-29-2022 12:52 PM
- Kudoed Re: Unable to retrieve incremental updates for table (users) for chriscas. 06-29-2022 12:51 PM
- Posted Unable to retrieve incremental updates for table (users) on Data Access Platform: Canvas Data 2 Alpha. 06-29-2022 10:01 AM
- Kudoed CD2 Table Listing for reynlds. 06-29-2022 09:56 AM
My Posts
| Post Details | Date Published | Views | Kudos |
|---|---|---|---|
|
Are incremental retrievals (via API) currently working? I've been trying to exercise the incremental retrieval functionality of the API, but getting 400 errors for all my requests. I've double checked that I am passing on a value for "since" with proper f... |
11-16-2022 |
100 |
0 |
|
Too many failed jobs (34) after last maintenance Yesterday evening I was finally able to run a complete snapshot job for all tables (a first after maintenance update). However, I saw what looks like too many failed jobs (34) and one 404 (although I... |
10-27-2022 |
191 |
0 |
|
Are authentication tokens correctly encoding their expiration time? I understand that authentication tokens are returned by the API together with an 'expires_at' attribute indicating their expiration time. In addition, the documentation also suggests that this expira... |
10-26-2022 |
56 |
0 |
|
Is the maintenance update over? I'm still unable to do any data retrievals We heard last week that the maintenance update was 'almost' over, but it's not clear to me whether it ever ended or not. In fact, I still get 401 errors which are accompanied by the 'System is under ... |
10-24-2022 |
70 |
0 |
|
Would like more information about hard-deletes I apologize if this question was asked before, but I couldn't find anything relevant online. I understand that soft-deletes are treated as a data 'update' , and therefore checking some attribute in t... |
10-18-2022 |
117 |
0 |
|
Unable to retrieve incremental updates for table (users) Not sure whether this is supposed to be enabled for us yet or not, but I tried earlier today to retrieve an update for the users table using the provided bash script, and the process did not work bec... |
06-29-2022 |
235 |
0 |
|
Has anybody managed to create a complete script for data retrieval and can share it? Hi fellow alpha testers! I wonder if any of you has already written a (bash?) script that will grab a snapshot of ALL the tables that are available on the server and output those files either locally... |
06-27-2022 |
595 |
0 |
|
Should I assume that the db schema current last September is now outdated? I have a snapshot of data from last September and I wonder whether to start over with fresh snapshot data (assuming the schema has changed due to some re-architecting), or if it is safe to simply get... |
06-23-2022 |
93 |
0 |
|
Unsure which hostname to hit to grab our CD2 data (I posted this first as an answer to Edina's initial post - I apologize for the redundancy) Just getting started trying to retrieve our data, but I'm not sure which hostname to use in the URL. Could ... |
06-22-2022 |
189 |
0 |
|
Looking for "enrollment drop" information within Canvas Live Events: is it there? I've been looking closely at Canvas Live Events (Caliper format) to find in there enrollment adds and drops, as it would be an ideal place to find that type of information (essentially enrollment tra... |
02-18-2022 |
416 |
0 |
|
Do courses created via API (SIS import or regular API) generate 'course created' Canvas Live Events? Since I don't have access to a repository of raw-formatted Canvas Live Events (only Caliper), I wonder if others can answer this question: does course creation via API (either regular API or SIS impo... |
10-26-2021 |
541 |
0 |
|
Add Canvas page creation setting accepting search tags/keywords for third party search engines A Canvas page creation setting - only applicable to public Canvas pages- could be added that allows for the specification of (search) tags/keywords. This would allow external search engines (Google o... |
07-29-2021 |
791 |
0 |
|
Add content creation setting (to quizzes, assignments, etc.) for disabling Google page translation Within quizzes (or assignments, or any other type of content item within Canvas) a setting could be added that at content creation time determines whether the item's Canvas page (implicitl... |
07-29-2021 |
797 |
0 |
|
Do (default) SIS imports override enrollment status when this field was set via API? I am wondering whether changes made via API (in particular the enrollment status) would be considered sticky for the purposes of a subsequent SIS import. I understand this is definitely the case for ... |
02-05-2021 |
461 |
0 |
|
Any recommendations for Java-based LTI 1.3 libraries? ...equire a Spring Boot application, that would be very helpful. I haven't seen much publicly available (I am aware of the libraries created at the UOC, although I found no examples of use for a regular web a... |
10-29-2020 |
7696 |
1 |
|
File views/downloads within Live Events I'm interested in confirming some assumptions about how file views and downloads (for individual files) are represented -or not- in Live Events (both formats), specifically: 1- It appears to me th... |
12-06-2019 |
1472 |
2 |
|
Quickest programmatic way to tell whether course is empty? What is the quickest way (via API) to tell whether any content has been created in a given course? (meaning *any* type, and not a particular type of content). I am aware of the 'unused' account repor... |
04-25-2019 |
708 |
2 |
|
Can a custom help menu item display within the Canvas frame (not in a popup) I see that custom help menu items open in a new tab. Is there a way to have them display in the regular Canvas frame instead? (as any LTI tool would). Ideally, this would not require any custom javas... |
04-02-2018 |
1319 |
5 |
|
Java lti library failing to validate request when special characters are present I'm having a situation in which a Canvas user has set special characters in his name (a Spanish accent, a tilde on top of an 'n') and this is causing the Java LTI library I use for validating the req... |
04-21-2016 |
1199 |
2 |
11-16-2022
01:13 PM
I've been trying to exercise the incremental retrieval functionality of the API, but getting 400 errors for all my requests. I've double checked that I am passing on a value for "since" with proper formatting, and also that this value coincides with the (time) value retrieved from the last snapshot run. Here's an example of the error I get for each of these requests: Obtained error from Axios when creating retrieval job: [AxiosError: Request failed with status code 400] { code: 'ERR_BAD_REQUEST', config: { transitional: { silentJSONParsing: true, forcedJSONParsing: true, clarifyTimeoutError: false }, adapter: [Function: httpAdapter], transformRequest: [ [Function: transformRequest] ], transformResponse: [ [Function: transformResponse] ], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, env: { FormData: [Function] }, validateStatus: [Function: validateStatus], headers: { Accept: 'application/json, text/plain, */*', 'Content-Type': 'application/json', Authorization: 'Bearer mytoken', 'User-Agent': 'axios/0.27.2', 'Content-Length': 49 }, method: 'post', url: 'https://dap.insk8s.net/query/canvas/table/course_sections/data', data: '{"format":"jsonl","since":"2022-11-03T17:00:29Z"}' }, request: <ref *1> ClientRequest { _events: [Object: null prototype] { abort: [Function (anonymous)], aborted: [Function (anonymous)], connect: [Function (anonymous)], error: [Function (anonymous)], socket: [Function (anonymous)], timeout: [Function (anonymous)], finish: [Function: requestOnFinish] }, _eventsCount: 7, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 49, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: false, socket: TLSSocket { _tlsOptions: [Object], _secureEstablished: true, _securePending: false, _newSessionPending: false, _controlReleased: true, secureConnecting: false, _SNICallback: null, servername: 'dap.insk8s.net', alpnProtocol: false, authorized: true, authorizationError: null, encrypted: true, _events: [Object: null prototype], _eventsCount: 10, connecting: false, _hadError: false, _parent: null, _host: 'dap.insk8s.net', _closeAfterHandlingError: false, _readableState: [ReadableState], _maxListeners: undefined, _writableState: [WritableState], allowHalfOpen: false, _sockname: null, _pendingData: null, _pendingEncoding: '', server: undefined, _server: null, ssl: [TLSWrap], _requestCert: true, _rejectUnauthorized: true, parser: null, _httpMessage: [Circular *1], [Symbol(res)]: [TLSWrap], [Symbol(verified)]: true, [Symbol(pendingSession)]: null, [Symbol(async_id_symbol)]: 848, [Symbol(kHandle)]: [TLSWrap], [Symbol(lastWriteQueueSize)]: 0, [Symbol(timeout)]: null, [Symbol(kBuffer)]: null, [Symbol(kBufferCb)]: null, [Symbol(kBufferGen)]: null, [Symbol(kCapture)]: false, [Symbol(kSetNoDelay)]: false, [Symbol(kSetKeepAlive)]: true, [Symbol(kSetKeepAliveInitialDelay)]: 60, [Symbol(kBytesRead)]: 0, [Symbol(kBytesWritten)]: 0, [Symbol(connect-options)]: [Object], [Symbol(RequestTimeout)]: undefined }, _header: 'POST /query/canvas/table/course_sections/data HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'Authorization: Bearer mytoken\r\n' + 'Content-Length: 49\r\n' + 'Host: dap.insk8s.net\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: Agent { _events: [Object: null prototype], _eventsCount: 2, _maxListeners: undefined, defaultPort: 443, protocol: 'https:', options: [Object: null prototype], requests: [Object: null prototype] {}, sockets: [Object: null prototype], freeSockets: [Object: null prototype] {}, keepAliveMsecs: 1000, keepAlive: false, maxSockets: Infinity, maxFreeSockets: 256, scheduling: 'lifo', maxTotalSockets: Infinity, totalSocketCount: 2, maxCachedSessions: 100, _sessionCache: [Object], [Symbol(kCapture)]: false }, socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, path: '/query/canvas/table/course_sections/data', _ended: true, res: IncomingMessage { _readableState: [ReadableState], _events: [Object: null prototype], _eventsCount: 4, _maxListeners: undefined, socket: [TLSSocket], httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: true, rawHeaders: [Array], rawTrailers: [], aborted: false, upgrade: false, url: '', method: null, statusCode: 400, statusMessage: 'Bad Request', client: [TLSSocket], _consuming: false, _dumped: false, req: [Circular *1], responseUrl: 'https://dap.insk8s.net/query/canvas/table/course_sections/data', redirects: [], [Symbol(kCapture)]: false, [Symbol(kHeaders)]: [Object], [Symbol(kHeadersCount)]: 14, [Symbol(kTrailers)]: null, [Symbol(kTrailersCount)]: 0, [Symbol(RequestTimeout)]: undefined }, aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'dap.insk8s.net', protocol: 'https:', _redirectable: Writable { _writableState: [WritableState], _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, _options: [Object], _ended: true, _ending: true, _redirectCount: 0, _redirects: [], _requestBodyLength: 49, _requestBodyBuffers: [], _onNativeResponse: [Function (anonymous)], _currentRequest: [Circular *1], _currentUrl: 'https://dap.insk8s.net/query/canvas/table/course_sections/data', [Symbol(kCapture)]: false }, [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kEndCalled)]: true, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype] { accept: [Array], 'content-type': [Array], authorization: [Array], 'user-agent': [Array], 'content-length': [Array], host: [Array] }, [Symbol(kUniqueHeaders)]: null }, response: { status: 400, statusText: 'Bad Request', headers: { date: 'Wed, 16 Nov 2022 20:58:17 GMT', 'content-type': 'application/json', 'content-length': '230', connection: 'close', 'x-amzn-requestid': 'f453f7cf-98ba-4b1b-8e99-e3f58a2b267f', 'x-amz-apigw-id': 'btlEIHwIoAMFuaA=', 'x-amzn-trace-id': 'Root=1-63754ee7-13df38f809e20d5b35ecad24;Sampled=0' }, config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, env: [Object], validateStatus: [Function: validateStatus], headers: [Object], method: 'post', url: 'https://dap.insk8s.net/query/canvas/table/course_sections/data', data: '{"format":"jsonl","since":"2022-11-03T17:00:29Z"}' }, request: <ref *1> ClientRequest { _events: [Object: null prototype], _eventsCount: 7, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 49, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: false, socket: [TLSSocket], _header: 'POST /query/canvas/table/course_sections/data HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'Authorization: Bearer mytoken\r\n' + 'User-Agent: axios/0.27.2\r\n' + 'Content-Length: 49\r\n' + 'Host: dap.insk8s.net\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, path: '/query/canvas/table/course_sections/data', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'dap.insk8s.net', protocol: 'https:', _redirectable: [Writable], [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kEndCalled)]: true, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(kUniqueHeaders)]: null }, data: { error: [Object] } } }
... View more
11-02-2022
07:12 AM
1 Kudo
Thanks Edina for the update. Glad that you are working to eventually prevent those failures from happening when many queries are submitted in parallel. I could certainly go back to running them in sequence (or fewer of them at once) for now.
... View more
10-27-2022
07:29 AM
Yesterday evening I was finally able to run a complete snapshot job for all tables (a first after maintenance update). However, I saw what looks like too many failed jobs (34) and one 404 (although I now understand that you may not be providing that particular able anymore, which may explain the 404 response). Here is the detail of the errors my script received, and which tables were successfully retrieved (the beginning is the list of errors; the successful retrievals are at the end). 35 errors have occurred as follows: [ { error: [AxiosError: Request failed with status code 404] { code: 'ERR_BAD_REQUEST', config: [Object], request: [ClientRequest], response: [Object] }, table: 'asset_user_accesses' }, { error: Error: Table: assignment_groups could NOT be retrieved since retrieval job: 40dd059c-e3a3-474d-8703-94e7a62dc602 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 6) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'assignment_groups' }, { error: Error: Table: attachments could NOT be retrieved since retrieval job: c6273b7c-85f2-4890-84d5-1cbb523a7a68 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 11) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'attachments' }, { error: Error: Table: comment_bank_items could NOT be retrieved since retrieval job: 4bf29cbc-5c36-4705-88b3-6cf8c3b63ed5 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 14) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'comment_bank_items' }, { error: Error: Table: content_participations could NOT be retrieved since retrieval job: 764b4ab4-cab9-40ce-a1f2-818a15c696b4 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 18) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'content_participations' }, { error: Error: Table: context_external_tools could NOT be retrieved since retrieval job: 36305a01-2aef-4d37-9bde-b5ba421f8a71 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 21) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'context_external_tools' }, { error: Error: Table: context_module_progressions could NOT be retrieved since retrieval job: 7e44c395-37ef-413b-9fc4-29e0c527e05a failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 22) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'context_module_progressions' }, { error: Error: Table: conversation_message_participants could NOT be retrieved since retrieval job: 976cce25-f6a1-4338-a8d3-e8d8e804a264 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 24) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'conversation_message_participants' }, { error: Error: Table: conversation_messages could NOT be retrieved since retrieval job: 4aa6cceb-e6ea-4a6e-8ed7-9be80a24fa11 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 25) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'conversation_messages' }, { error: Error: Table: conversations could NOT be retrieved since retrieval job: e4a7046f-e315-47d4-9398-d7d68e879bb4 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 27) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'conversations' }, { error: Error: Table: course_account_associations could NOT be retrieved since retrieval job: 8beab7e7-a93e-40d4-8803-a090ec1a6682 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 28) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'course_account_associations' }, { error: Error: Table: custom_gradebook_columns could NOT be retrieved since retrieval job: da0646ca-fb99-4b6b-b336-00816b0c9431 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 32) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'custom_gradebook_columns' }, { error: Error: Table: discussion_entries could NOT be retrieved since retrieval job: 2f8a3148-7c24-48b1-83ee-85bf223265f3 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 35) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'discussion_entries' }, { error: Error: Table: enrollment_states could NOT be retrieved since retrieval job: 0138f1f9-d7a6-42c1-97e6-f9479355c768 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 40) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'enrollment_states' }, { error: Error: Table: favorites could NOT be retrieved since retrieval job: 6a2d4ee5-684d-41b9-b44e-899dd2e6ec27 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 43) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'favorites' }, { error: Error: Table: grading_period_groups could NOT be retrieved since retrieval job: 15e06b8c-f874-424a-ae94-456a4ecee87e failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 45) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'grading_period_groups' }, { error: Error: Table: grading_standards could NOT be retrieved since retrieval job: 86344bc8-7542-4338-8f8d-7963813356f0 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 47) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'grading_standards' }, { error: Error: Table: group_categories could NOT be retrieved since retrieval job: 3082abc9-d2d8-4aab-b5a4-ca61b279d057 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 48) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'group_categories' }, { error: Error: Table: group_memberships could NOT be retrieved since retrieval job: 25bee3d4-04a3-42ac-98dc-d14cb781465f failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 49) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'group_memberships' }, { error: Error: Table: groups could NOT be retrieved since retrieval job: d2e1a278-9ee1-4a4c-8046-d1bf13d45010 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 50) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'groups' }, { error: Error: Table: learning_outcome_question_results could NOT be retrieved since retrieval job: 85624806-81dc-48f8-a3c7-f3f1b94952f0 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 53) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'learning_outcome_question_results' }, { error: Error: Table: learning_outcome_results could NOT be retrieved since retrieval job: 0d2df0ca-8057-4e3a-a0f4-e6ab1a7f89a1 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 54) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'learning_outcome_results' }, { error: Error: Table: lti_resource_links could NOT be retrieved since retrieval job: 45ccb717-67c5-43d2-86bf-a1770959d36c failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 57) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'lti_resource_links' }, { error: Error: Table: master_courses_child_subscriptions could NOT be retrieved since retrieval job: c70c54c0-9be4-40aa-8c01-9657f316a194 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 60) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'master_courses_child_subscriptions' }, { error: Error: Table: master_courses_master_content_tags could NOT be retrieved since retrieval job: 09023580-bc79-49b3-80be-bea156f5a45d failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 61) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'master_courses_master_content_tags' }, { error: Error: Table: outcome_proficiency_ratings could NOT be retrieved since retrieval job: 07753876-5ca4-47a9-8d88-43e80e6c56dc failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 67) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'outcome_proficiency_ratings' }, { error: Error: Table: quiz_groups could NOT be retrieved since retrieval job: 6e403dcf-a8dd-45cc-85ba-ff2a6e16d6bc failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 70) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'quiz_groups' }, { error: Error: Table: quiz_questions could NOT be retrieved since retrieval job: 541efc78-3a03-43f7-a141-d0973cd5f1ee failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 71) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'quiz_questions' }, { error: [AxiosError: Request failed with status code 404] { code: 'ERR_BAD_REQUEST', config: [Object], request: [ClientRequest], response: [Object] }, table: 'quiz_submissions' }, { error: Error: Table: rubric_assessments could NOT be retrieved since retrieval job: 996958d9-e7e6-4812-9f9e-4637df8393dc failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 76) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'rubric_assessments' }, { error: Error: Table: rubric_associations could NOT be retrieved since retrieval job: b5571eef-5834-4322-86f9-f342281e91d6 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 77) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'rubric_associations' }, { error: Error: Table: scores could NOT be retrieved since retrieval job: ef1f4f36-0238-48e5-ac36-cf19fe93dc32 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 80) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'scores' }, { error: Error: Table: submission_comments could NOT be retrieved since retrieval job: 1afd2b7e-9f93-426a-8377-2ebd1e736e6a failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 81) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'submission_comments' }, { error: Error: Table: submissions could NOT be retrieved since retrieval job: de77c9f8-5601-4434-aa0b-38ff1e678586 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 83) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'submissions' }, { error: Error: Table: users could NOT be retrieved since retrieval job: ec26274c-236e-43aa-9c45-aad2560bc473 failed! at retrieveCompleteTable (D:\eclipse-projects\CD2Scripts\allSnapshots.js:292:13) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Promise.all (index 86) at async retrieveTableSubset (D:\eclipse-projects\CD2Scripts\allSnapshots.js:328:19) at async retrieveAllTables (D:\eclipse-projects\CD2Scripts\allSnapshots.js:429:2), table: 'users' } ] The following tables were successfully retrieved: [ 'access_tokens', 'account_users', 'accounts', 'assessment_question_banks', 'assessment_questions', 'assignment_override_students', 'assignment_overrides', 'assignments', 'attachment_associations', 'calendar_events', 'canvadocs_annotation_contexts', 'communication_channels', 'content_migrations', 'content_participation_counts', 'content_shares', 'content_tags', 'context_modules', 'conversation_participants', 'course_sections', 'courses', 'custom_gradebook_column_data', 'developer_key_account_bindings', 'developer_keys', 'discussion_entry_participants', 'discussion_topic_participants', 'discussion_topics', 'enrollment_dates_overrides', 'enrollment_terms', 'enrollments', 'folders', 'grading_periods', 'late_policies', 'learning_outcome_groups', 'learning_outcomes', 'lti_line_items', 'lti_results', 'master_courses_child_content_tags', 'master_courses_master_migrations', 'master_courses_master_templates', 'master_courses_migration_results', 'originality_reports', 'outcome_proficiencies', 'post_policies', 'pseudonyms', 'quizzes', 'role_overrides', 'roles', 'rubrics', 'score_statistics', 'submission_versions', 'user_account_associations', 'user_notes', 'web_conference_participants', 'web_conferences', 'wiki_pages', 'wikis' ]
... View more
10-26-2022
11:43 AM
My mistake, sorry! I was incorrectly interpreting the value of the 'exp' claim within the auth token. I was assuming a millisecond value in there, but if I interpret it as seconds instead, it works as expected. Sorry for the confusion.
... View more
10-26-2022
10:09 AM
I understand that authentication tokens are returned by the API together with an 'expires_at' attribute indicating their expiration time. In addition, the documentation also suggests that this expiration time is encoded within the token itself. However, when I decode one such token, I obtain an 'exp' claim which does not seem to have a correct representation of time. That is, if I interpret the value as an absolute number of milliseconds (since the epoch), then it always gives me a time back in 1970 (that is, the token is always expired). If, on the other hand, I interpret it as a number of milliseconds from 'now', it gives me a time that is in two weeks or so (definitely incorrect). Could somebody please clarify whether the tokens' 'exp' claim is currently encoding the token's expiration time in a meaningful way (and if the answer is yes, then please clarify how)?
... View more
10-24-2022
09:19 AM
We heard last week that the maintenance update was 'almost' over, but it's not clear to me whether it ever ended or not. In fact, I still get 401 errors which are accompanied by the 'System is under maintenance' message. Could we get an update on this?
... View more
10-19-2022
11:11 AM
Thanks Edina, this certainly clarifies a bit my understanding. I'm still curious to learn about any examples your team can provide though. Thanks again!
... View more
10-18-2022
02:12 PM
I apologize if this question was asked before, but I couldn't find anything relevant online. I understand that soft-deletes are treated as a data 'update' , and therefore checking some attribute in the record (e.g. status or workflow status) should be enough to tell whether the record has been 'soft-deleted' or not. However, hard-deletes are less clear. Specifically, where do they fit in the normal lifecycle of Canvas data? (that is, when do they happen?, how common are they?, should we expect that every soft-deleted record will someday get hard-deleted?). Etc.
... View more
06-30-2022
07:47 AM
1 Kudo
@meghan_lewis , that seems like the right approach to me too. Thanks for sharing your planning process. I might work myself in a nodejs version - if so I'll share it too in case anybody is interested-.
... View more
06-29-2022
12:52 PM
1 Kudo
Thanks @chriscas for your response. However, when I correct for this, and then tried a couple different ways of representing the time zone (such as -05:00, -5:00) . none of them worked. So you definitely have a point about the formatting for the timestamp part (I hadn't realized I was using dashes instead of colons), but I also see that the API seems to be unable to understand an arbitrary time zone and is only accepting the universal timezone (Z). In re-reading the API reference, I see that indeed, the API *only* accepts the universal time zone in these timestamps, and so my inputs were definitely not as expected in more than one way.
... View more
06-29-2022
10:01 AM
Not sure whether this is supposed to be enabled for us yet or not, but I tried earlier today to retrieve an update for the users table using the provided bash script, and the process did not work because the Job_id was being set as null. This is what the output looked like: $ ./update.sh users 2022-06-24T10-37-00-05:00 update_users_06-29-22-10-55-am.json - Connecting to https://dap.insk8s.net - Table 'users', query '{"format":"jsonl","since":"2022-06-24T10-37-00-05:00"}', Canvas account 18760000000000195 - Authenticate in DAP - Initiating update request - Waiting for job null to complete - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null' - Received job state 'null' for job ID 'null'
... View more
- Tags:
- cd2
06-29-2022
08:32 AM
Hi again @meghan_lewis , I've now had a chance to look at the python script you provided above. Thanks for including that; I agree that it's much nicer than the original bash script for purposes of error handling and understanding of what's going on. Have you thought of including an indication of time within the file names that you are saving? I think it would be one way of enabling the running of the incremental data retrievals later on (otherwise, where would the 'since' value come from?). I may have missed it but I did not see this in your filename computation, so I thought I would mention. Otherwise, it would be helpful to understand how you and others are planning to obtain the 'since' value to enable the incremental retrievals.
... View more
06-29-2022
07:57 AM
I'm no expert on retrieving this data @jdh132 but just wanted to point out that these URLs are VERY temporary. Not sure how long they last (we should confirm this with the Docs or the Instructure team) but after they expire I assume they'll be useless. Not sure if this could be affecting you. I was able to retrieve data for the users table by following the URL almost immediately after retrieval BTW.
... View more
06-29-2022
07:51 AM
Thanks @meghan_lewis and @reynlds for your replies to this thread. On top of what has been said already, I believe that the point of the API is simply to enable you to retrieve the data and get it into your local storage, which in turn will enable you to make regular (sql) queries against the tables you create. Either an actual relational DB (perhaps AWS Aurora, Redshift or Athena, sound like they could be good options for this - although I haven't tried any of them yet,( with the exception of a very early trial of Athena by my group many years ago that was not continued I believe due to cost). @reynlds , if you could share the solution you are building for AWS, that would be great. Which brings me to the next question: for you and others who may have experience storing CD1 data on AWS, what is the approximate monthly cost to this? We've stored CD1 data directly in Splunk in the past, but would like to explore also an alternative combination of Splunk + AWS (the first for Canvas Live Events, the second for CD2).
... View more
06-27-2022
08:48 AM
Hi fellow alpha testers! I wonder if any of you has already written a (bash?) script that will grab a snapshot of ALL the tables that are available on the server and output those files either locally or on S3. I'm not an expert in shell scripting and would definitely help if I didn't have to write this myself, so I wonder if somebody has written it already. I was able to successfully run the snapshot_sh.sh file on an individual table BTW.
... View more
- Tags:
- cd2
06-23-2022
12:27 PM
I have a snapshot of data from last September and I wonder whether to start over with fresh snapshot data (assuming the schema has changed due to some re-architecting), or if it is safe to simply get updates. What do you recommend?
... View more
06-23-2022
07:39 AM
1 Kudo
Thanks Edina! For some reason I wasn't finding the blog when I searched, so I didn't have the right info. to get started. I'm finally able to get an auth token now.
... View more
06-22-2022
01:18 PM
(I posted this first as an answer to Edina's initial post - I apologize for the redundancy) Just getting started trying to retrieve our data, but I'm not sure which hostname to use in the URL. Could you please clarify? So far I've tried: https://data-access-platform.canvas.northwestern.edu (using the vanity URL) and also https://data-access-platform.northwestern.instructure.com, but none of these is found by POSTMAN when I try.
... View more
Labels
- Labels:
-
cd2
03-01-2022
01:17 PM
Thanks @igustafson for your input. I actually do have access to another source of information for this type of enrollment data - a database used by our custom SIS synchronizer software- although it would have been quicker to obtain it from live events.
I've received further communication from Instructure (thanks to Kelly Ohmen, our CSM!) that confirms what I had suspected: that there are no 'enrollment drop' live events for SIS-based drops. While the reason is not clear (and unfortunate, I think), that at least answers my question.
... View more
02-18-2022
08:28 AM
I've been looking closely at Canvas Live Events (Caliper format) to find in there enrollment adds and drops, as it would be an ideal place to find that type of information (essentially enrollment transactions). However, after careful inspection, I've only been able to find 'enrollment adds' in there but not drops. To be clearer, I believe there is a chance that 'Modified' live events which refer to enrollment objects as the object of the event, may actually include enrollment drops in there too (as one could think of a particular 'drop' event as a type of modification event performed on an enrollment). However, even if I'm right about this, I see currently no way to distinguish these 'drop' events from regular 'modified' events via the current live event attributes. For those of you who've been working with live events in either format (Caliper or raw), have you been successful at finding this type of information yourselves? If so, could you please share where you've found it?
On the other hand, if an Instructure expert is reading this and can provide an authoritative answer on this question, that would be great too, of course. Thanks in advance.
Quick update after receiving information from Instructure: turns out that there are documentation examples which show how manual enrollment drops are represented within Canvas Live Events (see https://canvas.instructure.com/doc/api/file.data_service_caliper_basic.html). However, in my experience this does not seem to cover SIS-based enrollment drops, so I'm leaving this post open until I get more information from somebody about that case in particular.
... View more
