2024-03-25 15:24:56,238 - DEBUG - Checking for valid database connection string (abstract_db_command.py:14) 2024-03-25 15:24:56,238 - DEBUG - Checking connection to database (abstract_db_command.py:23) 2024-03-25 15:24:56,378 - DEBUG - Client region: us-east-1 (api.py:114) 2024-03-25 15:24:56,379 - DEBUG - initializing table: canvas.accounts (sql.py:31) 2024-03-25 15:24:56,557 - DEBUG - No version records found in "instructure_dap"."database_version" (db_version_upgrader.py:67) 2024-03-25 15:24:56,558 - DEBUG - Current version: 0; Latest version: 1 (db_version_upgrader.py:40) 2024-03-25 15:24:56,558 - DEBUG - Upgrading database (db_version_upgrader.py:45) 2024-03-25 15:24:56,607 - DEBUG - Running upgrade scripts from version 0 to 1 (db_version_upgrader.py:81) 2024-03-25 15:24:56,615 - DEBUG - Upgrading from version 0 to 1 (db_version_upgrader.py:85) 2024-03-25 15:24:56,616 - DEBUG - Running upgrade script: C:\Users\maf11\AppData\Local\Programs\Python\Python312\Lib\site-packages\dap\version_upgrade\postgresql_0_to_1.sql (db_version_upgrader.py:89) 2024-03-25 15:24:56,620 - DEBUG - Executing SQL: DO $$ DECLARE ns VARCHAR[]; BEGIN ns := ARRAY ['canvas', 'catalog', 'canvas_logs']; EXECUTE ('TRUNCATE TABLE instructure_dap.table_sync'); FOR i IN 1..ARRAY_LENGTH(ns, 1) LOOP IF EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema = ns[i] AND table_name = 'dap_meta') THEN EXECUTE FORMAT(' INSERT INTO instructure_dap.table_sync ( source_namespace, source_table, timestamp, schema_version, target_schema, target_table, schema_description_format, schema_description ) SELECT namespace, source_table, timestamp, schema_version, target_schema, target_table, schema_description_format, schema_description FROM %I.dap_meta', ns[i]); EXECUTE FORMAT('ALTER TABLE %I.dap_meta RENAME TO dap_meta_backup', ns[i]); ELSE RAISE NOTICE 'Table %.dap_meta does not exist.', ns[i]; END IF; END LOOP; END $$; (db_version_upgrader.py:93) 2024-03-25 15:24:56,653 - DEBUG - fetching schema for table: canvas.accounts (sql_op.py:103) 2024-03-25 15:24:56,653 - DEBUG - Get schema of table: accounts (api.py:428) 2024-03-25 15:24:56,654 - DEBUG - Authenticating to DAP in region us-east-1 (api.py:371) 2024-03-25 15:24:57,222 - DEBUG - GET/POST response payload: {'schema': {'$schema': 'https://json-schema.org/draft/2020-12/schema', 'type': 'object', 'properties': {'meta': {'type': 'object', 'properties': {'ts': {'type': 'string', 'format': 'date-time'}, 'action': {'type': 'string', 'enum': ['U', 'D'], 'title': 'Identifies if a record is to be upserted (inserted or updated) or (hard) deleted.'}}, 'additionalProperties': False, 'required': ['ts'], 'title': 'Meta-information associated with the record.'}, 'key': {'type': 'object', 'properties': {'id': {'type': 'integer', 'format': 'int64', 'description': 'The ID of the Account object.'}}, 'additionalProperties': False, 'required': ['id'], 'title': 'Stores data about account objects in the Canvas system.', 'description': 'Accounts are most often used to represent a hierarchy of colleges, schools, departments, campuses.'}, 'value': {'type': 'object', 'properties': {'name': {'type': 'string', 'maxLength': 255, 'description': 'The display name of the account.'}, 'deleted_at': {'type': 'string', 'format': 'date-time', 'description': 'Timestamp of when the account was deleted. Will only ever be NULL for end customers.'}, 'parent_account_id': {'type': 'integer', 'format': 'int64', 'description': "The account's parent ID, or NULL if this is the root account."}, 'current_sis_batch_id': {'type': 'integer', 'format': 'int64', 'description': 'The ID of the currently processing SIS (Student Information System) batch (if submitted via UI, not API).'}, 'storage_quota': {'type': 'integer', 'format': 'int64', 'description': 'The storage quote for the account, in megabytes.'}, 'default_storage_quota': {'type': 'integer', 'format': 'int64', 'description': 'The storage quota for children accounts in megabytes, if not otherwise specified.'}, 'default_locale': {'type': 'string', 'maxLength': 255, 'description': 'Language for the account.'}, 'default_user_storage_quota': {'type': 'integer', 'format': 'int64', 'description': 'The default storage quota for users in the account in megabytes, if not otherwise specified.'}, 'default_group_storage_quota': {'type': 'integer', 'format': 'int64', 'description': 'The storage quota for a group in the account in megabytes, if not otherwise specified.'}, 'integration_id': {'type': 'string', 'maxLength': 255, 'description': "The account's identifier in the Student Information System."}, 'lti_context_id': {'type': 'string', 'maxLength': 255, 'description': 'UUID of the Canvas context in LTI standard. Secondary ID for this context, could be used in API to identify resource as well.'}, 'consortium_parent_account_id': {'type': 'integer', 'format': 'int64', 'description': 'The root account of the consortium account, if this root account is part of a consortium.'}, 'course_template_id': {'type': 'integer', 'format': 'int64', 'description': 'The course selected as a template for new courses created in this account. 0 if a template should not be used, nor inherited.'}, 'created_at': {'type': 'string', 'format': 'date-time', 'description': 'Timestamp of when the account was created.'}, 'updated_at': {'type': 'string', 'format': 'date-time', 'description': 'Timestamp of when the account was updated.'}, 'workflow_state': {'type': 'string', 'enum': ['__dap_unspecified__', 'active', 'deleted', 'suspended'], 'description': 'Life-cycle state for account.'}, 'default_time_zone': {'type': 'string', 'maxLength': 255, 'description': 'The default time zone of the account. Allowed time zones are [IANA time zones](https://www.iana.org/time-zones) or friendlier [Ruby on Rails time zones](https://api.rubyonrails.org/classes/ActiveSupport/TimeZone.html).'}, 'uuid': {'type': 'string', 'maxLength': 255, 'description': 'The UUID of the account.'}, 'sis_source_id': {'type': 'string', 'maxLength': 255, 'description': 'Correlated id for the record for this course in the SIS system (assuming SIS integration is configured)'}}, 'additionalProperties': False, 'required': ['created_at', 'updated_at', 'workflow_state'], 'title': 'Stores data about account objects in the Canvas system.', 'description': 'Accounts are most often used to represent a hierarchy of colleges, schools, departments, campuses.'}}, 'additionalProperties': False, 'required': ['key']}, 'version': 2} (api.py:351) 2024-03-25 15:24:57,432 - DEBUG - Start operation - initialize. Namespace: canvas, table: accounts, conn: (sql_op_init.py:49) 2024-03-25 15:24:57,446 - DEBUG - fetching meta-data about table accounts (sql_op_init.py:53) 2024-03-25 15:24:57,450 - DEBUG - fetching data for table from DAP API (sql_op_init.py:58) 2024-03-25 15:24:57,450 - DEBUG - Query snapshot of table: accounts. Query: SnapshotQuery(format=, mode=) (api.py:391) 2024-03-25 15:24:57,451 - DEBUG - POST request payload: {'format': 'tsv', 'mode': 'condensed'} (api.py:304) 2024-03-25 15:24:57,810 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'waiting', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:24:57,811 - INFO - Query started with job ID: c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 2024-03-25 15:24:57,812 - INFO - Query job still in status: waiting. Checking again in 5 seconds... 2024-03-25 15:25:02,815 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:03,080 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'running', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:25:03,081 - INFO - Query job still in status: running. Checking again in 5 seconds... 2024-03-25 15:25:08,086 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:08,380 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'running', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:25:08,381 - INFO - Query job still in status: running. Checking again in 5 seconds... 2024-03-25 15:25:13,388 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:13,633 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'running', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:25:13,634 - INFO - Query job still in status: running. Checking again in 5 seconds... 2024-03-25 15:25:18,636 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:18,819 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'running', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:25:18,821 - INFO - Query job still in status: running. Checking again in 5 seconds... 2024-03-25 15:25:23,824 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:23,995 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'running', 'expires_at': '2024-03-26T19:24:57Z'} (api.py:351) 2024-03-25 15:25:23,996 - INFO - Query job still in status: running. Checking again in 5 seconds... 2024-03-25 15:25:29,000 - DEBUG - Retrieving job state for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:461) 2024-03-25 15:25:29,434 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'complete', 'objects': [{'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz'}], 'expires_at': '2024-03-26T19:24:57Z', 'schema_version': 2, 'at': '2024-03-25T18:51:44Z'} (api.py:351) 2024-03-25 15:25:29,435 - DEBUG - Query job finished with status: complete (api.py:620) 2024-03-25 15:25:29,438 - DEBUG - Retrieving object IDs for job c8ca2a40-a09c-4c86-8bd5-6cc4443a3320 (api.py:478) 2024-03-25 15:25:29,755 - DEBUG - GET/POST response payload: {'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320', 'status': 'complete', 'objects': [{'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz'}], 'expires_at': '2024-03-26T19:24:57Z', 'schema_version': 2, 'at': '2024-03-25T18:51:44Z'} (api.py:351) 2024-03-25 15:25:29,756 - INFO - Data has been successfully retrieved: {"id": "c8ca2a40-a09c-4c86-8bd5-6cc4443a3320", "status": "complete", "expires_at": "2024-03-26T19:24:57Z", "objects": [{"id": "c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz"}], "schema_version": 2, "at": "2024-03-25T18:51:44Z"} 2024-03-25 15:25:29,762 - DEBUG - Retrieve resource URLs for objects: (api.py:487) 2024-03-25 15:25:29,763 - DEBUG - ['c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz'] (api.py:488) 2024-03-25 15:25:29,764 - DEBUG - POST request payload: [{'id': 'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz'}] (api.py:304) 2024-03-25 15:25:30,057 - DEBUG - GET/POST response payload: {'urls': {'c8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz': {'url': 'https://data-access-platform-output-prod-iad.s3.amazonaws.com/output/rootAccountId%3DWxEkP3KbeKvs2QRKgR0zMx3yEbV8cBhjevC6PPdU/queryId%3Dc8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz?X-Amz-Security-Token=FwoGZXIvYXdzEI3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDOFd6V2f9KXzMedn2iK7AaAlyIDbdd%2BE8OUf%2Bx2ELFEZelcPzR77SQlK0ZtQ6VfWRmJjAwhLD59VmlvioG0PWlgoYY%2BwSlE%2FshgkLTYWu0H9VcZDK7Fk8A8Xvp1FqfC0uN1KZkePfKV51XeeNA%2BXvMAAPlvAuOnwJi13Uvbsf751rfWCtBGqZc7IJ9PILUPglZKQSCjC%2BHAOguOS5NeqX0BC%2BzbkYGeCVuarqx9eRu3oIHNGuupnoVhn%2FmwkWY05%2FI3pT5E%2B%2BzOaybco0Z6HsAYyLdBQjjVaJu2hsEsBVcf4IMY3%2Fv57ToSjg07KKYS1%2B6usdI6YEDPyQ6xyEoMC5w%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240325T192530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIAXX2PINZLEZWLL5QJ%2F20240325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=37673e3ad50182bfe2999b644bcddbc5e5e02343e7dd9257bddc86e59dd7c4de'}}} (api.py:351) 2024-03-25 15:25:30,059 - DEBUG - Downloading: https://data-access-platform-output-prod-iad.s3.amazonaws.com/output/rootAccountId%3DWxEkP3KbeKvs2QRKgR0zMx3yEbV8cBhjevC6PPdU/queryId%3Dc8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz?X-Amz-Security-Token=FwoGZXIvYXdzEI3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDOFd6V2f9KXzMedn2iK7AaAlyIDbdd%2BE8OUf%2Bx2ELFEZelcPzR77SQlK0ZtQ6VfWRmJjAwhLD59VmlvioG0PWlgoYY%2BwSlE%2FshgkLTYWu0H9VcZDK7Fk8A8Xvp1FqfC0uN1KZkePfKV51XeeNA%2BXvMAAPlvAuOnwJi13Uvbsf751rfWCtBGqZc7IJ9PILUPglZKQSCjC%2BHAOguOS5NeqX0BC%2BzbkYGeCVuarqx9eRu3oIHNGuupnoVhn%2FmwkWY05%2FI3pT5E%2B%2BzOaybco0Z6HsAYyLdBQjjVaJu2hsEsBVcf4IMY3%2Fv57ToSjg07KKYS1%2B6usdI6YEDPyQ6xyEoMC5w%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240325T192530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIAXX2PINZLEZWLL5QJ%2F20240325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=37673e3ad50182bfe2999b644bcddbc5e5e02343e7dd9257bddc86e59dd7c4de to C:\Users\maf11\AppData\Local\Temp\tmp8t10k3yy\part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv (api.py:535) 2024-03-25 15:25:30,207 - DEBUG - Download complete of https://data-access-platform-output-prod-iad.s3.amazonaws.com/output/rootAccountId%3DWxEkP3KbeKvs2QRKgR0zMx3yEbV8cBhjevC6PPdU/queryId%3Dc8ca2a40-a09c-4c86-8bd5-6cc4443a3320/part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv.gz?X-Amz-Security-Token=FwoGZXIvYXdzEI3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDOFd6V2f9KXzMedn2iK7AaAlyIDbdd%2BE8OUf%2Bx2ELFEZelcPzR77SQlK0ZtQ6VfWRmJjAwhLD59VmlvioG0PWlgoYY%2BwSlE%2FshgkLTYWu0H9VcZDK7Fk8A8Xvp1FqfC0uN1KZkePfKV51XeeNA%2BXvMAAPlvAuOnwJi13Uvbsf751rfWCtBGqZc7IJ9PILUPglZKQSCjC%2BHAOguOS5NeqX0BC%2BzbkYGeCVuarqx9eRu3oIHNGuupnoVhn%2FmwkWY05%2FI3pT5E%2B%2BzOaybco0Z6HsAYyLdBQjjVaJu2hsEsBVcf4IMY3%2Fv57ToSjg07KKYS1%2B6usdI6YEDPyQ6xyEoMC5w%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240325T192530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIAXX2PINZLEZWLL5QJ%2F20240325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=37673e3ad50182bfe2999b644bcddbc5e5e02343e7dd9257bddc86e59dd7c4de to C:\Users\maf11\AppData\Local\Temp\tmp8t10k3yy\part-00000-e8082bf6-f229-4488-911e-04d8e22e6134-c000.tsv (api.py:546) 2024-03-25 15:25:30,208 - DEBUG - init: insert data from resources saved to disk into database table (sql_op_init.py:87) 2024-03-25 15:25:30,212 - DEBUG - processing file 1 of 1 (sql_op_init.py:100)