CD2 Connection reset by peer error

[SamVinh]

Starting on Friday, 9/15/2023, I encountered the following error: aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api-gateway.instructure.com:443 ssl:default [Connection reset by peer].

This error has occurred with every download attempt and it errors out immediately, so no job is being created. Since typically the first log message to indicate that a job has been created is the "Query started with job ID: ..." but that is not the case here. Prior to Friday, I have never encountered this problem using the same scripts.

Here is the log:

```

2023-09-18.log
Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1112, in create_connection
transport, protocol = await self._create_connection_transport(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1145, in _create_connection_transport
await waiter

File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 970, in _read_ready__get_buffer
nbytes = self._sock.recv_into(buf)
^^^^^^^^^^^^^^^^^^^^^^^^^

ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/u10/etl_data/oracle/sqlloader/datawarehouse/webcampus/python/src/unlv_dap/cli.py", line 181, in <module>
cli()

File "/u10/etl_data/oracle/sqlloader/datawarehouse/webcampus/python/src/unlv_dap/cli.py", line 129, in snapshot
result = asyncio.run(unlv_dap_client.snapshot_query_table(table=table, namespace=namespace))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^

File "/u10/etl_data/oracle/sqlloader/datawarehouse/webcampus/python/src/unlv_dap/unlv_dap_client.py", line 362, in snapshot_query_table
result = await session.download_table_data(namespace, table, query, packed_output_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 560, in download_table_data
job = await self.execute_job(namespace, table, query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 532, in execute_job
job = await self.query_snapshot(namespace, table, query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 350, in query_snapshot
job = await self._post(f"/dap/query/{namespace}/table/{table}/data", query, Job) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 261, in _post
await self.authenticate()

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 338, in authenticate
properties = await self._post_auth_request(self._credentials.basic_credentials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/dap/api.py", line 283, in _post_auth_request
async with self._session.post(

File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
self._resp = await self._coro
^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
raise last_exc

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api-gateway.instructure.com:443 ssl:default [Connection reset by peer]

```

[SamVinh · Accepted solution]

After investigating with members of our System Engineering and Network Engineering team, we found that it was not the api-gateway that caused this issue. It was a firewall migration that affected our Canvas Data 2 testing environment and production environment as the servers did not get updated to accept traffic from Canvas on port 443.