The Instructure Community is in read-only mode as we prepare to migrate to our new Community platform in early December.
Community members cannot post or update content in the Community during this time. All content is still available to view.
Read our blog post for more info about this change.Found this content helpful? Log in or sign up to leave a like!
Got this little ditty today:
2025-01-22 10:05:05,294 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: JcMid_eNLgZT_7qI2WBseNq4zJlRhXRajHT5Rlm1AUrGwdO6FP9m2A==
Along with the "malformed HTTP response" errors, I'm receiving these on a regular basis:
- ERROR - table not initialized, use `initdb`
- ERROR: The request could not be satisfied
- ERROR - invalid_client
My target DB is a horribly over-resourced PG instance in AWS/RDS and I'm using DAP client ver. 1.3.1 on Python 3.12.5. These errors have only coming fast 'n furious in the last few weeks. Before that it's been smooth sailing. Thoughts?
Timeouts to the service and other errors continue.
Seeing WAY too many of these lately:
"ERROR - An exception occurred while executing the command: DAP Querying service is overloaded or under maintenance"
What's going on with this service as it doesn't appear to be scaling to meet need. What are are the published maintenance times for DAP??
We're getting a bunch of "ERROR - malformed HTTP response" lately. In the past it meant an unusual amount of traffic..
@sgergely Any response to these performance/networking issues we're seeing???
Hello, thank you for the mention @reynlds! Yes, we are seeing unusual amount of traffic and we are working on it. Sorry for the issues it causes.
@sgergely here are summaries of the most recent errors:
025-02-21 06:05:33,902 - ERROR - malformed HTTP response:
2025-02-21 06:05:33,919 - ERROR - malformed HTTP response
2025-02-21 06:05:48,928 - ERROR - malformed HTTP response:
2025-02-21 06:05:48,946 - ERROR - malformed HTTP response
2025-02-21 06:06:07,508 - ERROR - malformed HTTP response:
2025-02-21 06:06:07,526 - ERROR - malformed HTTP response
2025-02-21 06:07:12,645 - ERROR - malformed HTTP response:
2025-02-21 06:07:12,661 - ERROR - malformed HTTP response
2025-02-21 06:07:16,024 - ERROR - malformed HTTP response:
2025-02-21 06:07:16,043 - ERROR - malformed HTTP response
2025-02-21 06:23:09,501 - ERROR - malformed HTTP response:
2025-02-21 06:23:09,519 - ERROR - malformed HTTP response
2025-02-21 00:17:31,592 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:17:31,609 - ERROR - malformed HTTP response
2025-02-21 00:25:24,502 - ERROR - malformed HTTP response:
2025-02-21 00:25:24,516 - ERROR - malformed HTTP response
2025-02-21 00:40:15,464 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:40:15,481 - ERROR - malformed HTTP response
2025-02-21 00:41:49,074 - ERROR - malformed HTTP response:
<TITLE>ERROR: The request could not be satisfied</TITLE>
<H1>504 ERROR</H1>
2025-02-21 00:41:49,090 - ERROR - malformed HTTP response
Just for me to understand your current situation: can you get any data through CD2 and there are issues time-to-time or nothing comes through because everything fails?
I know exactly which tables are failing from the above error messages. I take that and do a manual syncdb, and the next automated one (which has been in place for years) will fail with a variety of errors. it looks like most of the failures happen overnight, but once a malformed http response happens, it requires manual intervention on the table(s). Having our Db out of sync with CD2 is not good.
Since this last post things have gotten markedly better. However, I can't claim that the service becoming "stable" is due to this post, so I'm not holding my breath that it will stay this way. All we can do is monitor and report, which is what I'll continue doing.
Has it been better already? I have been checking our logs and since last week and we haven't seen any issues. If you have can you please help me with details?
@sgergely Latest errors from this morning. FYI...I only log "errors", not "debug". Also, I wrap the log entries from DAP with some extra to show things like the command used, etc.:
==================================================
=== Processing table: content_participation_counts
= 2025-03-05 06:15:15
= Starting syncdb: content_participation_counts
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table content_participation_counts
2025-03-05 06:15:46,926 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: kfDFvzEfFavJ3FNgRhktP1Mvlfmj-Zt_fs1slgeFY4O5_JsX2_3pEQ==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:15:46,942 - ERROR - malformed HTTP response
==================================================
=== Processing table: course_sections
= 2025-03-05 06:22:12
= Starting syncdb: course_sections
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table course_sections
2025-03-05 06:22:43,528 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: GrM5NnjG_JK44tSb-51TSl6syfly-i8Z0tofY2qUC5ujVRLZ9_YEzw==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:22:43,547 - ERROR - malformed HTTP response
==================================================
=== Processing table: learning_outcome_question_results
= 2025-03-05 06:38:57
= Starting syncdb: learning_outcome_question_results
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table learning_outcome_question_results
2025-03-05 06:39:29,158 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: jOkN0XtqJv4UPjJFabk2N6O9oX5MsF1axYu8YIDI2jRNLWHUhHtSyQ==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:39:29,175 - ERROR - malformed HTTP response
==================================================
=== Processing table: quizzes
= 2025-03-05 06:49:04
= Starting syncdb: quizzes
= Command used: dap --loglevel error --logfile /var/lib/canvas-mgmt/logs/cd2/2025-03-05_06:05:01_syncdb.log syncdb --namespace canvas --table quizzes
2025-03-05 06:49:35,408 - ERROR - malformed HTTP response:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: LBtIiw4p0MuQmX3BEf4imwe_Yn4fPXQkWVHnIpd-mvRQ50vQaIPebA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
2025-03-05 06:49:35,426 - ERROR - malformed HTTP response
@sgergely my noon run of syncdb on the following 3 tables failed with the exact same errors as above:
assignment_overrides
context_module_progressions
master_courses_master_migrations
Performing a manual syncdb works and moves things forward, but it should never be expected that a subscribing institution should have to do this multiple times a day (or even once a day or week)
Community helpTo interact with Panda Bot, our automated chatbot, you need to sign up or log in:
Sign inTo interact with Panda Bot, our automated chatbot, you need to sign up or log in:
Sign in
