Canvas Data 2 - New installation GCC Error

[Jo-lCollins]

I'm updating our init and sync code to reflect what is prescribed in the documentation's Instructions and I am receiving the following error when i run pip install

   × Building wheel for tsv2py (pyproject.toml) did not run successfully.
   │ exit code: 1
   ╰─> [26 lines of output]
       compiling with AVX2
       compiling with limited C API
       running bdist_wheel
       running build
       running build_py
       creating build
       creating build/lib.linux-aarch64-cpython-310
       creating build/lib.linux-aarch64-cpython-310/tsv
       copying tsv/helper.py -> build/lib.linux-aarch64-cpython-310/tsv
       copying tsv/__init__.py -> build/lib.linux-aarch64-cpython-310/tsv
       running egg_info
       writing tsv2py.egg-info/PKG-INFO
       writing dependency_links to tsv2py.egg-info/dependency_links.txt
       writing top-level names to tsv2py.egg-info/top_level.txt
       reading manifest file 'tsv2py.egg-info/SOURCES.txt'
       adding license file 'LICENSE'
       writing manifest file 'tsv2py.egg-info/SOURCES.txt'
       copying tsv/parser.pyi -> build/lib.linux-aarch64-cpython-310/tsv
       copying tsv/py.typed -> build/lib.linux-aarch64-cpython-310/tsv
       running build_ext
       building 'tsv.parser' extension
       creating build/temp.linux-aarch64-cpython-310
       creating build/temp.linux-aarch64-cpython-310/lib
       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPy_LIMITED_API=0x03080000 -Ilib -I/tmp/venv6nmns471/include -I/usr/include/python3.10 -c lib/tsv_parser.c -o build/temp.linux-aarch64-cpython-310/lib/tsv_parser.o -fvisibility=hidden -mavx2
       gcc: error: unrecognized command line option ‘-mavx2’
       error: command '/usr/bin/gcc' failed with exit code 1
       [end of output]
 
   note: This error originates from a subprocess, and is likely not a problem with pip.
   ERROR: Failed building wheel for tsv2py
 Failed to build tsv2py
 ERROR: Could not build wheels for tsv2py, which is required to install pyproject.toml-based projects

Some preliminary investigating makes me think the `tsv2py` dependency in incompatible with Mac M1 computers. 

[LeventeHunyadi]

As an immediate work-around, I would suggest setting the environment variable TSV_AVX2=0. The default behavior for the library tsv2py is to compile with AVX2 (Advanced Vector Extensions), which are specific to Intel/AMD CPUs. We are already working on a patch that will auto-detect M1 and other ARM-based CPUs, and omit AVX2 CPU instructions for these platforms.

[DanielMoore]

Good morning, Levente. We're throwing this error on a Windows 10 virtual desktop we connect to via Remote Desktop Connection:

tsv2py has been compiled with AVX2 instruction set enabled but AVX2 is not detected on this machine

I set the environment variable as you suggested, but continue to throw the error. Do you have a different suggestion based on the configuration we are using?

Many thanks in advance.
~Dan

[LeventeHunyadi]

What CPU does the machine have that you are connecting to? If it's an older machine, AVX2 instructions might not be available. This guide helps determine if your Intel CPU has AVX2 instructions.

Setting the environment variable TSV_AVX2=0 only helps in the compile phase when the Python package is installed from a source distribution as in the original post. It has no effect when the package is installed from a binary wheel. By default, pip chooses the binary wheel over the source distribution.

If you pass the option --no-binary to pip, you can force installing from a source distribution:

pip install instructure-dap-client --no-binary "tsv2py"

Don't forget to set the environment variable TSV_AVX2=0 beforehand. The machine on which you are installing has to have a C compiler such as MSVC (Microsoft Visual C++).

Please note that Instructure does not officially support Windows 10, only Windows 11.

[venkat5121]

Hi @LeventeHunyadi 

do you have any suggestion for RHEL ?

i am doing it in .venv

set TSV_AVX2=0

pip install instructure-dap-client --no-binary "tsv2py"

it gives me following error:

error: subprocess-exited-with-error

× Building wheel for tsv2py (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /u01/projects/canvasdata2/.venv/bin/python3 /u01/projects/canvasdata2/.venv/lib64/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmp_jkqotr0
cwd: /tmp/pip-install-02f7ioud/tsv2py_bea04a413efa4bc4b40ee74cc11074e3
Building wheel for tsv2py (pyproject.toml) ... error
ERROR: Failed building wheel for tsv2py
Failed to build tsv2py
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tsv2py)

I am attaching full log here to review.

[LeventeHunyadi]

With the option --no-binary, you are instructing pip to install tsv2py from source (and not any of the available binary wheels) but then you don't have developer tools (e.g. GCC) installed such that pip can compile the C sources into binaries.

If you want to use--no-binary, you must have a working C compiler such as GCC or Clang.

[venkat5121]

Thank you for clarifying this.

my intention is not use --no-binary option.

I am trying to overcome following error that I am getting after I installed instructure-dap-client 1.1.0 in my RHEL server

ERROR: Could not build wheels for tsv2py, which is required to install pyproject.toml-based projects

I found this community post and thought the solution offered to microsoft machines would work for my linux machine too but now I realized that is not the case.

Is there any solution for this error on RHEL server. Will it be there in next patch release? is there a workaround until then?

[LeventeHunyadi]

Linux and Windows machines are not much different w.r.t. --no-binary. If you pass --no-binary, pip won't pull one of the pre-compiled binary wheels but instead try to build the wheel locally from source, which requires a C compiler. On Linux (including RHEL), you would typically use GCC or Clang, on Windows, you would typically use MSVC (Microsoft Visual C++ Compiler). One option, therefore, is to install a compiler with yum on your RHEL server, and build tsv2py from source. This is mentioned in one of the earlier answers.

However, what I would recommend is to use the available binaries, which is the way easier option. tsv2py is compiled for the so-called manylinux target, which works on many Linux distributions, including RHEL 9+. You will need pip version >= 20.3 such that pip picks up the right binary from the available binary wheels. You may want to check if you have the latest version of pip.

[DanielMoore]

Thank you for your feedback, Levente. I'm still throwing the same error, after these checkpoints:

My CPU is:
Intel® Xeon® Gold 6326 Processor

It does have the AVX2 Instruction Set Extension.

I uninstalled dap and reinstalled it using your suggestion to force the install from a source distribution, ensuring that my environment variable was previously set. I do have a C compiler installed.

As a side note, I had been running dap beta successfully, since October, on this same environment. I'd been keeping a mySQL database sync'd twice daily, and dap beta was running nicely. That said, I do realize this is an entirely new dap, and that what had been working previously may not necessarily be a configuration that will be successful now.

Other than the possibility of my Windows OS being an issue; would you have anything else to suggest?

Many thanks, grateful for any additional feedback you might have.
~Dan

[LeventeHunyadi]

I have just run the following commands on a Windows 10 Professional Edition:

set TSV_AVX2=0
pip install instructure-dap-client --no-binary "tsv2py" --no-cache-dir --verbose

When looking at the console output, I am able to spot the following lines, confirming that the package builds from source without AVX2 instructions, which works across all CPU architectures:

Running command Building wheel for tsv2py (pyproject.toml)
compiling without AVX2
compiling with limited C API

Can you confirm you see the same console output?

Note that instructure-dap-client version 1.0 is essentially a different piece of software than version 0.3, the package has been subject to a complete rewrite. While executing an initdb command could previously take as much as 24 hours in version 0.3, this is now reduced to less than an hour in version 1.0 due to optimizations in data transfer, parsing and database connectivity.

[DanielMoore]

Hello and thank you, Levente. I appreciate the clarification in the differences between 0.3 and 1.0, that's helpful.

We're on Windows 10 Enterprise. I uninstalled the dap client, uninstalled Python, then reinstalled Python. Next I ran the commands you show to bypass AVX2; but I did not receive the same console output that you show, however.

I'm attaching my results. Do you see anything there that might help?

Thanks in advance,
~Dan

[LeventeHunyadi]

You need to uninstall tsv2py first:

pip uninstall tsv2py

Otherwise, pip will simply conclude that tsv2py is already installed (with the AVX2-enabled build that fails on your hardware) and won't do anything. This is confirmed by the log output you attached:

Requirement already satisfied: tsv2py>=0.6.1 in c:\users\xxxxx\appdata\local\programs\python\python312\lib\site-packages (from instructure-dap-client) (0.6.3)

[DanielMoore]

Greetings, Levente. Thank you so much, I've been able to successfully install the dap client.

I appreciate your clear direction, guidance, and patience in stepping me through this troubleshooting process.

Best,
~Dan

[DanielMoore]

Greetings once more, Levente. I've been successfully using dap 1.1 on Windows 10 thanks to your help last week. I'd like to install it on a Windows Server 2022 environment, same cpu which is the Intel® Xeon® Gold 6326 Processor

I've successfully installed dap 1.1 on Windows Server 2022, following your helpful guidelines above. But I'm encountering the authentication error that is raised in the following, which in turn references other posts on this topic:

https://community.canvaslms.com/t5/Data-and-Analytics-Group/Canvas-Data-2-authentication-Error/m-p/5...

I've worked through all the suggestions in the above and it's references, but am still throwing the error that others have reported.

dap list --namespace canvas

2024-06-12 17:31:58,178 - ERROR - Cannot connect to host api-gateway.instructure.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')]

Do you have any suggestions, or can I provide more information? Here is everything I can tell you about my build.

dap 1.1.0c
Python 3.12.4
pip 24.0
aiohttp 3.9.5

aws-cli/2.16.4 Python/3.11.8 Windows/10 exe/AMD64
Microsoft Visual C++ 14.38.33130
Microsoft SDK 8.1.23107

OsName : Microsoft Windows Server 2022 Standard
OsVersion : 10.0.20348
OsBuildNumber : 20348
OsHardwareAbstractionLayer : 10.0.20348.2031
WindowsVersion : 2009

I'm not an accomplished Python developer nor a dba, but I'm comfortable using commands and scripts in Python, Powershell, the terminal, and a command prompt. If you can give me some guidance on what else I can check, I'd appreciate it.

Many and continued thanks in advance,
~Dan

[LeventeHunyadi]

I would recommend submitting a support ticket such that Instructure can track this issue.

I have a hunch that the third-party dependency aiohttp is looking for certificates at the wrong place, e.g. using bundled certificates instead of those in the Windows certificate store. A (completely untested) work-around could be to use a package like certifi, and instantiate the SSL context and the TCP connector explicitly:

import ssl
import certifi
from aiohttp import TCPConnector
ssl_context = ssl.create_default_context(cafile=certifi.where())
conn = TCPConnector(ssl=ssl_context)

 You could then pass the connection object conn as the connector parameter of ClientSession:

session = aiohttp.ClientSession(
    connector=conn,
    headers=...,
    timeout=...,
)

I have found two places where ClientSession is instantiated in the code of the DAP client library. Both would need the same work-around.

However, I totally understand that this may be a little too technical given your background. If you open a support ticket, our team could look into how we can incorporate a similar solution to what is shown above into the DAP client library itself.

[DanielMoore]

Thank you, Levente. I've submitted a support ticket.
~Dan

[DanielMoore]

Good morning, Levente. I wanted to let you and the Community at large know that I continued to research the certificate issue and was able to defeat it with this statement:

python -m pip install pip-system-certs --use-feature=truststore

Perhaps this will be helpful to others. My research source:

python - SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: unable to get local issuer certif...

Continued thanks for your guidance. I'll lend this information to the support ticket I have open as well.

~Dan