Uploading files to Canvas via SFTP with passphrase

[ToddDetwiler]

Hi all, 

We are new to Canvas and still involved in the data provisioning phase. I was told that there are multiple ways of uploading files (coming from Banner, in our case), but was under the impression that the SFTP to a Canvas server would be the simplest (no need to deal with throttling, for example). However, I have hit a snag that is really wasting a lot of time. In the case of Canvas, I've been told that I have to use an RSA key for which the private key has a passphrase. With a passphrase, even if I pass the identity file to my SFTP command, it still requires a user to interactively enter the passphrase. What I need is something that can run non-interactively, to automate file uploads.

I thought I'd found a way around this using ssh-agent and ssh-add. The agent did allow me to ssh, using the key, without the passphrase. However it turned out not to be persistent. Based on how you script agent launch, it may last only as long as the session, but certainly not past the next reboot. And then someone has to type in a passphrase again in the ssh-add call.

So, my question is, are others using SFTP without user interaction to upload to Canvas? If so, how are you handling the requirement to enter a passphrase?

[ToddDetwiler]

Someone here just mentioned to me that you may be able to pass a passphrase with the open command to winscp. This reminded me that I needed to mention my environment and my constraints. I am trying to send these from our hosted Banner server. The SFTP tools available are the standard versions found in RedHat. I do not have privileges to install additional tools there. I am just wondering if there is a standard method that I've overlooked for managing passphrases.

[robotcars · Accepted solution]

Hi @ToddDetwiler 

I am not using SFTP. But if you're referring to SIS CSV Imports, you should be able to zip up all files and send them in a single request. There shouldn't be a concern for throttling if you are doing this daily or hourly.

Also, maybe checkout sshpass, and export the password in your bash_profile.

[ToddDetwiler]

Thanks @robotcars , your suggestion to use sshpass was useful. I haven't sorted out yet how to put the passphrase in my bash profile. I've seen examples where they export a password in their profile, but since my "password" is actually a passphrase, I need to include additional arguments to sshpass. I will need to work out how to do that. 

Additionally, and this tip may help others, I was previously using batch mode and passing my SFTP commands (i.e. ls, put, etc.) inline. That does not seem to work with sshpass. I had to extract my SFTP commands to a separate file and set batch mode to "no" but use the "-b" argument to provide the command file. 

[robotcars]

@ToddDetwiler 

I think a simple export should work in ~/.bash_profile, with passphrase in quotes.

export SFTP_PASSPHRASE="all your base"

Then in your shell script

# this is not necessary when you are in session, but would be if you run a cronjob, it allows cron to load settings before the rest of your script
. ~/.bash_profile

# test
# echo "$SFTP_PASSPHRASE"

# try
sshpass -p "$SFTP_PASSPHRASE" sftp -oBatchMode=no "$SFTP_USER$SFTP_HOST" << !
   put $local_path $remote_path
   bye
!

Again, I don't do this with Canvas, so not sure if this gets you there, or just closer.

[ToddDetwiler]

Nice! I can see in your example how I would include the other parameters. It is different than the examples I've seen using the "-e" argument to sshpass to let it know to use a fixed shell variable. I'm unsure if I can mark two posts as the solution (accepted your last post), but I appreciate your help.

Thanks, Todd

PS: Kudos for slipping in a reference to "All Your Base". 😉

[robotcars]

Looks like there's also an argument to get the password directly as SSHPASS

-e The password is taken from the environment variable "SSHPASS".

https://linux.die.net/man/1/sshpass

[ToddDetwiler]

For others attempting this same thing, it looks like you can use other arguments to sshpass in conjunction with the -e to reference a password stored in an environment variable. Here is an example (items inside <> are installation specific):

sshpass -Ppassphrase -e sftp -o BatchMode=no -o IdentityFile=<private_key_url -b <file_containing_remote_script> <user_name>@<server_url>

In the above, the "-Ppassphrase" part tells sshpass to wait for the prompt containing the word passphrase. This is an important part to get sshpass to work with passphrase protected keys instead of user passwords. I only had luck getting it to work (it being sshpass in conjunction with sftp) with BatchMode set to "no", but was still able to provide a remote script via a separate file and using the "-b" argument.  As you can see, I can pass all of the other arguments I want to sshpass, but the "-e" argument instructs it to look for an environment variable named SSHPASS and to use that where you would normally use the "-p" (lowercase) argument.