Scaling tests for the last three assignments in the course ( PS6 , PS7 , and the Final ) will be done using a server instance on AWS.
These machines will have substantially more cores (16 to 32) than your laptop and can therefore support better scaling studies. Note however that these machines are shared resources. For PS6 and PS7 you should be able to complete the assignments on a shared instance. But, be considerate of your fellow students and try to limit the size and duration of your runs while you are testing and debugging.
One useful workflow would be to do your initial development, testing, and debugging on your laptop and then switch to the AWS instance for final
Currently, the IP address for the instance is 35.162.140.108. It also has DNS name ec2-35-162-140-108.us-west-2.compute.amazonaws.com.
The IP address of the instance we are using may change and if we add instances, those will have different IP addresses. I will keep a current list of instance IP addresses on Piazza and on Canvas.
An account has been made for you on the instance, with the same name as your UW netid. Accessing the instance will be done via ssh using publickey encryption (no passwords). Sometime on or about Friday May 24, I will be sending out invitations to each student in the course to acess two files: <netid>_id_rsa and <netid>.pub – this is your private and public ssh key pair for accessing the AWS instances for this course. Note that even if the instance IP addresses change, you will not need a new key pair.
Using ssh is much more secure than using passwords – and, in fact, passwords are disabled altogether on the instances; you can only connect with ssh. The ssh client-server system fills two roles for us. First, it provides a secure mechanism to authenticate – to establish your identity to the system so that you can login and access the privileges associated with your account. Second, ssh encrypts the transmission between your client and the server you connect to (in this case, the instance).
Ssh is built on public-private key cryptography. Public-private key cryptography is asymmetric – meaning one key is used to encrypt data and another is used to decrypt it. One of the keys is denoted the public key and one the private key. The keys are strings of bits (and related to each other) such that the public key can be used to encrypt a message and the private key can be used to decrypt it. The public key can be widely disseminated and anyone can use it to encrypt a message. But, the holder of the private key does not share that – and only that key can be used to decrypt the message.
There are two parts to connecting with ssh: importing your private key into your ssh client and then using your ssh client to connect to one of the hosts above. The process is essentially the same for Mac OS, WSL, and Linux. (I have already copied your public key to your AWS directory.)
I have my public-private keys set up in the following way. In my home directory there is a subdirectory .ssh:
drwx------ 49 al75 faculty 1568 Nov 26 11:33 .ssh
Note that the permissions of the subdirectory are “700” – only the owner of the subdirectory (in this case, user al75) can access, read from, or write to the directory. Within that subdirectory are the two key-pair files:
-rw-r--r-- 1 al75 faculty 397 Jun 4 22:17 .ssh/al75.pub
-rw------- 1 al75 faculty 1675 Jun 4 22:17 .ssh/al75_id_rsa
Note again that the private key al75_id_rsa is only readable and writable by the
owner. If your subdirectory or your private key have different permissions than
these, your ssh client will not accept them.
To set up your keys to use with ssh, firt create a .ssh subdirectory under your home directory if you do not already have one.
$ cd
$ mkdir .ssh
Then change the permissions to “700” for the .ssh subdirectory
$ chmod 700 .ssh
Copy or move the keys I provided to you into the .ssh subdirectory
$ cp </location/of/keys> .ssh
And, finally, set the permissions of the keys
$ chmod 600 .ssh/<netid>_id_rsa
$ chmod 644 .ssh/<netid>.pub
To connect to the AWS instance:
ssh -i ~/.ssh/<netid>_id_rsa -l <netid> 35.162.140.108
The -i option specifies the private key to use and the -l option specifies the user id on the target machine (35.162.140.108).
You can shortcut some of this by putting necessary information into a .ssh/config file (which does not need any special permissions):
host aws
HostName 35.162.140.108
User al75
IdentityFile ~/.ssh/al75_id_rsa
(And, of course, use your own id rather than al75.)
If you have this information in your .ssh/config file, you can connect simply with
ssh aws
and the client program will grab the hostname, identity file, and user id from the config file. And, if/when the instance IP address changes, you just need to edit this file rather than try to remember the IP addresses.
Now, when you have connected to the instance in this way, you will have a text-based shell on that machine. I will have already copied the necessary starting files to your home directory for each assignment.
The Visual Studio Code Insiders application (distinct from Visual Studio Code) has an available extension that allows you to connect directly to a remote host via ssh and transparently work on the remote host using the app on your local machine. This is probably the best workflow for using AWS.
I have posted short walkthrough on installing and connecting with VS Code Insiders here: [here.] (https://uw.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=5013bc01-e865-4a82-aa01-aa58003c097c)
If you connect to the instance with just a text-based shell, you will have to edit the remote files using a text-based editor such as emacs, vi, or nano. If you have become accustomed to using Visual Studio Code for this coure, this may be awkward.
One way to use Visual Studio Code (which has been installed on the instance), is to connect to the instance and enable X11 forwarding – and you will need to be running X11 on your local machine (XQuartz on Mac OS or MobaXterm on Windows). Forwarding is enabled by adding a -X flag to the ssh command. In addition, if you are using X11 forwarding you should also specify compression with the -C option:
ssh -i ~/.ssh/al75_id_rsa -l al75 -X -C 35.162.140.108
or
ssh -X -C aws
Even with compression, the responsiveness will not be great, but should be usable. I explored a number of other remote desktop options – which generally have better responsiveness – but there were other issues in making that work. Primarily there is not a straightforward way of using public-private key encryption for authentication.
Ssh also provides the command “scp” (secure copy) to copy files back and forth from a remote host.
To copy a file to AWS
$ scp -i ~/.ssh/al75_id_rsa local_name al75@35.162.140.108:remote_name
Similarly, to copy a file from AWS:
$ scp -i ~/.ssh/al75_id_rsa al75@35.162.140.108:remote_name local_name
The local and remote names can include directory paths.
One very handy command for keeping two directories synchronized is “rsync” – which also uses ssh for authentication and transport.
Assume your ps6 files are in your directory amath583/assignments/ps6. To synchronize your local ps6 with the AWS ps6:
$ cd amath583/assignments
$ rsync -abvz ps6/ aws:ps6/
Note the trailing slash in both the local and remote names
Note also that you can’t specify the identity file with rsync – so you should consider using the config file (it is much more convenient anyway).
Rsync is not a bulk copy. It will check the local files against the remote files and only copy files that are different or that are new.
You can synchronize from AWS to your local machine by reversing the arguments
$ cd amath583/assignments
$ rsync -abvz aws:ps6/ ps6/