Before starting

Before diving into this project there are some skills and software that are needed.

Required skills

This project documentation assumes some basic skills using the command line and the version-control tool, Git. By “command line,” we mean a Unix-like shell environment. For this documentation, we assume you are using the Bash shell specifically.

For learning basic Bash skills, the lab has a video to help:

Also, Software Carpentry has a nice lesson:

For learning basic Git skills, the lab has a series of videos:

Also, Software Carpentry has a nice lesson for that as well:

If you are a member of the Phyletica Lab, you will be doing most of the work for the project on Auburn University’s Hopper cluster. The guide for using the Hopper cluster is here:

Required software

Unix-like shell environment

As alluded to above in the required-skills section, you will need to have access to a Unix-like shell environment (or Unix-like “command line”). If you are not sure what this is, please try out this Software Carpentry lesson.

Folks in the Phyletica Lab will be using Bash on Auburn University’s Hopper cluster.

Git

This project requires Git. If you want to contribute to the project, you should also have the Git Large File Storage extension to Git. Git LFS is not required to replicate our analyses, but Phyletica Lab members will be using it to add large archives of simulation files to the project repository.

Python

Several Python scripts are used in this project, and we provide a shell script that allows you to easily set up a Python virtual environment for these scripts to work successfully.

C++ compiler

This project requires CMake and a new-ish C++ compiler (one that supports the C++11 standard). This are needed to compile ecoevolity.

Getting set up

This section is intended specifically for Phyletica Lab members to get their accounts on the Hopper cluster set up for working on this project. If you are not in the lab, some of the instructions below might still be useful for setting up your own system to work on this project.

Setting up Hopper account

Before logging into Hopper

Setting up 2-factor authentication

To log in to Hopper you will need to have Duo 2-factor authentication set up. Go to https://duo.auburn.edu/ for how to set up Duo. Also, you will need to have your AU 2-factor authentication configured to send you a Duo request by default. Go to https://auburn.edu/2factor and on the left side of the screen, click on “My settings & Devices.” For the “When I log in” setting, choose “Automatically send this device a Duo Push.”

Setting up AU VPN

To access Hopper from off of AU’s campus, you will need to be connected to the AU virtual private network (VPN). Here’s a guide for how to set up the VPN so that you can log in from off campus. See the section below for an altnernative to using VPN to access Hopper from off-campus.

Logging into Hopper

Once you have an account on the Hopper cluster, you can log in from the command line using (please read the notes below before you try this for the first time):

ssh YOUR-AU-USERNAME@hopper.auburn.edu

Note

After entering this command, you will get asked to enter the password associated with your AU account.

Note

The first time you log in you will receive a warning stating something like “the authenticiy of the host cannot be established.” This is normal; it just means that the IP address of the server you are logging in to is new and hasn’t been added to the known_hosts file on your computer. Simply type yes and hit enter to log in.

To log out of Hopper, simply type:

exit

Setting up passwordless login

We can log in to Hopper more securely and without needing to enter our AU password. To do this, we will use SSH authentication keys. Below is a short video about SSH keys followed by instructions on how to create and use them.

To see if you have SSH keys on your computer (the computer you use to log in to Hopper), open your terminal and enter:

ls ~/.ssh

If you see id_rsa and id_rsa.pub listed, you already have a pair of SSH keys. If you don’t see these listed (or the .ssh directory does not exist), you need to create a pair of SSH keys. To do this, we will use ssh-keygen. When you enter this on the command line, it will prompt you with some options; simply hit enter for each prompt to accept the default (note, if you already have a pair of SSH keys, skip this step):

ssh-keygen

Now, when you:

ls ~/.ssh

You should see the files id_rsa and id_rsa.pub listed.

id_rsa

This is the private component of your SSH key pair; do not share it with anyone.

id_rsa.pub

This is the public component of your SSH key pair and can be safely shared.

Now, we need to put the contents of id_rsa.pub into the .ssh/authorized_keys file in your home directory on Hopper. We can do this with one command entered on your computer:

cat ~/.ssh/id_rsa.pub | ssh YOUR-AU-USERNAME@hopper.auburn.edu "cat >> ~/.ssh/authorized_keys"

Now, after you start a fresh shell session, you should be able to log in to Hopper without entering your password.

Setting up alternative to VPN

Everything above about setting up Hopper assumes you were either on AU’s campus, or were connected to the AU VPN. AU’s VPN is … let’s just say, not always great. If you are having trouble with the VPN and need to access Hopper from off campus, we can set up a workaround. To do this you will need an account on AU’s Venus Linux server.

Once you have an account on Venus, you can log in via:

ssh YOUR-AU-USERNAME@venus2.auburn.edu

When logged in to Venus, check to see if you have SSH keys:

ls ~/.ssh

If you see id_rsa and id_rsa.pub listed, you already have a pair of SSH keys. If you don’t see these listed (or the .ssh directory does not exist), you need to create a pair of SSH keys, like we did above, using ssh-keygen. Enter (make sure you are logged in to Venus for this):

ssh-keygen

and hit enter at all the prompts to accept the defaults. Now, when you:

ls ~/.ssh

You should see the files id_rsa and id_rsa.pub listed. Now, we need to add the content of your public SSH key on Venus to your authorized_keys file on Hopper (again, make sure you are logged in to Venus for this):

cat ~/.ssh/id_rsa.pub | ssh YOUR-AU-USERNAME@hopper.auburn.edu "cat >> ~/.ssh/authorized_keys"

Now, log out of Venus using:

exit

so that you are back on your computer. Now, we need to add the content of your public SSH key on your computer to the authorized_keys file on Venus (make sure you run this command on your computer):

cat ~/.ssh/id_rsa.pub | ssh YOUR-AU-USERNAME@venus2.auburn.edu "cat >> ~/.ssh/authorized_keys"

Next, we need to set up an SSH configuration file on your computer. So, on your machine enter:

touch ~/.ssh/config

This will create the file ~/.ssh/confg if it doesn’t exist, and does nothing if it already exists. Next, open the file ~/.ssh/config with a “raw” text editor. One option is to edit this file using nano:

nano ~/.ssh/config

Add the following content to the file, replacing YOUR-AU-USERNAME with your AU username:

Host venus
    HostName    venus2.auburn.edu
    User        YOUR-AU-USERNAME
Host hopper
    HostName    hopper.auburn.edu
    User        YOUR-AU-USERNAME
    ProxyJump   venus

After saving this content to your SSH config file, and starting a fresh shell session, you should be able to log in to Hopper using:

ssh hopper

This command will use “SSH tunnelling” to log you in to Hopper by using Venus as an intermediary. Because Venus is accessible off-campus, this login should work on and off campus, regardless of whether you are connected to the AU VPN.

Setting up Github account

If you don’t already have an account on GitHub, go there and sign up for a free account. Log in to your GitHub account, and go to the “Settings” for your account (there should be a drop down near the top right corner of the GitHub page, with a “Settings” option). Along the left side of your settings page, there should be a “SSH and GPG keys” link; click on this.

Open a shell (terminal) session on your computer and enter:

cat ~/.ssh/id_rsa.pub

The output of this command will show your public SSH key, which should look something like:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCaZow6ifUAg3g7Qj7N5zJ5fMnQoP
mpAhUqwsvHu/BoXH17TcqP4HdcNoDcprVRwAJL/6ECabdzDyUlGThMKB8w3APqQjqa
7yc98ymdK0LXDpU0IuWCktW9pyn72XizE1bSIOhDkrFKGmtpLW/0jxGhcRN6OtCUI4
V98c2AuU3RKZeTSgfEAWnPJcCQaJkvzktVXO55rsNRe6UxqV4B6O/29YhBeCqyLDL6
VUa7hT+4cqVX8gjLjgDq8jWwxkgeifEt9G1j41 jamie@jamie-XPS-13-9350

Copy the content of this output. Now, go back to GitHub and click “New SSH key.” Paste your copied SSH key into the “Key” field on GitHub. Enter a “Title” that will help you remember what computer the key is from (e.g., xps-laptop). Then hit “Add SSH Key.”

Next, we will log in to Hopper and repeat the same steps. So, log in to Hopper using:

ssh YOUR-AU-USERNAME@hopper.auburn.edu

or, if you have SSH-tunnelling setup:

ssh hopper

Now, on Hopper cat the contents of your public SSH key on Hopper:

cat ~/.ssh/id_rsa.pub

Copy the output SSH key, click “New SSH key” on GitHub, and paste your SSH key into the “Key” field, add a meaningful name into the “Title” field (e.g., Hopper), and click “Add SSH Key.”

Setting up Git on Hopper

Installing Git

The system-wide version of Git on Hopper is too old for our needs, and the version of Git that is available as a module was not built correctly. So, each of us will install our own version of Git to our home directory on Hopper.

Log in to Hopper using:

ssh YOUR-AU-USERNAME@hopper.auburn.edu

or, if you have SSH-tunnelling setup:

ssh hopper

To avoid library conflicts, let’s deactivate conda before we build git:

conda deactivate

If you get a message that the ‘conda’ command was not found, this just means that conda is not installed, which is not problem, and you can proceed. On hopper, download an archive of the Git source code:

wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.29.2.tar.gz

After this command finishes, extract the archive:

tar xzf git-2.29.2.tar.gz

and then cd into the directory with the Git source code:

cd git-2.29.2

Compile the source code using:

make

This will build the git executables from the source code. This will take a few minutes. When it finishes, install the git executables into a bin directory within your Hopper home directory using:

make install

After it finishes, you can cd back out of the directory with the Git source code:

cd ..

and then cleanup by removing everything you downloaded and extracted:

rm git-2.29.2.tar.gz
rm -r git-2.29.2

Next, you need to add the bin directory in your home folder to your PATH variable so that Bash knows where to find the git executable when you type git at the command line. To do this, we will use echo and redirection (>>) to append a knew line to the hidden Bash configuration file called .bashrc:

echo 'export PATH="${HOME}/bin:${PATH}"' >> ~/.bashrc

Now, log off Hopper with exit and then log back in. After logging back in, when you type:

which git

you should get the output:

~/bin/git

and when you enter:

git --version

you should see:

git version 2.29.2

Configuring Git

Next, we will set some configuration settings for git. The latest versions of git issue an annoying warning when you git pull. Historically, the pull command was simply a shortcut to git fetch followed by git merge. However, you can also replace the merge step with rebase. We do not want to use rebase, so we will tell git to use merge when we pull:

git config --global pull.rebase false

If you haven’t configured git on Hopper with your name and email, go ahead and do that now using the following 2 commands (just change my name and email to yours, and use the email associated with your GitHub account):

git config --global user.name "Jamie R. Oaks"
git config --global user.email "joaks1@gmail.com"

If you have an account on GitHub, give Git your GitHub username too (again, change my username to yours):

git config --global github.user "joaks1"

When you commit content to a git repository, git will open a text editor for you to write a message to accompany your commit. By default, it will open vim. So, if you are not confortable using vim (if you don’t know what vim is, then you won’t be comfortable using it), then configure git to open nano instead:

git config --global core.editor "nano -w"

Next, tell git to use pretty colors when it gives you messages:

git config --global color.ui "true"

Installing git-lfs

Git is great at tracking the contents of “raw” text files. In this project, we will produce some large, compressed archives of files. We do not want Git to try to track the contents of these files, because it will slow things down a lot, and blow up the size of the Git database in our project. Luckily, there is a really nice extension for Git for handling large files called Git Large File Storage (LFS). This extension allows us to add large files to out git repo, but prevents git from trying to track the content of these files “line-by-line”.

So, let’s install Git LFS. Use the next three commands to navigate to your home directory, create a new directory called git-lfs, and then cd into it:

cd ~

Next, download an archive of the Git LFS software using wget:

wget https://github.com/git-lfs/git-lfs/releases/download/v2.12.0/git-lfs-linux-amd64-v2.12.0.tar.gz

When the download finishes, make a new directory called git-lfs-files and then extract the files from the archive into the this new directory:

mkdir git-lfs-files
tar xzf git-lfs-linux-amd64-v2.12.0.tar.gz -C git-lfs-files

Then, cd into the git-lfs-files directory:

cd git-lfs-files

Now, set the PREFIX variable to tell the installation script where to put git-lfs, and then run the installation script:

export PREFIX="$HOME"
bash install.sh

Move back out of the git-lfs and cleanup by removing everything we downloaded and extracted:

cd ..
rm git-lfs-linux-amd64-v2.12.0.tar.gz
rm -r git-lfs-files

Lastly, run the following command to configure git-lfs for your Hopper account:

git lfs install --skip-smudge