📚 Jiaqi's Knowledge Repository

Set Up Remote Server for Deep Learning

4 min read

Install Pyenv

Tip

Manage multiple versions of Python with pyenv.

Update System Packages

sudo apt update

(Optional): Install Dependencies

Install optional dependencies with

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

or

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

according to OpenAI’s gpt-4o-2024-05-13.

Install Pyenv

curl https://pyenv.run | bash

Set Up Environment Variables

if [ -n "$BASH_VERSION" ]; then
  # Load pyenv automatically by appending the following to 
  # ~/.bash_profile if it exists, otherwise ~/.profile (for login shells)
  # and ~/.bashrc (for interactive shells) :
  echo -e 'export PYENV_ROOT="$HOME/.pyenv"\n[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
  echo -e 'eval "$(pyenv init -)"' >> ~/.bashrc
  echo "Added pyenv virtualenv-init to ~/.bashrc"
  # Restart your shell for the changes to take effect.
  # Load pyenv-virtualenv automatically by adding the following to ~/.bashrc:
  echo -e 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
  echo "Added pyenv virtualenv-init to ~/.bashrc"
  # Don't forget to restart your shell for the changes to take effect.
  source ~/.bashrc
elif [ -n "$ZSH_VERSION" ]; then
  # Running in Zsh
  echo -e 'export PYENV_ROOT="$HOME/.pyenv"\n[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
  echo -e 'eval "$(pyenv init -)"' >> ~/.zshrc
  echo "Added pyenv virtualenv-init to ~/.zshrc"
  echo -e 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshrc
  echo "Added pyenv virtualenv-init to ~/.zshrc"
  source ~/.zshrc
else
  # Unknown shell
  echo "Unsupported shell."
fi

Install Python & Set Up Virtual Environment with Pyenv

Check available versions

pyenv install --list

Install specific version

pyenv install 3.10.14

Set global version with

pyenv global 3.10.14

or just assign to current shell with

pyenv shell 3.10.14

Install pyenv-virtualenv

git clone https://github.com/pyenv/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

Update shell configuration:

eval "$(pyenv virtualenv-init -)"

Restart your shell:

exec $SHELL

Create a new virtual environment:

pyenv virtualenv 3.10.14 myenv

Activate the virtual environment:

pyenv activate myenv

Install Packages with pip

install packages with pip using mirrors

For one time use, install packages with mirror address

pip install <package> -i https://pypi.tuna.tsinghua.edu.cn/simple
# pip install <package> -i http://mirrors.aliyun.com/pypi/simple --truste-host mirrors.aliyun.com

For global set up, create file ~/.pip/pip.conf with the following content.

[global]
index-url=https://pypi.tuna.tsinghua.edu.cn/simple
[install]
trusted-host=https://pypi.tuna.tsinghua.edu.cn

Create folder ~/.pip/ if it does not exist.

$ pip config list
global.index-url='https://pypi.tuna.tsinghua.edu.cn/simple'
install.trusted-host='https://pypi.tuna.tsinghua.edu.cn'

Tip

Some other significant mirrors

install packages from setup.py

  • The install_requires list in your setup.py file specifies the core dependencies that will always be installed.
  • The extras_require dictionary specifies optional dependencies that are only installed if explicitly requested (e.g., pip install -e ".[dev]").
set(
    ......
    extras_require=extras,
    python_requires=">=3.10.9",
    install_requires=install_requires,
)

Python Package Management with Poetry

install poetry and add to PATH

curl -sSL https://install.python-poetry.org | python3 -
echo 'export PATH=$PATH:$HOME/.local/bin' >> ~/.bashrc
source ~/.bashrc

additional setup

I prefer to use virtual environment under the current project directory.

poetry config virtualenvs.in-project true

use poetry config --list to check current settings.

create new project

cd my-project
poetry init
poetry env use python
# poetry env use /Users/caijiaqi/.pyenv/versions/3.10.14/bin/python

Tip

Note that the python version is determined by python so choose the desired one.

install dependencies

poetry add <package>

Two files, pyproject.toml and poetry.lock, are created and updated accordingly by now. When you use poetry add command, Poetry will automatically do the following three things in this order:

  1. update pyproject.toml
  2. update poetry.lock according to pyproject.toml
  3. update virtual environment according to poetry.lock

Caution

Do not manually edit poetry.lock and pyproject.toml. If you manually edited pyproject.toml, run poetry lock to update poetry.lock file.

read more

Git Clone Projects for Remote Development

clone private projects

git clone https://<github-username>:<PAT>@github.com/<your-repository>.git
Create GitHub Personal Access Token(PAT)
  1. In GitHub, go to Settings ⇒ Developer Settings ⇒ Personal access tokens.
  2. Click Generate new token and assign the token a name.
  3. Grant the token privilege to the repo group.
  4. Copy the token to a secure location. You won’t be able to view the token again.
  5. At the bottom of the page, click Generate Token.

Run Scripts in the Background

nohup [env key=val] script > output.log 2>&1 &

Graph View