This guide is suitable for Linux servers and the WSL2 Linux subsystem on Windows for setting up a PyTorch environment (GPU version). The second section covers the installation of the D4RL reinforcement learning library. Students not focusing on reinforcement learning can refer to only the first part. This tutorial was written while installing from scratch, ensuring it works for me; if issues arise, use Bing or Google (not Baidu) and DeepSeek.
Continuing from the previous article, after installing D4RL on Windows, everything seemed to work, but when running code, an error occurred:
Exception: Please add mujoco library to your PATH: set PATH=C:\Users\baoda.mujoco\mujoco210\bin;%PATH%
After some operations, another error appeared:
Warning: Mujoco-based envs failed to import. Set the environment variable D4RL_SUPPRESS_IMPORT_ERROR=1 to suppress this message. DLL load failed while importing cymj: 找不到指定的模块.
After extensive troubleshooting, I gave up and switched to configuring the Linux subsystem on WSL2. I had previously successfully configured the D4RL environment on a server but didn't document it, so I'm recording the process here for my own reference.
The following installation starts from scratch; the configuration process is the same for Linux servers and WSL Linux subsystems.
Part 1: PyTorch Environment Setup
(1) Install WSL2 and Ubuntu
Refer to the tutorials "Custom Installation of WSL2" and "WSL Installation and Usage" for detailed steps. Avoid downloading directly from the Microsoft Store as the first tutorial explains how to install it on a non-C drive.
Students configuring on a server do not need to focus on this step.
(2) Install Miniconda and PyTorch
To reduce disk usage, install Miniconda instead of Anaconda. If disk space is sufficient or on a server, you can install Anaconda (refer to other Anaconda tutorials), which does not affect other tool installations.
1. Install CUDA Toolkit
(1) Run nvidia-smi in the terminal to check the highest possible CUDA version.
Check the official documentation for the corresponding CUDA version based on your device's driver version (shown by nvidia-smi). Download the appropriate version using the commands provided by the official site.
(2) After installing the CUDA toolkit, open the ~/.bashrc file:
nano ~/.bashrc
Add the following lines at the end of the ~/.bashrc file:
# add nvcc compiler to path
export PATH=$PATH:/usr/local/cuda-11.3/bin
# add cuBLAS, cuSPARSE, cuRAND, cuSOLVER, cuFFT to path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.3/lib64:/usr/lib/x86_64-linux-gnu
Replace cuda-11.3 with your actual installed version. After saving, reload the bashrc file:
source ~/.bashrc
Run nvcc -V in the terminal. If the version number appears, CUDA is installed correctly.
2. Install cuDNN
CUDA is a general-purpose GPU computing platform provided by NVIDIA, while cuDNN is a deep learning library built on top of CUDA, relying on CUDA for GPU acceleration. If you are running frameworks like TensorFlow or PyTorch, you usually need to install cuDNN, otherwise critical libraries may be missing, preventing GPU training.
Download cuDNN from the official site or check historical versions (which include correspondence between CUDA and cuDNN versions, but require login). I used the historical version method (if installing a cuDNN starting with 9.x, there will be an official command, follow the given command operation, generally no errors occur). This will result in a local .deb file (remember to copy it into your Linux user folder, located at /home/YourName).
Use the official installation command for historical versions (note to change the specific version number, e.g., "9.0.0"):
sudo dpkg -i cudnn-local-repo-ubuntu2004-9.0.0_1.0-1_amd64.deb
sudo cp /var/cudnn-local-repo-ubuntu2004-9.0.0/cudnn-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cudnn
Installation may have errors, such as mine, where I found that related libraries were already complete (the first three items):
Then install directly:
sudo apt-get install -y libcudnn8
sudo apt-get install -y libcudnn8-dev
sudo apt-get install -y libcudnn8-samples # optional
If you installed libcudnn8-samples, run the flolowing command to test whether cuDNN is installed:
cd /usr/src/cudnn_samples_v8/mnistCUDNN
make
./mnistCUDNN
If Test passed! is output, it is successful.
Or after installing PyTorch, enter the virtual environment with PyTorch and test the example:
python # enter the Python environment import torch print(torch.backends.cudnn.version()) # should return a number like 90100 from torch.backends import cudnn # if normal, it will be silent cudnn.is_available() # should return True a=torch.tensor(1.) cudnn.is_acceptable(a.cuda()) # should return True
3. Install Miniconda
Choose a Linux version from the mirror source and copy its download link Index of /anaconda/miniconda/ | Tsinghua University Open Source Software Mirror
Select the 64-bit Linux version, py39 represents Python 3.9 (the Python version in bash, which doesn't matter as we usually don't use the base environment)
Then execute the download and installation operations:
wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
bash Miniconda3-py39_4.9.2-Linux-x86_64.sh
The URL after wget is the one you just copied, and the content after bash is the last part of the link you copied. Note that during installation, you might find the window frozen and the word MORE displayed. Press Enter multiple times to proceed, and follow the prompts to input yes or press Enter.
!! Take my experience as a warning,
This is pressing Enter key! Not entering ENTER, I was brain dead back then... Now it's fixed, but there's no way to change it (if anyone knows how to modify it, please feel free to share, it's driving me crazy)
Restart the terminal (exit first, then wsl), if the prompt shows base, it means the configuration is successful.
4. Create a Virtual Environment
There's nothing more to say here. In the bash environment, directly create:
(Replace env_name with your own environment name, and choose the Python version according to your needs, but don't go too new)
conda create -n env_name python=3.8
5. Install PyTorch in the Created Virtual Environment
You can go to the PyTorch website to select the corresponding version and install with a single command. However, due to network speed issues, I recommend installing via downloaded .whl files locally.
Here are the download links for torch, torchvision, and torchaudio (installing PyTorch requires these three, note the version match):
torch
vision
torchaudio
For example, if I installed python=3.8 and cuda-11.3 (the same as the previously installed CUDA toolkit version, make sure to match it!), the corresponding three files and installation commands are:
# After entering the directory of .whl files, run the following commands
pip install torch-1.10.0+cu113-cp38-cp38-linux_x86_64.whl
pip install torchaudio-0.10.0+cu113-cp38-cp38-linux_x86_64.whl
pip install torchvision-0.11.0+cu113-cp38-cp38-linux_x86_64.whl
The cu113 matches the cuda-11.3/cuda toolkit 11.3. From my observation, 1.x versions of torch seem to have an extra 1 compared to torchaudio and torchvision, and the difference between 10 and 11 seems to have little impact.
🎉🎉 At this point, the Miniconda and GPU version of the PyTorch environment is fully configured and ready to play!
Part 2: Installing D4RL Library
The following content is executed after entering WSL via win+r cmd; there is no difference on a Linux server, except for the absence of /mnt in path writing. ==Note: Before pip installing packages, ensure that the correct virtual environment is active. If the download speed is slow, switch to the Tsinghua source by running the following command:
python -m pip install --upgrade pip
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
- Install mujoco210
(1) Download the corresponding version of the compressed package from the official website; then create a folder in the current user's home directory
mkdir ~/.mujoco
(2) Go to the directory where your compressed package is located, for example, if it is on the desktop, cd /mnt/desktop, then extract it
cd /mnt/desktop
tar -zxvf mujoco210-linux-x86_64.tar.gz -C ~/.mujoco
(3) Add environment variables and refresh Open the environment variable file
nano ~/.bashrc
Add the following at the end of the file (before the #>>conda initialize >>):
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/XXX/.mujoco/mujoco210/bin
Save and close, then refresh the environment variables
source ~/.bashrc
(4) Test if mujoco is installed successfully
cd ~/.mujoco/mujoco210/bin
./simultae ../model/humanoid.xml
If the simulation environment shown in the figure appears, the installation is successful:
- Install mujoco-py
(1) mujoco-py is the Python binding for MuJoCo, required by D4RL. Here I selected a suitable version. First, activate the virtual environment where PyTorch was installed, then execute
python -m pip install mujoco-py==2.1.2.14
(2) Add environment variables Open the environment variable file
nano ~/.bashrc
Add after the previously added environment variables:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia
Save and close (press CTRL X, Y, Enter), then reload the environment variables
source ~/.bashrc
(3) Test if it is installed successfully
# After entering the corresponding virtual environment, run the following sequentially
python
import mujoco_py
No error indicates success. For example, I encountered an error due to a high Cython version, which I modified to the corresponding version (cython==0.29.37); then I encountered a missing OSMesa (Off-Screen Mesa) development library (method: sudo apt-get update and sudo apt-get install build-essential libosmesa6-dev libgl1-mesa-glx libglfw3-dev patchelf). The errors encountered vary, and you can ask AI for help.
- Install D4RL Dependencies
The official repository actually provides the installation method,
git clone https://github.com/Farama-Foundation/d4rl.git cd d4rl pip install -e .
>
> But there may be timeout issues, so here we use another way to install.
Install some dependencies before installing D4RL.
(1) gym
pip install gymnasium==1.1.1
If the D4RL documentation specifies an older version of Gym (e.g., gym instead of gymnasium), install the older version:
pip install gym==0.21.0
(2) mjrl
The reason the official repository installation of D4RL times out is because mjrl fetches remotely. We install it locally. Download the mjrl zip file, right-click to extract it. cd into the directory and install it.
For example, my path is as follows
cd /mnt/d/desktop/mjrl-master
Check the contents before proceeding to avoid mistakes
ls
Install
pip install .
(3) Other dependencies
- Installing D4RL also requires some other libraries
pip install numpy==1.24.4 h5py==3.11.0 termcolor==1.1.0 pybullet==3.2.7 click==8.1.8 -i https://pypi.tuna.tsinghua.edu.cn/simple
4. Install D4RL
---------
(1) Download the zip file provided by the official D4RL repository and right-click to extract it.
(2) Make the following changes in the setup.py file at the bottom. Do not change any other content. Comment out these packages, which we have already installed above.
install_requires=[
"gym<0.24.0",
#"numpy",
#"mujoco_py",
#"pybullet",
#"h5py",
#"termcolor", # adept_envs dependency
#"click", # adept_envs dependency
"dm_control>=1.0.3",
#"mjrl @ git+https://github.com/aravindr93/mjrl@master#egg=mjrl",
],
(3) cd into the directory and install it.
> It is recommended to place the folder in a suitable location before installation, rather than on the desktop! <font>Because after installing with -e, you cannot delete the source files!!</font>
For example, my path
cd /mnt/d/desktop/D4RL-master
Check if you have entered the correct directory
ls
Install, the -e option indicates editable mode installation, convenient for subsequent code modifications
pip install -e .
At this point, the installation is complete~🎉🎉🎉
Part 3: Random Thoughts
========
Students who have read this far have already succeeded 99.99%! 🥳
We have successfully configured the offline reinforcement learning library D4RL on WSL2. However, when running specific code, additional issues may still arise.
For example, after strictly following the above steps to configure from scratch, running the TD3+BC code resulted in an error. Asking AI, the problem was a missing package six, causing the hopper-medium-v2 not to register to gym. Then we installed the six package (e.g., six==1.17.0) and continued.
Next, run
python main.py --env hopper-medium-v2
When running for the first time, you will see it downloads the dataset, which is very slow, so you can terminate it with ctrl c. Then go to your home directory and find a .d4rl folder. Double-click to open it, and you will find the datasets folder, which is where the dataset is stored. So, we can directly go to the official website to download the required dataset, which looks like the following👇
Next time you run your experiments, you will feel it flies ~ (It's still very slow!! Just not having to re-download the dataset, and reinforcement learning experiments are really slow!! However, it doesn't take much VRAM, so you can debug on your computer and then run it on the server~!)
(The extra commands below are because I made changes to TD3+BC)
Wishing you all the best in your research!
(Opportunity to write about Atari, dm-control configuration process, which I have already configured once on the server, but not on my own computer)