Reproducing Vision-Based End-to-End Autonomous Flight on UAVs: Ubuntu Tutorial
Read the previous article about Tianjin University Interpretation of YOPO papers Already? YOPO reduces latency to milliseconds through end-to-end trajectory planning, driving UAVs to fly quickly and stably autonomously.
We have implemented the reproduction of the YOPO algorithm on SU17, and now we will share the reproduction tutorials with you. We will serialize three reproduction tutorials including:Ubuntu training/simulation, ProSim training/simulation, real machine reproduction, please continue to pay attention!
This article mainly introduces the training and simulation reproduction tutorial based on the Ubuntu platform. By following the tutorial, you can run smoothly and easily realize visual end-to-end flight on your own UAV.
01 Environment deployment
Install Conda, ROS, CUDA, system library dependencies
Conda download link:
https://www.anaconda.com/download/success
ROS quick installation link:
wget http://fishros.com/install -O fishros && . fishros
CUDA link:
https://developer.nvidia.com/cuda-toolkit-archive
System library dependencies:
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
build-essential \
cmake \
libzmqpp-dev \
libopencv-dev \
libpcl-dev
02 environment variables
Add the source of catkin workspace as FLIGHTMARE_PATH environment variable
echo "export FLIGHTMARE_PATH=~/YOPO" >> ~/.bashrc
source ~/.bashrc
03 Path modification
Place the YOPO compressed package in the user directory and extract it:
Place the RPG_Flightmare folder in the flightrender directory. The structure is as shown below:
04 Create a Python environment
Use conda to create a python environment (since YOPO already exists, use yopo_test to demonstrate)
conda create --name yopo python=3.8
After downloading, as shown in the figure below, and use conda activate yopo activation:
After activation, it will look like the picture below:
05 Configure Python environment
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
pytorch-cuda=11.8 -c pytorch -c nvidia
The normal situation is as shown below:
Install opencv-python
pip install opencv-python
Install gym and stable-baseline3
pip install gym stable-baselines3
Install other tools
pip install scipy==1.10.1 scikit-build==0.18.1 ruamel-yaml==0.17.21 numpy==1.22.3
tensorboard==2.8.0 empy catkin_pkg
06 Build flightlib
conda activate yopo
cd YOPO/flightlib/build
cmake ..
07 Compile flightlib
make -j4
Things to note
You need to modify arch and code to parameters suitable for your computer graphics card and CUDA version.
After compilation, the following figure appears:
08 Install flightgym
cd YOPO/flightlib/build
pip install -e .
Pip install .
09 Configure flightpolicy module
echo "export PYTHONPATH=$PYTHONPATH:~/YOPO" >> ~/.bashrc
source ~/.bashrc
10 Data Collection Module
cd ~/YOPO/run
conda activate yopo
python data_collection_simulation.py
Install tqdm:
pip install tqdm
problem solving
After installation, the error is reported again as follows:
Indicates that the protobuf version is incompatible
Solution:
pip install protobuf==3.20.3
The data collection process is as follows:
The collection is completed as follows:
11 Model training
cd ~/YOPO/run
conda activate yopo
python run_yopo.py --train=1
After training is completed, the model is saved as follows:
12 test 1
Test without dynamics model and controller
cd ~/YOPO/run
conda activate yopo
python run_yopo.py --train=0 --render=1 --trial=1 --epoch=0 --iter=0 --supervised=0
compile controller
The error is reported as follows:
Reason: empy is not compatible
Solution:
pip install empy==3.3.4
After installation, recompile
13 test 2
Testing using dynamics models and controllers
Each underscore is executed in a terminal, multiple terminals are required.
Start the simulation environment with the Unity and ROS interfaces:
cd ~/YOPO/flightrender/RPG_Flightmare./flightmare.x86_64
------------------------------
cd ~/YOPO/flightlib/build
./flightros_node
------------------------------------------------------
Start the controllercd ~/YOPO source devel/setup.bashroslaunch so3_quadrotor_simulator simulator.launch
-----------------------------------------------------------------------------
Load the model and start planning:
cd ~/YOPO/runconda activate yopo
python test_yopo_ros_new.py
---------------------------------------------------
Start RViz visualization
cd ~/YOPO/
rviz -d yopo.rviz
-------------------------------------------------------
Start map visualization
cd ~/YOPO/flightlib/build
./map_visual_node
problem solving
The error is reported as follows:
Solution:
pip install rospkg
Once completed, end-to-end path planning testing can begin.
End-to-end path planning testing
After starting YOPO in the rviz environment, clicking the target point with the mouse can trigger a complete closed loop: the target coordinates are sent to the YOPO inference module, the corresponding trajectory is generated and solved into control instructions and sent to the UAV; in the actual measurement, the UAV can complete autonomous obstacle avoidance and movement according to the planned path.
When entering the set threshold range of the target point, the terminal outputs "arrive", then stops issuing control instructions, and the UAV hovers at the current location; if you need to go to a new location, just continue to click on the target in rviz. The process is stable and reusable.
Resource Express
Paper analysis:
Tianjin University end-to-end planning algorithm open-source! Autonomous flight no longer lags/recurrence video attached
Paper link:
https://ieeexplore.ieee.org/document/10528860
open-source code:
