How to Test Whether Radar and Camera Time Are Synchronized

In multi-sensor fusion systems, the collaborative perception of cameras and radars has become the key to environmental understanding. Cameras identify object categories by capturing texture information, while radar uses lasers or millimeter waves to achieve accurate ranging all day long. The data fusion of the two can not only avoid the defects of a single sensor (such as the camera being affected by lighting and the radar lacking semantic information), but also significantly improve the stability and accuracy of dynamic obstacle tracking.

Application scenarios

Autonomous driving
In autonomous driving, cameras are responsible for acquiring visual information about the environment and classifying and identifying targets such as pedestrians, vehicles, and traffic signs; radar provides high-precision distance and speed measurements to avoid misjudgments (such as distinguishing road signs from real obstacles).

localization and mapping
LiDAR SLAM can generate a three-dimensional contour map of the environment, combined with the semantic landmarks provided by the camera, to provide more stable support for high-precision localization and path planning.

The importance of time synchronization

Currently, multi-sensor fusion technology is widely used in the field of SLAM (simultaneous localization and mapping). Ensuring the time synchronization of each sensor data is the key to improving the robustness and accuracy of the system. Due to differences in sampling frequency, startup delay, and data transmission between cameras and radars, if effective time alignment is not performed, data fusion errors may occur and affect system performance. Therefore, when designing a fusion system of cameras and radars, the gap between sensors should first be resolved.Time synchronization problem, to achieve high-precision environment perception and localization.

Common time synchronization methods

NTP time synchronization

NTP (Network Time Protocol): Synchronizes system time through a network server. The accuracy is usually millisecond level, which is suitable for low-precision requirements. It is easy to deploy and is greatly affected by network latency. Widely used in servers, industrial equipment, robots and other fields.

PTP time synchronization

PTP (Precision Time Protocol, IEEE 1588): It is a high-precision time synchronization protocol, mainly used to achieve nanosecond-level time synchronization in local area networks (LAN). Its core principle is based on master-slave architecture and bidirectional message exchange, and achieves clock alignment between devices by measuring and compensating for network delays.

GPS time synchronization

The GPS receiver outputs a PPS signal that is strictly aligned with UTC second pulses. Other devices capture the PPS rising edge through the hardware interface (GPIO, TTL) and use this hardware pulse as a time base to correct the local clock or flip-flop.

Hardware triggered synchronization

Use a dedicated trigger or FPGA/MCU to generate a uniform physical trigger pulse (such as a TTL low/high signal). All devices collect data frames at the same time, and the timestamps are generated directly by the hardware.

How to test whether the camera and radar are synchronized?

Taking the quad-camera and Livox Mid-360LiDAR as an example, in order to fuse the data of the camera and radar, we first need to test whether the camera and radar have reached time synchronization, so we need to test that the difference in data timestamps between the two remains in a stable range.

sync mode

According to the radar and camera parameters, both devices support hardware synchronization. The radar hardware synchronization method is shown in the figure below. Here we use the serial port method for synchronization. The specific pin signal requirements are as shown in the table below. In order to meet the hardware triggering methods of radar and camera at the same time, the test uses STM32 to send pulse signals.

Test steps

• Configure the camera for equal-spaced sampling, so that the camera will send out a pulse when sampling, receive this pulse through the STM32 microcontroller and convert it into a 1Hz synchronization signal (1PPS)

• Complete the configuration of the hardware trigger requirements of the radar and camera in STM32, and connect the corresponding pins to the hardware trigger ports of the radar and camera. The trigger frequency of this test is 10Hz. The principle is shown in the figure below, where t0 is the interval between two adjacent second pulse rising edges; t1: the high level time of the second pulse; t2: the transmission time of GPRMC; t3: the delay between the start of GPRMC data transmission and the rising edge of the pulse.

• In order to test whether the camera and radar are synchronized successfully, we wrote a test demo, mainly by recording the timestamp of each frame received by the radar and camera, and comparing the difference between the two.convergence to determine whether the synchronization is successful. What should be noted here is that the timestamp formats of radar and camera are different and need to be converted into a unified format. This test uses a unified UTC format.

Test results

The test results are shown in the figure below. From the results in figure (a), it can be seen that the difference between the two has been converging. In Figure (b), the difference between the two is always less than 1ms. This experimental result shows that the time synchronization of the radar and camera is successful.

Figure (a)

Figure (b)