① 3D object detection using multiple cameras

By integrating images from multiple cameras, the system detects objects such as people, vehicles, and forklifts in three-dimensional space. The detection results are output as a 3D bounding box that includes not only position but also height, width, depth, and orientation, allowing for an understanding of the spatial relationships between objects.

② Continuous tracking of objects (Multi-Object Tracking)

Each detected object is assigned a Tracking ID. By maintaining the same ID for the same object across frames, it is possible to continuously track the movement paths and dwell times of people and transport equipment.

③ Visualization of space using Bird's Eye View (BEV)

Sparse4D integrates recognition results from multiple cameras into a common coordinate system, allowing for an overhead view as a BEV (Beam Elevation Scale). This makes it possible to intuitively check the positional relationships and movement of objects, which are difficult to grasp from the viewpoint of each individual camera, as a single space.

④ 3D recognition utilizing existing camera equipment

Sparse4D achieves 3D recognition using multiple RGB cameras rather than dedicated sensors such as LiDAR. Therefore, a major advantage is that it allows for the construction of a surrounding environment recognition system while utilizing existing camera equipment.

⑤ Adaptation to the environment through fine tuning

Sparse4D is compatible with the NVIDIA TAO Toolkit, allowing for fine-tuning using data collected or simulated by your company. This enables you to adapt the model to the target object and installation environment, creating an AI model that is more suitable for practical use.

① Safety management and work analysis in logistics warehouses

In logistics warehouses, workers, forklifts, and transport robots operate in the same space. By using Sparse4D, the position and movement of each can be understood in three dimensions, which can be used to visualize contact risks, detect intrusion into hazardous areas, and analyze work flow. In addition, BEV allows for an overview of the entire site, providing information useful for operational improvements such as reviewing transport routes and analyzing bottlenecks.

② Equipment monitoring in factories

In factories, it is common to monitor the areas around production and conveying equipment with multiple cameras. By using Sparse4D, the relative positions of workers and conveying equipment around the equipment can be understood in three dimensions, allowing for more accurate monitoring of approaches to hazardous areas and congestion around the equipment. The acquired location information and movement history can also be used to verify compliance with safety standards, improve equipment layouts, and review operational rules.

③ Integration with autonomous transport systems including AGVs and AMRs

In environments where AGVs (Automated Guided Vehicles) and AMRs (Autonomous Mobile Robots) are in operation, it is crucial to understand the positional relationship between the system and surrounding people and transport equipment. Sparse4D can recognize the surrounding environment in three dimensions based on information acquired from multiple cameras, and when combined with operation management systems and monitoring systems, it can support safe operation.

For example, visualizing the proximity of vehicles to people and other transport equipment, and analyzing congested areas and times, can be expected to lead to the optimization of operational plans and improvements in safety.

Recommended environment and testing environment

In this verification, we used a Warehouse Blueprint 4-stream sample dataset on a GPU server equipped with an NVIDIA L40S-48Q (48GB VRAM) to confirm 3D object detection and tracking using Sparse4D. Note that the actual GPU performance required will vary depending on the number of input cameras, video resolution, frame rate, and combination with other AI models. The environment described in this article should be used as an example for evaluating and verifying Sparse4D.

① Obtaining the Warehouse Blueprint

First, obtain the Warehouse Blueprint repository from GitHub.

git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout tags/v3.2.0
git lfs install
git lfs pull
cd deploy/docker

Note: The repository uses Git LFS (Large File Storage), so you need to install Git LFS beforehand.

② Setting the NGC API Key

Warehouse Blueprint requires an NGC API Key to retrieve container images and sample data from NVIDIA NGC™. Set it as an environment variable as follows:
For information on how to obtain an NGC API Key, please refer to the NGC User Guide.

export NGC_CLI_API_KEY=<NGC_CLI_API_KEY>
export NGC_CLI_ORG='nvidia'

③ Download sample data

Warehouse Blueprint includes sample video data that allows you to immediately check its functionality.

ngc \
   registry \
   resource \
   download-version \
   nvidia/vss-warehouse/vss-warehouse-app-data:3.2.0
cd vss-warehouse-app-data_v3.2.0
tar -xvf vss-warehouse-app-data.tar.gz
sudo chmod -R 777 /path/to/vss-warehouse-app-data

The necessary data has now been downloaded.

④ Deploy Warehouse Blueprint

MODE=3d # Sparse4Dを使用する場合は'3d'を選択 BP_PROFILE=bp_wh_kafka # 'MODE=3d'の場合は'bp_wh_kafka'を選択 SAMPLE_VIDEO_DATASET="warehouse-4cams-20mx20m-synthetic" # Sparse4D用のデータセットを選択 HARDWARE_PROFILE=L40S # 使用しているHWを選択 VSS_APPS_DIR="/path/to/deploy/docker" # Blueprintのダウンロードパスを選択 VSS_DATA_DIR="/path/to/vss-warehouse-app-data" # サンプルデータのダウンロードパスを選択 HOST_IP='localhost' # Host IPを入力 NGC_CLI_API_KEY='YOUR_KEY' # NGC CLI API Keyを入力 NVIDIA_API_KEY='YOUR_KEY' # NVIDIA API Keyを入力

source /path/to/deploy/docker/industry-profiles/warehouse-operations/.env
cd /path/to/deploy/docker
docker login \
    --username '$oauthtoken' \
    --password "${NGC_CLI_API_KEY}" \
    nvcr.io
docker compose \
    --env-file industry-profiles/warehouse-operations/.env \
    up \
    --detach \
    --pull always \
    --force-recreate \
    --build

① Access VST

Access http://<HOST_IP>:30888/vst from your web browser.

The image above shows an example of what the VST (Video Storage Toolkit) main screen looks like when accessed through a browser. You can view the Sparse4D inference results by selecting "Video Wall" from the menu on the left.

② Display inference results in Video Wall

Opening Video Wall from the left-hand menu will run Sparse4D on sample footage from Warehouse Blueprint, allowing you to view the recognition results from multiple cameras in real time.

In Video Wall, objects such as people and vehicles are displayed as 3D bounding boxes for each camera feed. Furthermore, detected objects are assigned a Tracking ID, allowing you to confirm that the same object is being continuously tracked across frames.

The image above shows an example of displaying video feeds from four cameras simultaneously using Video Wall.

③ Check the BEV display.

By displaying BEV from the Video Wall settings, you can view the recognition results acquired from each camera from an overhead perspective. A key feature is that it allows you to intuitively understand the positional relationships and movement of objects as a single space, which can be difficult to grasp with multiple cameras.

Information you can check on Video Wall

In Video Wall, you can primarily check the following information as a result of recognition by Sparse4D: