I think the method of video input should be the first consideration for any video processing system. Part 2 of this series will show you how to capture images from a CSI-connected Libargus compliant camera connected to Jetson.

 

[Jetson video processing programming]

Episode 1 What you can do with JetPack and SDK provided by NVIDIA

Episode 2 Video input (CSI-connected Libargus-compliant camera)

Episode 3 Video input (USB-connected V4L2-compliant camera)

Episode 4 Resize and format conversion

Episode 5 Image display

Episode 6 Video encoding

Episode 7 Video decoding

Episode 8 Image Processing

Episode 9 Deep Learning Inference

Episode 10 Maximum Use of Computing Resources

Camera connection interface

Jetson has a dedicated camera interface called MIPI Camera Serial Interface (CSI). This time, we will talk about this CSI-connected camera (hereinafter referred to as "CSI camera"), but first, let's look at other connection methods.

Connection method

Strong Points

Cons/Challenges

CSI
  • low latency
  • high frame rate
  • Device drivers required for each camera model
  • When using Jetson's ISP, support from NVIDIA's certified camera partner is required (paid)
USB Video Class (UVC)
  • easy to obtain
  • Wide variety from low-priced webcams to high-performance industrial cameras
  • High latency compared to CSI method
USB3 Vision
  • high frame rate
  • Few Jetson compatible products
GigE
  • long distance transmission
  • Few Jetson compatible products

IP

(network camera)

  • easy to obtain
  • high latency
  • Low quality (usually due to compression)

CSI

CSI is a standard established by the Mobile Industry Processor Interface (MIPI) Alliance, and all current Jetson products are CSI-2 compliant and have dedicated I/O pins for CSI. However, since the number of lanes and PHY differ for each Jetson model, the maximum number of cameras that can be connected and the corresponding frame rate will differ within the Jetson series.

CSI cameras compatible with Jetson are sold by third-party companies, but a typical example that can be easily used is the Raspberry Pi camera module V2(note that V1 is not compatible), and the Jetson Nano developer kit and Jetson Xavier NX Developer Kit.

CSI cameras compatible with Jetson

CSI cameras compatible with Jetson are sold by third parties. Be sure to purchase a Jetson compatible CSI camera, as each camera model requires a corresponding Linux device driver.

Example: Leopard Imaging camera

Considerations when choosing a CSI camera

Please note that using a CSI camera requires a Linux device driver for each model of CSI camera (image sensor). The device driver for the Raspberry Pi Camera Module V2 mentioned earlier is already included in Jetson's L4T OS, so it's ready to go.

CSI cameras usually output image data in a format called Raw (Bayer). A module called Image Signal Processor (ISP) inside Jetson performs a number of processes such as CFA interpolation (demosaicing), noise removal, pixel defect correction, distortion correction, gamma correction, etc., and is handled by image processing software. Converts to image data that can be processed. However, the details of Jetson's ISP module are private, and only third parties called Jetson camera partners who have received special disclosure from NVIDIA can handle Jetson's ISP. Therefore, in order to use an unsupported CSI camera with Jetson, it is necessary to receive support (paid) from a Jetson camera partner. If the camera has an ISP function, bypassing Jetson's ISP may reduce the difficulty of device driver development, but development is still necessary. This situation is not specific to Jetson, and may apply to other platforms as well.

 

 

Connection method

Jetson's

use your ISP

APIs

CSI

Yes

Libargus

CSI

No

Bypass your ISP

V4L2

USB (UVC)

No (Note 1)

V4L2

Note 1: There is no way to use Jetson's ISP with a USB connection.

Image acquisition method

There are two ways to acquire images from a CSI camera.

・How to use GStreamer

・How to use the Libargus Camera API

Each method will be explained below.

How to use GStreamer

nvarguscamerasrc element provided by NVIDIA. The easiest way to do this is to run the following command from a Jetson command window.You can access the CSI cameras using the

$ gst-launch-1.0 nvarguscamerasrc ! nvoverlaysink

In the above example, the nvarguscamerasrc element is working with default settings, but you can check the parameters that can be set with the following command.

$ gst-inspect-1.0 nvarguscamerasrc

For example, to capture images from a CSI camera attached to the second camera connector on the Jetson Nano developer kit, set the sensor-id parameter to:

$ gst-launch-1.0 nvarguscamerasrc sensor-id=1 ! nvoverlaysink

However, if you start it from the command line as shown above, it will be less flexible and not suitable for embedding in your application.

Access nvarguscamerasrc element from OpenCV

By utilizing the OpenCV VideoCapture GStreamer backend, you can acquire images from a CSI camera with OpenCV's VideoCapture.

 

reference:

Jetson Hacks Nano/CSI-Camera

MACNICA-CLAVIS-NV/camcapture

How to use the Libargus Camera API

The Libargus Camera API, included in the Jetson Multitimedia API, allows fine-grained control over the CSI camera. There are many controls available and a wide variety of data that can be accessed, but that makes programming complicated. The sample code is included in JetPack and can be found in the path below.

/usr/src/jetson_multimedia_api/argus

Next time, I will explain how to input video from a USB camera!

How was the video input method in the second episode of the series "Jetson Video Processing Programming"?

Next time, we will introduce V4L2-compliant cameras such as USB cameras, which are also video input.

If you have any questions, please feel free to contact us.

We offer selection and support for hardware NVIDIA GPU cards and GPU workstations, as well as facial recognition, route analysis, skeleton detection algorithms, and learning environment construction services. If you have any problems, please feel free to contact us.