how to get camera data by TCP/IP and display the image in CoppeliaSim

[MaJiamu]

Hello, I want to connect the PC with a teleoperation robot through a cable.

More important, I want to display the image or video from camera equipped on the robot in CoppeliaSim directly. By the way, the camera is pre-configured with a destination IP address.

Therefore, the question is how can I build the connection between the CoppeliaSim and the Camera by the TCP/IP. Is there any tutorial related to that.

Appreciate anyone's help!

[fferri]

Is it an IP camera? What protocol does it use? Are you able to acquire video frames (e.g.: from a C/C++/Python client) already?

[MaJiamu]

Is it an IP camera? What protocol does it use? Are you able to acquire video frames (e.g.: from a C/C++/Python client) already?

Yes, it is a IP camera, and we use the RTMP（Real-Time Messaging Protocol）. The video frames can be acquire already.
It seems that the main problem is how to transform the received video data by TCP/IP from robot camera into image in CoppeliaSim.

[fferri]

You can use sim.setVisionSensorImg to write the raw uncompressed image data (8-bit Grayscale or RGB, i.e. 1 pixel is either 1 or 3 bytes) to a vision sensor.

[MaJiamu]

You can use sim.setVisionSensorImg to write the raw uncompressed image data (8-bit Grayscale or RGB, i.e. 1 pixel is either 1 or 3 bytes) to a vision sensor.

Thanks for your advice. However, how can I get the video image from my robot IP camera? Without the video image, the sim.setVisionSensorImg got no 'buffer image' input.

By the way, I try to use the simIM.openVideoCapture(int deviceIndex) to get video image just like this .ttt https://github.com/CoppeliaRobotics/sim ... apture.ttt. But, it doesn't work because the 'deviceIndex' require an 'int' value rather than an IP 'string', such as simIM.openVideoCapture('192.168.0.70:8000') or something like that.

Besides, I read the sourceCode in simIM Gituhub https://github.com/CoppeliaRobotics/sim ... plugin.cpp.

Code: Select all

    void openVideoCapture(openVideoCapture_in *in, openVideoCapture_out *out)
    {
        if(!videoCapture[in->deviceIndex].isOpened())
            videoCapture[in->deviceIndex].open(in->deviceIndex);

        if(!videoCapture[in->deviceIndex].isOpened())
            throw std::runtime_error("failed to open device");
    }

It seems that the IP address input is not available for simIM.openVideoCapture(int deviceIndex).
So how can I modify the sourceCode in simIM to make the expression like simIM.openVideoCapture('192.168.0.70:8000') run successfully and get the robot video image?

[fferri]

simIM.openVideoCapture is only for cameras directly connected to the computer (e.g. via USB, FireWire, etc...).

If you already have the code to read frames from the remote camera in C/C++, the simplest way is to write a plugin with a function that either returns the image buffer and the resolution, or directly writes that to a vision sensor given its handle using simSetVisionSensorImg (C API).

If you have the code in Python, you could create an embedded python script, and in there acquire the camera frame and write it using sim.setVisionSensorImg.

[MaJiamu]

Thank you very much for your advice. We are going to try the second method.

By the way, is it possible for CoppeliaSim to have both a Lua script and a Python script in one scene?
For example, attaching a Lua script to Dummy1 and a Python script to Dummy2, both scripts can run and perform their respective functions after the simulation running in the same .ttt.

[fferri]

By the way, is it possible for CoppeliaSim to have both a Lua script and a Python script in one scene?

Yes.

You can also call a Lua function from Python and vice-versa, using sim.callScriptFunction.

[MaJiamu]

Finally, it works! Following are the details.
1. Test Python Scripts in CoppeliaSim:
First, ensure that any Python-coded scenes work well in CoppeliaSim. You can test this with a sample scene, such as C:\Program Files\CoppeliaRobotics\CoppeliaSimEdu\scenes\vision\imageProcessingDemo1-python.ttt.

2. Obtain IP Camera Address:
Obtain your IP camera address or URL, such as rtsp://192.168.0.30:454/user=admin&password=&channel=1&stream=0.sdp?.
Ensure that your PC can communicate with the IP camera, and test the video stream in VLC or another media player to confirm it is working properly.

3. Build a New Scene in CoppeliaSim:
Create a new scene in CoppeliaSim and add a Vision Sensor. Attach a Floating View to the sensor.
Configure the Vision Sensor according to your camera's specifications.
Add a Python script to the scene, with code similar to the following:

Code: Select all

#python

import cv2 as cv
import numpy as np
result = cv.VideoCapture('http://192.168.0.103:8080/?action=stream?')#
sensor=0

def sysCall_init():
    global sensor
    sim = require('sim')
    simIM = require('simIM')
    
    sensor=sim.getObject('/Vision_sensor2')

def sysCall_thread():
    # Put your main code here, e.g.:
    global result
    global sensor
    while result.isOpened:
        result.set(cv.CAP_PROP_BUFFERSIZE, 3) 
        fps = result.get(cv.CAP_PROP_FPS)
        ret,frm = result.read()
        #---------
        frm_rgb = cv.cvtColor(frm, cv.COLOR_BGR2RGB)
        frm = cv.resize(frm_rgb, (640,480))
        frm = cv.flip(frm,0)
        image_buffer = frm.tobytes()
        sim.setVisionSensorImg(sensor,image_buffer)
        
        #--------
        cv.waitKey(1) 
    pass

# See the user manual or the available code snippets for additional callback functions and details
def sysCall_cleanup():
    global result
    result.release();
    cv.destroyAllWindows()