RemoteAPIClient object serialization

[projectgoo]

Hello,

I am running couple of headless instances of the CoppeliaSim environment in parallel and I have created a wrapper around the python code that you have provided.

Code: Select all

class EnvWrapper:
	def __init__(self, port: int)
		self.client = RemoteAPIClient(port)
		self.sim = client.getobject('sim')
		self.sensor1Handle = sim.getObject('/VisionSensor')
		self.sensor2Handle = sim.getObject('/PassiveVisionSensor')

Until this point everything is working great and all the environments are configured as expected.

I would like to run the algorithm for example 300 times at each iteration, to learn features and exploit the environments. For this purpose I have the following:
- 8 available processes
- 5 available CoppeliaSim environments (for example ) running in parallel
For the purpose I have made an environment pool so a process can use an environment and when it is done to return it so other process can use it later on. The problem is the following: There is no way to serialize the wrapper class so I can pass it to a shared memory and share it among the processes. I know that this class is a complex data type and it is not an easy task and it holds a state information, but still. my question is, is there a way to make this happen and pass the instance and state of the RemoteAPIClient so I wont be needed to create an instance every time and to reduce the amount of time to initialize those variables every time, and if it is not possible, is it safe to make a new instance every time ?

Thanks in advance!

[fferri]

The RemoteAPIClient class uses ZeroMQ sockets. Python can't easily serialize objects that contain sockets, file handles, or similar resources across processes. ZeroMQ sockets, in particular, are not designed to be shared between processes like normal Python objects, because each socket connection maintains its own context and state.

So, in short, instead of serializing an object of that class, create it from scratch.

If the initialization of RemoteAPIClient takes a significant overhead, use a Pool or Manager Pattern, to re-use the same set of Remote API clients over multiple runs (only within the same process).

[projectgoo]

Thanks a lot for the answer. I have played a little bit with it and actually find a workable solution using ray.io core module. I will share the approach once it is done!