Deepbots GoalEnv

[tsampazk]

No description provided.

[tsampazk]

@KelvinYang0320 Are you interested in taking look at gym's GoalEnv and what we need to do in order to integrate it with deepbots?

[KelvinYang0320]

@tsampazk Yes, I will spend some time to look into GoalEnv.

[tsampazk]

@KelvinYang0320 Thank you!

[KelvinYang0320]

@ManosMagnus @tsampazk
At the end of last year, GoalEnv has been removed from Gym.
I think there are two alternative solutions:

1. Deepbots can still inherit GoalEnv by pip install gym-robotics, but users will need to install gym-robotics when installing deepbots.
2. We can copy/modify their GoalEnv and integrate it with deepbots without installing gym-robotics.
   https://github.com/Farama-Foundation/Gym-Robotics/blob/main/gym_robotics/core.py

What do you think?

[tsampazk]

This is quite tricky i think. On the one hand, we very much prefer not to introduce any additional dependencies to the framework. Especially for a library that will be or is used by very specific use cases and not universally. On the other hand, i think that it is not proper and we shouldn't just copy the existing GoalEnv from the gym-robotics library.

@KelvinYang0320 is this actually needed for the panda example? Or any other current example?
Edit: I think i remember that some implementations of RL algorithms require GoalEnv and that's why we started working on it, right?

[KelvinYang0320]

@KelvinYang0320 is this actually needed for the panda example? Or any other current example?

I think GoalEnv is quite useful for our robotic tasks as I mentioned here.
I have made a Panda-GoalEnv draft PR to check these features.

1. Goal-compatible observation space
   • observation, desired_goal, and achieved_goal
2. compute_reward(achieved_goal, desired_goal, info)
   • Our get_reward(action) is actually not that suitable for robotics control task.

Another example is that Panda-gym v2.0.1 works with gym-robotics.GoalEnv.

• Gym 0.22.0 compatibility qgallouedec/panda-gym#20
• Add Gym 0.26 support DLR-RM/stable-baselines3#780 (comment)

Edit: I think i remember that some implementations of RL algorithms require GoalEnv and that's why we started working on it, right?

Yes, HER requires the environment to inherits from gym.GoalEnv in SB3.
However, it seems that they are fixing that now.

• Add Gym 0.26 support DLR-RM/stable-baselines3#780

[tsampazk]

Alright thanks for the information!

1. Deepbots can still inherit GoalEnv by `pip install gym-robotics`, but users will need to install `gym-robotics` when installing `deepbots`.

I thought about the additional dependency on gym-robotics. Using this would mean adding additional inheritance to some gym-robotics class that includes GoalEnv. I will need some time to look into this further in detail, so i think we can put this on hold for a while and focus on the fixes regarding deepworlds. Unless you have a clear idea on what we can do specifically, so we can discuss it.

[KelvinYang0320]

@tsampazk I agree to put this on hold. 👌
Check this openai/gym#2456 about the robotics environments.

My goal is to make all the environments inside Gym itself good general purpose benchmarks suitable that someone new to the field of reinforcement learning can look at and say "okay, there's are the things everyone uses that I should play with and understand."

[tsampazk]

Yeah i skimmed through that issue and discussion and followed a couple of links, but it started to look like a long rabbit hole of links and discussions and just stopped myself. :P

That quote is a really really good summary of gym's mission, but i'm not exactly sure what that means for us, being a framework for a robotic simulator. I guess we'll see in the future.

[KelvinYang0320]

@tsampazk I think we can move this issue to Pre-release 0.2.1-dev0 or close it. What do you think?

[tsampazk]

Yes @KelvinYang0320, it seems to still be an open issue, i think that we can move it to the pre-release.