stable-baselines3 ValueError when creating DummyVecEnv

[sebnapi]

When running:

df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs,
                                             PPO_model_kwargs,
                                             DDPG_model_kwargs,
                                             timesteps_dict)

I run into this issue:

DLR-RM/stable-baselines3#1151

"You tried to create multiple environments, but the function to create them returned the same instance "
            "instead of creating different objects. "
            "You are probably using `make_vec_env(lambda: env)` or `DummyVecEnv([lambda: env] * n_envs)`. "
            "You should replace `lambda: env` by a `make_env` function that "
            "creates a new instance of the environment at every call "
            "(using `gym.make()` for instance). You can take a look at the documentation for an example. "
            "Please read https://github.com/DLR-RM/stable-baselines3/issues/1151 for more information."

in this spot:

FinRL/finrl/agents/stablebaselines3/models.py

Lines 287 to 310 in 7414adb

	trade_env = DummyVecEnv(
	[
	lambda: StockTradingEnv(
	df=trade_data,
	stock_dim=self.stock_dim,
	hmax=self.hmax,
	initial_amount=self.initial_amount,
	num_stock_shares=[0] * self.stock_dim,
	buy_cost_pct=[self.buy_cost_pct] * self.stock_dim,
	sell_cost_pct=[self.sell_cost_pct] * self.stock_dim,
	reward_scaling=self.reward_scaling,
	state_space=self.state_space,
	action_space=self.action_space,
	tech_indicator_list=self.tech_indicator_list,
	turbulence_threshold=turbulence_threshold,
	initial=initial,
	previous_state=last_state,
	model_name=name,
	mode="trade",
	iteration=iter_num,
	print_verbosity=self.print_verbosity,
	)
	]
	)

[sebnapi]

Hey, @XiaoYangLiu-FinRL can you give me some ideas on how to fix that? I don't really understand why the reference is the same, the StockTradingEnv object should be always a new different one.

[sebnapi]

I tried downgrading to stable-baselines3==1.7.0 before the check, that had some other consequences:

!pip3 install setuptools==65.5.0 # openai/gym#3176
!pip install git+https://github.com/openai/gym.git@9180d12e1b66e7e2a1a622614f787a6ec147ac40 # openai/gym#3202
!pip install stable_baselines3==1.7.0

But now training wont work at all, because there is a mismatch between gym==0.17 and gymnasium. Absolute bottom less pit, machine learning broke python, this feels like javascript.

FinRL is using gymnsium and the elegantrl dependency is using gym.

It might be helpful to just dockerize the application with fixed requirements, as the different versions break the code all the time.

167 if supported_action_spaces is not None:
--> 168     assert isinstance(self.action_space, supported_action_spaces), (
 169         f"The algorithm only supports {supported_action_spaces} as action spaces "
 170         f"but {self.action_space} was provided"
 171     )
 173 if not support_multi_env and self.n_envs > 1:
 174     raise ValueError(
 175         "Error: the model does not support multiple envs; it requires " "a single vectorized environment."
 176     )

AssertionError: The algorithm only supports (<class 'gym.spaces.box.Box'>, <class 'gym.spaces.discrete.Discrete'>, <class 'gym.spaces.multi_discrete.MultiDiscrete'>, <class 'gym.spaces.multi_binary.MultiBinary'>) as action spaces but Box(-1.0, 1.0, (29,), float32) was provided

[sebnapi]

See #1002 (comment)

[sebnapi]

See DLR-RM/stable-baselines3#1151 (comment)