c# - 如何在 DQN(深度 Q 网络)中分配状态？

Question

我正在使用自动驾驶仪进行飞行模拟，所以我需要制作一个 DQN(深度 Q 网络)来控制自动驾驶仪，但我不知道最佳状态数。

模拟是统一完成的，所有的环境和物理也都完成了，DQN 只需要输出 (W,A,S,D) 来控制飞机，我找到了一个控制 CARTPOLE 的代码，理论上应该能够很好地训练和控制飞机，唯一的问题是我不确定我选择的状态是否正确。

这是代码:

    
    import os
    import random
    import gym
    import numpy as np
    from collections import deque
    from keras.models import Sequential
    from keras.layers import Dense
    from keras.optimizers import Adam



    class DQNAGENT:

        def __init__(self,state_size,_action_size):

            self.state_size = state_sizes
            self.action_size = actions_sizes
            self.memory = deque(maxlen=2000)
            self.gamma = 0.95
            self.epsilon = 1.00
            self.epsilon_decay_rate = 0.995
            self.epsilon_min = 0.01
            self.learning_rate = 0.001
            self.model = self.build_model()

        def buildmodel(self):

            model = Sequential()
            model.add(Dense(24, input_dim=self.state_size, activation='relu'))
            model.add(Dense(24, activation='relu'))
            model.add(Dense(self.action_size, activation='linear'))
            model.compile(loss='mse',optimizer=Adam(lr=self.learning_rate))
            return model

        def remember(self, state, action, reward, next_state, done):

            self.memory.append((state, action, reward, next_state, done))

        def act(self, state):

            if np.random.rand()  self.epsilon_min:
                self.epsilon *= self.epsilon_decay_rate

        def load(self, name):

            self.model.load_weights(name)

        def save(self, name):

            self.model.save_weights(name)


    def main():
        #environemnet variables
        state_sizes=0
        actions_sizes=4
        #training Variables
        batch_size=32
        n_episodeds=100
        output_directory= 'model_output/autopilot'
        if not os.path.exists(output_directory):
            os.makedirs(output_directory)

        agent = DQNAGENT(state_sizes,actions_sizes)
        done = False

        for e in range(n_episodeds):
            state = #states of the game
            for time in range(5000):
                action = agent.act(state)
                #next_state, reward, done, _ = ##env.step(action)
                #put the next state from unity
                reward = reward if not done else -10
                agent.remember(state, action, reward, next_state, done)
                state = next_state
                if len(agent.memory) > batch_size:
                    agent.replay(batch_size)
    

where the agent class is the agent that will be trained those functions are okay but in the Main the state sizes are set to zero because i don't know yet the number also these three lines i can't convert to be able to run on my project

state = #states of the game
action = agent.act(state)
next_state, reward, done, _ = ##env.step(action)

原始代码有这些行:

env = gym.make('CartPole-v1')
state_size = env.observation_space.shape[0]
state = env.reset()
next_state, reward, done, _ = env.step(action)

因为它从 Gym 包中获取这些变量，但我需要手动输入这些变量，我的环境将包括空速、飞机位置、机场位置等，这就是我认为会写的，所以如果有人可以帮我弄清楚这是否正确，或者更好地告诉我什么是最佳状态，我们将不胜感激。

异常(exception)的结果是这样的。

statesizes = 4
states= "how to write those states in this variable"

Answer 1

状态只是您的代理在当前“帧/步骤”中拥有的信息。
它是代理选择 Action 所需要的 agent.act(state)。

在 CartPole 示例中，状态是一个包含 4 个值的框:

• 购物车位置
• 购物车速度
• 极角
• 尖端极速

在您的飞行自动驾驶仪中，状态将是您需要代理人做出决定所需的信息，例如:

• 当前纬度
• 当前速度
• ...