on Real-World Problems is Hard

Reinforcement learning looks straightforward in controlled settings: well-defined states, dense rewards, stationary dynamics, unlimited simulation. Most benchmark results are produced under those assumptions. The real world violates nearly all of them.

Observations are partial and noisy, rewards are delayed or ambiguous, environments drift over time, data collection is slow and expensive, and mistakes carry real cost. Policies must operate under safety constraints, limited exploration, and non-stationary distributions. Off-policy data accumulates bias. Debugging is opaque. Small modeling errors compound into unstable behavior.

Again, reinforcement learning on real world problems is really hard.

Outside of controlled simulators like Atari which live in academia, there is very little practical guidance on how to design, train, or debug. Remove the assumptions that make benchmarks tractable and what remains is a problem space that seems near impossible to actually solve.

But, then you have these examples, and you regain hope:

1. OpenAI Five defeated the reigning world champions in Dota 2 in full 5v5 matches. Trained using deep reinforcement learning.
2. DeepMind’s AlphaStar achieved Grandmaster rank in StarCraft II, surpassing 99.8% of human players and consistently defeating professional competitors. Trained using deep reinforcement learning.
3. Boston Dynamic’s Atlas trains a 450M parameter Diffusion Transformer-based architecture using a combination of real world and simulated data. Trained using deep reinforcement learning.

In this article, I’m going to introduce practical, real-world approaches for training reinforcement learning agents with parallelism, employing many, if not the exact same, techniques that power today’s superhuman AI systems. This is a deliberate selection of academic techniques + hard-won experience gained from building agents which work on stochastic, nonstationary domains.

If you intend on approaching a real-world problem by simply applying an untuned benchmark from an RL library on a single machine, you will likely fail.

One must understand the following:

1. Reframing the problem so that it fits within the framework of RL theory
2. The techniques for policy optimization which actually perform outside of academia
3. The nuances of “scale” in regards to reinforcement learning

Let’s begin.

Prerequisites

If you have never approached reinforcement learning before, attempting to build a superhuman AI—or even a halfway decent agent—is like trying to teach a cat to juggle flaming torches: it mostly ignores you, occasionally sets something on fire, and somehow you’re still expected to call it “progress.” You should be well versed in the following subjects:

1. Markov Decision Processes (MDPs) and Partially Observable Markov Decision Processes (POMDPs): these provide the mathematical foundation for how modern AI agents interact with the world
2. Policy Optimization (otherwise known as Mirror Learning) Details as to how a neural network approximates an optimal policy using gradient ascent 
3. Follow up to 2) Actor Critic Methods and Proximal Policy Optimization (PPO), which are two widely used methods for policy optimization

Each of these requires some time to fully understand and digest. Unfortunately, RL is a difficult problem space, enough so that simply scaling up will not solve fundamental misunderstandings or misapplications of the prerequisite steps as is sometimes the case in traditional deep learning.

---

A real-world reinforcement learning problem

To provide a coherent real-world example, we use a simplified self-driving simulation as the optimization task. I say “simplified” as the exact details are less important to the article’s purpose. However, for real world RL, ensure that you have a full understanding of the environment, inputs, outputs and how the reward is actually generated. This understanding will help you frame your real world problem into the space of MDPs.

Our simulator procedurally generates stochastic driving scenarios, including pedestrians, other vehicles, and varying terrain and road conditions which have been modeled from recorded driving data. Each scenario is segmented into a variable-length episode.

Although many real-world problems are not true Markov Decision Processes, they are typically augmented so that the effective state is approximately Markov, allowing standard RL convergence guarantees to hold approximately in practice.

States
The agent observes camera and LiDAR inputs along with signals such as vehicle speed and orientation. Additional features may include the positions of nearby vehicles and pedestrians. These observations are encoded as one or more tensors, optionally stacked over time to provide short-term history.

Actions
The action space consists of continuous vehicle controls (steering, throttle, brake) and optional discrete controls (e.g., gear selection, turn signals). Each action is represented as a multidimensional vector specifying the control commands applied at each timestep.

Rewards
The reward encourages safe, efficient, and goal-directed driving. It combines multiple objectives Oi, including positive terms for progress toward the destination and penalties for collisions, traffic violations, or unstable maneuvers. The per-timestep reward is a weighted sum:

We’ve built our simulation environment to fit within the 4 tuple interface popularized by Brockman et al., OpenAI Gym, 2016

env = DrivingEnv()
agent = Agent()

for episode in range(N):
   # obs is a multidimensional tensor representing the state
   obs = env.reset()
   done = false

   while not done:
       # act is the application of our current policy π
       # π(obs) returns a multidimensional action
       action = agent.act(obs)
       # we send the action to the environment to receive
       # the next step and reward until complete
       next_obs, reward, done, info = env.step(action)
       obs = next_obs

The environment itself needs to be easily parallelized, such that one of many actors can simultaneously apply their own copy of the policy without the need for complex interactions or synchronizations between agents. This API, developed by OpenAI and used in their gym environments has become the defacto standard.

If you are building your own environment, it would be worthwhile to build to this interface, as it simplifies many things.

Agent

We use a deep actor–critic agent, following the approach popularized in DeepMind’s A3C paper (Mnih et al., 2016). Pseudocode for our agent is below:

class Agent:
   def __init__(self, state_dim, action_dim):

       # --- Actor ---
       self.actor = Sequential(
           Linear(state_dim, 128),
           ReLU(),
           Linear(128, 128),
           ReLU(),
           Linear(128, action_dim)
       )

       # --- Critic ---
       self.critic = Sequential(
           Linear(state_dim, 128),
           ReLU(),
           Linear(128, 128),
           ReLU(),
           Linear(128, 1)
       )

   def _dist(self, state):
       logits = self.actor(state)               
       return Categorical(logits=logits)

   def act(self, state):
       """
       Returns:
           action
           log_prob (behavior policy)
           value
       """
       dist = self._dist(state)

       action = dist.sample()
       log_prob = dist.log_prob(action)
       value = self.critic(state)

       return action, log_prob, value

   def log_prob(self, states, actions):
       dist = self._dist(states)
       return dist.log_prob(actions)

   def entropy(self, states):
       return self._dist(states).entropy()

   def value(self, state):
       return self.critic(state)

   def update(self, state_dict):
       self.actor.load_state_dict(state_dict['actor'])
       self.critic.load_state_dict(state_dict['critic'])

You may be a bit puzzled by the additional methods. More explanation to follow.

Very important note: Poorly chosen architectures can easily derail training. Make sure you understand the action space and verify that your network’s input, hidden, and output layers are appropriately sized and use suitable activations.

Policy Optimization

In order to update the agent, we follow the Proximal Policy Optimization (PPO) framework (Schulman et al., 2017), which uses the clipped surrogate objective to update the actor in a stable manner while simultaneously updating the critic. This allows the agent to improve its policy gradually based on its collected experience while keeping updates within a trust region, preventing large, destabilizing policy changes.

Note: PPO is one of the most widely used policy optimization methods, used to develop both OpenAI Five, Alphastar and many other real world robotic control systems

The agent first interacts with the environment, recording its actions, the rewards it receives, and its own value estimates. This sequence of experience is commonly called a rollout or, in the literature, a trajectory. The experience can be collected to the end of the episode, or more commonly, before the episode ends for a fixed number of steps. This is especially useful in infinite horizon problems with no predefined start or finish, as it allows for equivalent sized experience batches from each actor. 

Here is a sample rollout buffer. However you choose to design your buffer, It’s very important that this rollout buffer be serializable so that it can be sent over the network.

class Rollout:
   def __init__(self):
       self.states = []
       self.actions = []
       # store logprob of action!
       self.logprobs = []
       self.rewards = []
       self.values = []
       self.dones = []

   # Add a single timestep's experience
   def add(self, state, action, logprob, reward, value, done):
       self.states.append(state)
       self.actions.append(action)
       self.logprobs.append(logprob)
       self.rewards.append(reward)
       self.values.append(value)
       self.dones.append(done)
   # Clear buffer after updates
   def reset(self):
       self.states = []
       self.actions = []
       self.logprobs = []
       self.rewards = []
       self.values = []
       self.dones = []

During this rollout, the agent records states, actions, rewards, and next states over a sequence of timesteps. Once the rollout is complete, this experience is used to compute the loss functions for both the actor and the critic.

Here, we augment the agent environment interaction loop with our rollout buffer

env = DrivingEnv()
agent = Agent()
buffer = Rollout()

trainer = Trainer(agent)

rollout_steps = 256

for episode in range(N):
   # obs is a multidimensional tensor representing the state
   obs = env.reset()
   done = false
   steps = 0
   while not done:
       steps += 1
       # act is the application of our current policy π
       # π(obs) returns a multidimensional action
       action, logprob, value = agent.act(obs)
       # we send the action to the environment to receive
       # the next step and reward until complete
       next_obs, reward, done, info = env.step(action)
       # add the experience to the buffer
       buffer.add(state=obs, action=action, logprob=logprob, reward=reward,
                   value=value, done=done)
       if steps % rollout_steps == 0:
           # we'll add more detail here
           state_dict = trainer.train(buffer)
           agent.update(state_dict)
       obs = next_obs

I’m going to introduce the objective function as used in PPO, however, I do recommend reading the delightfully short paper to get a full understanding of the nuances.

For the actor, we optimize a surrogate objective based on the advantage function, which measures how much better an action performed compared to the expected value predicted by the critic. 

The surrogate objective used to update the actor network:

Note that the advantage, A, can be estimated in various ways, such as Generalized Advantage Estimation (GAE), or simply using the 1-step temporal-difference error, depending on the desired trade-off between bias and variance (Schulman et al., 2017).

The critic is updated by minimizing the mean-squared error between its predicted value V(s_t) and the observed return R_t​ at each timestep. This trains the critic to accurately estimate the expected return of each state, which is then used to compute the advantage for the actor update.

In PPO, the loss also includes an entropy component, which rewards policies that have higher entropy. The rationale is that a policy with higher entropy is more random, encouraging the agent to explore a wider range of actions rather than prematurely converging to a deterministic behavior. The entropy term is typically scaled by a coefficient, β, which controls the trade-off between exploration and exploitation.

The total loss for PPO, then becomes:

Again, in practice, simply using the default parameters set forth in the baselines will leave you disgruntled and possibly psychotic after months of tedious hyperparameter tuning. In order to save you costly trips to the psychiatrist, please watch this very informative lecture by the creator of PPO, John Schulman. In it, he describes very important details, such as value function normalization, KL penalties, advantage normalization, and how commonly used techniques, like dropout and weight decay will poison your project. 

These details in this lecture, which are not specified in any paper, are critical to building a functional agent. Again, as a cautious warning: if you simply try to use the defaults without understanding what is actually happening with policy optimization, you will either fail or waste tremendous time. 

Our agent can now be updated. Note that, since our optimizer is minimizing an objective, the signs from the PPO objective as described in the paper need to be flipped.

Also note, this is where our agent’s functions will come in handy.

def compute_advantages(rewards, values, gamma, lambda):
   # calc advantages as you'd like

def compute_returns(rewards, gamma):
   # calc returns as you'd like

def get_batches(buffer):
   # randomize and return tuples
   yield batch

class Trainer:
   def __init__(self, agent, config):
       self.agent = agent                # ActorCriticAgent instance
       self.lr = config.get("lr", 3e-4)
       self.num_epochs = config.get("num_epochs", 4)
       self.eps = config.get("clip_epsilon", 0.2)
       self.entropy_coeff = config.get("entropy_coeff", 0.01)
       self.value_loss_coeff = config.get("value_loss_coeff", 0.5)
       self.gamma = config.get("gamma", 0.99)
       self.lambda_gae = config.get("lambda", 0.95)
      
       # Single optimizer updating both actor and critic
       self.optimizer = Optimizer(params=list(agent.actor.parameters()) +
                                         list(agent.critic.parameters()),
                                  lr=self.lr)

   def train(self, buffer):
       # --- 1. Compute advantages and returns ---
       advantages = compute_advantages(buffer.rewards, buffer.values, self.gamma, self.lambda_gae)
       returns = compute_returns(buffer.rewards, self.gamma)

       # --- 2. PPO updates ---
       for epoch in range(self.num_epochs):
           for batch in get_batches(buffer):
               states, actions, adv, ret = batch

               # --- Probability ratio ---
               ratio = actor_prob(states, actions) / actor_prob_old(states, actions)

               # --- Actor loss (clipped surrogate) ---
               surrogate1 = ratio * adv
               surrogate2 = clip(ratio, 1 - self.eps, 1 + self.eps) * adv
               actor_loss = -mean(min(surrogate1, surrogate2))

               # --- Entropy bonus ---
               entropy = mean(policy_entropy(states))
               actor_loss -= self.entropy_coeff * entropy

               # --- Critic loss ---
               critic_loss = mean((critic_value(states) - ret) ** 2)

               # --- Total PPO loss ---
               total_loss = actor_loss + self.value_loss_coeff * critic_loss

               # --- Apply gradients ---
               self.optimizer.zero_grad()
               total_loss.backward()
               self.optimizer.step()

        return self.agent.state_dict()

The three steps, defining our environment, defining our agent and its model, as well as defining our policy optimization procedure are complete and can now be used to build an agent with a single machine.

Nothing described above will get you to “superhuman.”

Let’s wait for 2 months for your Macbook Pro with the overpriced M4 chip to start showing a 1% improvement in performance (not kidding).

---

The Distributed Actor-Learner Architecture

The actor–learner architecture separates environment interaction from policy optimization. Each actor operates independently, interacting with its own environment using a local copy of the policy, which is mirrored across all actors. The learner does not interact with the environment directly; instead, it serves as a centralized hub that updates the policy and value networks according to the optimization objective and distributes the updated models back to the actors.

This separation allows multiple actors to interact with the environment in parallel, improving sample efficiency and stabilizing training by decorrelating updates. This architecture was popularized by DeepMind’s A3C paper (Mnih et al., 2016), which demonstrated that asynchronous actor–learner setups could train large-scale reinforcement learning agents efficiently.

Actor

The actor is the component of the system that directly interacts with the environment. Its responsibilities include:

1. Receiving a copy of the current policy and value networks from the learner.
2. Sampling actions according to the policy for the current state of the environment.
3. Collecting experience over a sequence of timesteps 
4. Sending the collected experience to the learner asynchronously.

Learner

The learner is the centralized component responsible for updating the model parameters. Its responsibilities include:

1. Receiving experience from multiple actors, either in full rollouts or in mini-batches.
2. Computing loss functions
3. Applying gradient updates to the policy and value networks.
4. Distributing the updated model back to actors, closing the loop.

This actor–learner separation is not included in standard baselines such as OpenAI Baselines or Stable Baselines. While distributed actor–learner implementations do exist, for real-world problems the customization required may make the technical debt of adapting these frameworks outweigh the benefits of use.

Now things are beginning to get interesting.

With actors running asynchronously, whether on different parts of the same episode or entirely separate episodes our policy optimization gains a wealth of diverse experiences. On a single machine, this also means we can accelerate experience collection dramatically, cutting training time proportionally to the number of actors running in parallel.

However, even the actor–learner architecture will not get us to the scale we need due to a major problem: synchronization.

In order for the actors to begin processing the next batch of experience, they all need to wait on the centralized learner to finish the policy optimization step so that the algorithm remains “on policy.” This means each actor is idle while the learner updates the model using the previous batch of experience, creating a bottleneck that limits throughput and prevents fully parallelized data collection. 

Why not just use old batches from a policy that was updated more than one step ago?

Using off-policy data to update the model has proven to be destructive. In practice, even small policy lag introduces bias in the gradient estimate, and with function approximation this bias can accumulate and cause instability or outright divergence. This issue was observed early in off-policy temporal-difference learning, where bootstrapping plus function approximation caused value estimates to diverge instead of converge, making naïve reuse of stale experience unreliable at scale.

Luckily, there is a solution to this problem.

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Invented at DeepMind, IMPALA (and it’s predecessor, SEED-RL) introduced a concept called V-Trace, which allows us to update on policy algorithms with rollouts which were generated off policy.

This means that the utilization of the entire system remains constant, instead of having synchronization wait blocks (the actors need to wait for the latest model update as is the case in A3C). However, this comes at a cost: because actors use slightly stale parameters, trajectories are generated by older policies, not the current learner policy. Naively applying on-policy methods (e.g., standard policy gradient or A2C) becomes biased and unstable.

To correct for this, we introduce V-Trace. V-Trace uses an importance-sampling–based correction that adjusts returns to account for the mismatch between the behavior policy (actor) and target policy (learner).

In on-policy methods, the starting ratio (at the beginning of each mini-epoch as is the case in PPO) is ~ 1. This means the behavior policy is equal to the target policy.

In IMPALA, however, actors continuously generate experience using slightly stale parameters, so trajectories are sampled from a behavior policy μ that may differ nontrivially from the learner’s current policy π. Simply put, the starting ratio != 1. This importance weight, allows us to approximate how stale the policy which generated the experience is.

We only need one more calculation to correct for this off-policy drift, which is to calculate the ratio of the behavior policy μ, compared to the current policy, π at the start of the policy update. We can then recalculate the policy loss and value targets using a clipped versions of these importance weights — rho for the policy and c for the value targets.

We then recalculate our td-error (delta):

Then, use this value to calculate our importance weighted values.

Now that we have sample corrected values, we need to recalculate our advantages.

Intuitively, V-trace compares how probable each sampled action is under the current policy versus the old policy that generated it.

If the action is still likely under the new policy, the ratio is near one and the sample is trusted.

If the action is now unlikely, the ratio is small and its influence is reduced.

Because the ratio is clipped at one, samples can never be upweighted — only downweighted — so stale or mismatched trajectories gradually lose impact while near-on-policy rollouts dominate the learning signal.

This very important set of methods allows us to extract all of the horsepower from our training infrastructure and completely removes the bottleneck from synchronization. We no longer need to wait for all the actors to finish their rollouts, wasting costly GPU + CPU time.

Given this method, We need to make some modifications to our actor learner architecture to take advantage.

Massively Distributed Actor-Learner Architecture

As described above, we can still use our Distributed Actor-Learner architecture, however, we need to add a few components and use some techniques from NVIDIA to allow for trajectories and weights to be received without any need for synchronization primitives or a central manager.

Key-Value (KV) Database

Here, we add a simple KV database like Redis to store trajectories. The addition requires us to serialize each trajectory after an actor completes gathering experience, then each actor can simply add it to a Redis list. Redis is thread safe, so we don’t need to worry about synchronization for each actor. 

When the learner is ready for a new update, it can simply pop the latest trajectories off of this list, merge them, and perform the policy optimization procedure.

# modifying our actor steps
r = redis.Redis(...)Py

...

if steps % rollout_steps == 0:
 # instead of training, just serialize and send to a buffer
 buffer_data = pickle.dumps(buffer)
 r.rpush("trajectories", buffer_data)


The learner can simply grab trajectories in a batch as needed from this list, 
which updates the weights.


# on the learner
trajectories = []

while len(trajectories) <= trajectory_batch_size:
 trajectory = pickle.loads(r.lpop("trajectories"))
 trajectories.append(trajectory)

# we can merge these into a single buffer for the purposes of training
buffer = merge_trajectories(trajectories)

# continue training

Multiple Learners (optional)

When you have hundreds of workers, a single GPU on the learner can become a bottleneck. This can cause the trajectories to be very off-policy, which degrades learning performance. However, as long as each learner runs the same code (same backward passes), they can each process completely different trajectories independently.

Under the hood, if you are using PyTorch, NVIDIA’s NCCL library handles the all-reduce operations required to synchronize gradients. This ensures that model weights remain consistent across all learners. You can launch each learner process using torchrun, which manages the distributed execution and coordination of the gradient updates automatically.

import torch.distributed as dist

r = redis.Redis(..)

def setup(rank, world_size):
   # Initialize the default process group
   dist.init_process_group(
       backend="nccl",
       init_method=os.environ["MASTER_ADDR"],  # will set in launch command
       rank=rank,
       world_size=world_size
   )
   torch.cuda.set_device(rank % torch.cuda.device_count())

# apply training as above
...

total_loss = actor_loss + self.value_loss_coeff * critic_loss

# applying our training step above
self.optimizer.zero_grad()
total_loss.backward()
# we need to use a dist operatiom
for p in agent.parameters():
  dist.all_reduce(p.grad.data)
  p.grad.data /= world_size

optimizer.step()
if rank == 0:
  # update params from the master
  r.rpush("params", agent.get_state_dict())

I’m dramatically oversimplifying the application of NCCL. Read the PyTorch documentation regarding distributed training

Assuming we use 2 nodes, each with 2 learners — 

On node 1:

MASTER_ADDR={use your ip} \
MASTER_PORT={pick an unused port} \
WORLD_SIZE=4 \
RANK=0 \
torchrun --nnodes=2 --nproc_per_node=2 \
--rdzv_backend=c10d --rdzv_endpoint={your ADDR}:{your port} learner.py

and on node 2:

MASTER_ADDR={use your ip} \
MASTER_PORT={pick an unused port} \
WORLD_SIZE=4 \
RANK=2 \
torchrun --nnodes=2 --nproc_per_node=2 \
--rdzv_backend=c10d --rdzv_endpoint={your ADDR}:{your port} learner.py

Wrapping up

In summary, scaling reinforcement learning from single-node experiments to distributed, multi-machine setups is not just a performance optimization—it’s a necessity for tackling complex, real-world tasks.

We covered:

1. How to refactor problem spaces into an MDP
2. Agent architecture
3. Policy optimization methods that actually work
4. Scaling up distributed data collection and policy optimization

By combining multiple actors to collect diverse trajectories, carefully synchronizing learners with techniques like V-trace and all-reduce, and efficiently coordinating computation across GPUs and nodes, we can train agents that approach or surpass human-level performance in environments far more challenging than classic benchmarks.

Mastering these strategies bridges the gap between research on “toy” problems and building RL systems capable of operating in rich, dynamic domains, from advanced games to robotics and autonomous systems.

References

• Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., … & Silver, D. (2019). Grandmaster level in StarCraft II using multi‑agent reinforcement learning. Nature.
• Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., … & Salimans, T. (2019). Dota 2 with large scale deep reinforcement learning. arXiv:1912.06680

• Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., … & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533.
• Schulman, J., Levine, S., Moritz, P., Jordan, M., & Abbeel, P. (2015). Trust Region Policy Optimization. ICML 2015.
• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., … & Kavukcuoglu, K. (2018). IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. ICML 2018.
• Espeholt, L., Stooke, A., Ibarz, J., Leibo, J.Z., Zambaldi, V., Song, F., … & Silver, D. (2020). SEED RL: Scalable and Efficient Deep-RL with Accelerated Centralized Learning. arXiv:1910.06591.