RL & Self-
Improvement

Agents, Environments, and Reward Signals

Reinforcement Learning (RL) describes a training paradigm where an agent takes actions in an environment, receives a reward signal, and updates its policy to maximize cumulative future reward. Unlike supervised learning — which needs labeled input-output pairs — RL only requires a reward function that evaluates outputs after the fact.

The core loop is: observe state → select action → receive reward → update policy. Formally, this is modeled as a Markov Decision Process (MDP): at each step, the policy maps the current state to a probability distribution over actions, and the environment returns a new state and a scalar reward.

Reinforcement Learning from Human Feedback

Pretraining teaches an LLM to predict text. But predicting text is not the same as being helpful, honest, or harmless. RLHF — introduced at scale in the InstructGPT paper (Ouyang et al., 2022) — is the training pipeline that bridges this gap. It has three stages:

The reward model is trained on human preferences (ranking A over B), not on absolute quality labels. This is cheaper to collect and more reliable — it is easier for humans to say "this response is better than that one" than to assign a number from 1 to 10. The Elo-style comparisons also scale across annotators with differing absolute standards.

Replacing Human Labels with AI Feedback

Human preference labeling is expensive, slow, and inconsistent across annotators. Constitutional AI (CAI), introduced by Anthropic in Bai et al. (2022), proposes a partial solution: replace the human preference labeling step with AI-generated feedback, guided by a written set of principles called a constitution.

The CAI pipeline has two phases:

Dropping the RL in RLHF

RLHF is effective but complex: it requires training a separate reward model, running PPO (which is notoriously unstable), and carefully tuning the KL penalty. Direct Preference Optimization (DPO), introduced by Rafailov et al. (2023), achieves the same alignment objective — training the model to produce preferred responses — using a single supervised learning step, with no RL loop and no explicit reward model.

DPO's key insight: there is a closed-form relationship between the optimal policy and the reward function. You can reparametrize the reward in terms of the policy itself, and then directly optimize the policy on preference pairs using a binary cross-entropy loss — without ever constructing a reward model or running PPO.

Reflexion, Verifiable Rewards, and Process Reward Models

Beyond training-time alignment, agents can improve their behavior at inference time through self-reflection and iterative revision — without gradient updates. These self-improvement loops are increasingly central to how high-capability agents operate.

memory = []

for attempt in range(MAX_ATTEMPTS):
    result = agent.run(task, memory=memory)

    if result.success:
        break

    # Reflect on failure in natural language
    reflection = llm.generate(
        f"You attempted: {task}\n"
        f"Your output: {result.output}\n"
        f"The outcome: {result.feedback}\n"
        "What went wrong and what will you do differently?"
    )
    memory.append(reflection)   # stored for next attempt

Verified References

Every claim in this section is grounded in one of these sources. No content is generated from model training data alone.